If I had to guess, I would assume that the phenomenon that you’re experiencing may actually be largely in part to you being a non-native speaker to begin with. I get this all the time as well. For example, if one Japanese person were to ask another Japanese person if something is natural, they will say yes, because they assume the other person has the same level of context/education to understand how something can subtly change in meaning based on scenario.
However, if you get a foreigner to ask a Japanese native speaker the question, the native speaker will assume that there is no way they could have the knowledge to understand something subtle, so will default to the simplest way to interpret it, even if they themselves don’t interpret it that way as a native speaker.
This is not just a Japanese to English problem. It is a type of mental bias that makes us view our conversation partner as ‘not as capable’ as us if they aren’t a native speaker. This is just my personal theory though. It’s a shame really, it can hinder useful conversations sometimes.