I never thought of it that way. Compares to English and Thai (my 2 known languages), I view kanji just like the other alphabets. Just more complicated and there are way more of them. You don’t really think about how you know English alphabets right? You just do.
Learning a language is part understanding and part memorisation anyway, and it takes time. Kanji having different reading can be seen as similar to English words with the same spelling but actually pronounced differently (read present tense vs read past tense). At some points, after lots of exposure/experience, you stop feeling like you are memorising them, but it just came to you.
Japanese being my 3rd language I’m attempting to study is probably why I’m not too hard on myself. With English it look me a long time before I became confident. I’m still definitely making mistakes, but that confidence weirdly enough also came in the form of not giving a crap about making mistakes lol.