This is an interesting thread. Many people have laid out a version of this idea, but I want to try my hand at explaining it. I generally gloss over the role that Kanji and vocab play in this discussion as it is a constant background consideration.
N5 is difficult for 2 main reasons. First, it is all new, every aspect of Japanese, especially if you are coming to grammar lessons with no kana or kanji background. Second, you are learning the ‘universal grammar’ it is very amorphous at this stage. You are not just learning what the conjunctions are, but how conjunctions function as a whole in Japanese. Same goes for particles, Japanese’s conjugation system, word/phrase/clause order, etc.
Once you get to N4 you should be comfortable enough with those foundations, but now you have to iterate on each of them in a multiplicity of contexts. How does any given particle interact with the others. What are the main patterns to conjunctions, what are the different ways to express modal concepts, etc.
N3 as most people has pointed out is really the point where you have 80% of core grammar locked in and reading becomes much more accessible, however you are now dipping into the concept of nuance. Not just that you should use a certain grammar in a certain context, but that given your nuance your mood, or social situation, one is more or less appropriate.
N2 is the same but more so, there are so many near synonyms that have shades of meaning and Japanese patterns or thinking that are more or less foreign to non natives that it takes a lot to unlock that idea, but you are still using that foundation from N5 and N4, unless it is a formal or archaic conjugation you do not need to learn new particles, or new conjugations, but how in situations those take on additional meaning.
I am not to N1 fully yet, but in principle it is the same, but the domains of contexts are even further expanded.
For those truly ambitious, N1 is also really just the start. Think in English that there is so much more English than what you learn as a senior in high school. There are higher level tests as well: