For what it’s worth:
I started immersing with native material after I finished N5 here on bunpro. That’s about 1000 vocab and no knowledge of N4 grammar. But from the beginning, I always tried to immerse myself with at least some sentences, some glimpses at manga, websites, tadoku readers, etc. Bunpro is doing a really good job providing you with example sentences and reading practices.
Still my first native manga (chis sweet home) was harsh! I heavily relied on the wanikani bookclub to help me through. But by book 3 or 4 or something, I was able to breeze through the series, I switched from intensively reading to extensively reading (both have their place and are important imho!).
Now, a few months later, I’m still somewhere in the beginner stages of N4 of bunpro. It looks like slow progress, but most of the time I add vocab or grammar I already know about half of it. I have done a lot of reading manga and watching anime and without trying too much, I learned to recognize a lot of grammar and vocab just by reading.
I hope by sharing my experience you will gain some confidence for your immersion practice
Have fun with reading 