Don’t be discourage just about the N4 degree or sth like that, even N5 certification is an achievement. You’re doing better than your past self, and those little, step-by-step degrees will show that. And from my own experience, merely the feeling of doing well in the exam is good enough motivation for me to pursuit higher N level.
So, do it!
I started studying way back in 2010 when I was 16 (and I’m 31 now!). On and off as well, to be honest I really didn’t take it seriously at all and eventually life and work stuff took over and I only got back to studying this year (seriously this time) so I completely relate to you. Even though I felt like I could probably pass N4, I still took the N5 JLPT. I do consider myself a completionist though 
Imo it’s actually a nice introduction for you to get a feel of how JLPT exams are coordinated in your area so you won’t get a shock when you sit for the advanced levels. The rules and regulations can get quite overwhelming sometimes!
Also, @Minh_Nguyen is right! Don’t be discouraged at all, it’s a big deal going from no certificate to actually holding one in your hands! Always remember what they always say here; it’s a marathon, not a sprint 
Everything at your own pace 
All the best!
@Minh_Nguyen, @virtualputeri - thank you both so much for the encouragement! Writing this next to all those amazing people tackling the upper tier levels is a little humbling, but I love the take of 'doing better than my past self ’ and actually holding a certificate of any kind in Japanese would absolutely make my day!
Besides, I haven’t really considered the aspect of getting used to the exam process, but that makes a ton of sense for higher tier levels too.
So… ありがとうござます - this helped a ton!