I just finished up another depressing study session with ~55% accuracy. When thinking about how this session has taken so many cards down an SRS level, and how many extra reviews that will generate in the future, I’m overwhelmed with frustration.
I’m about two weeks past the point of finishing N1 grammar, and so my daily Bunpro sessions are all about trying to push grammar points to higher levels and achieve full recall of all the 915 grammar points I’ve learned. But Bunpro’s SRS algorithm is sabotaging this goal, and instead just producing a daily ritual of feeling bad about what I’ve forgotten.
The basic problem is shown in an example like this. Let’s say you have a grammar point at level 9. This means you haven’t seen it in 2 months. You miss it. Now it’s down to level 8.
The next time you see this grammar point will be in 1 month! So you’re probably going to miss it again too. Now it’s down to level 7.
Again, you still won’t see this grammar point for 2 weeks. 2 weeks is quite a while; a lot of other Japanese is being studied in those 2 weeks, during which you haven’t seen the problematic grammar point. So there’s a good chance you’ll miss it again! So down to level 6.
And even level 6 is an 8 day interval, which doesn’t give us a great chance of success. I think you see the problem.
This has been my experience. Grammar points that aren’t completely trivial spend a lot of time bouncing between levels 10 (4 months) and 6 (4 days). Sometimes I’ll get “lucky” twice in a row and push from level 10 to 12, so the grammar point will disappear forever. Even this is a bittersweet victory; I don’t feel confident I truly remember the 410 level 12 grammar points I’ve supposedly “mastered”, since I haven’t been tested on them since whenever it was that I “mastered” them.
Personally, I try to mitigate this problem a little bit by turning on full ghosts. That gives me a 4 + 12 + 24 + 48 = 88 hour ≈ 4 day period in which I’m getting reminded of the problematic grammar point to try to boost it. But there’s plenty of time between those 4 days and the 14 day interval it’ll take for a level 7 grammar point to come around, for it to fall out of my brain again. And if I get “unlucky” too much, I’ll end up back at level 5, seeing the same grammar point while I’m still reviewing its ghost. (Sometimes even the same sentence, for the higher-level grammar points where Bunpro’s sentence inventory is low!)
Let’s take a step back and remember what SRS is meant to accomplish and how it works.
Remember that the goal of spaced repetition is to remind you of something right before you’re likely to forget it, thus reinforcing the memory. And in theory, each time you do this, it’ll reinforce the memory for a little longer.
Bunpro’s design assumes that all of our brains are calibrated to match Bunpro’s SRS intervals: i.e., we forgot things after a bit more than 4 hours, but reminding us then extends the memories for 8 hours, after which they fade in ~24 hours, but reminding us extends the memories for 2 days, etc. Obviously, this is not true for all brains. And it’s worse than that: different grammar points are hard for each brain in different ways. Using the exact same intervals for every brain + grammar point combination is bad!
But it’s even worse than that. Bunpro’s SRS design assumes that we never forgot something: it assumes we can always remind ourselves right before forgetting. But if you’ve forgotten something that’s at level X, actually, your brain is much closer to the initial zero-knowledge state (i.e. level 1) than it is to the current level (Bunpro’s assumption, of just level X - 1). Sure, you’ve gained something from repeated exposure, but you need to build that memory back up before we can be confident that it’s truly back to near level X.
How could Bunpro do better? Fortunately, this is a solved problem.
Remember, our goal is to figure out a SRS system that will prompt the user “right before” they’re about to forget. A good way of operationalizing this is something like: when the user has a 90% chance of remembering the grammar point. So we need to take all the data so far about the user’s performance, both on Bunpro grammar points in general and on this specific grammar point we’re quizzing them on. And we need to predict when they have a 90% chance of remembering it.
It turns out, over the last few years computers have gotten really good at using data to make predictions! And even refining those predictions over time! This field is called machine learning. And some smart people have applied it to spaced repetition systems, and created one that is extremely accurate. It’s what powers the latest version of Anki.
This method also does a good job of finding the right “re-learning” curve, after the user forgets a grammar point. In my experience using this in Anki, it starts out with a small-but-not-tiny interval (e.g. 5 days). If you’re good at that point, it’ll pick increasingly-large intervals. If you forget again, it’ll bump you down to 1 day, building you back up to large intervals more slowly. The key thing is that the intervals it chooses are customized to your overall review performance, and to the specific problematic grammar point. It’s much better than Bunpro’s two options, of “reset to level - 1” or “reset to level - 1 and also add some short-interval ghosts over the next 4 days”.
And this method is generally much less discouraging, since it trains itself to give the user 90% accuracy in every review session.
Finally, as a bonus, SRS systems based on real user performance like this will never count an item as “Mastered” and thus fail to remind you of them ever again. You’ll keep getting reminded of them as necessary forever, and if you forget them (which will happen, statistically, 10% of the time), they’ll come back into the rotation for just long enough to keep them in your brain.
Well, I got that off my chest.
Honestly, I’m not that hopeful that Bunpro will make core changes to their platform to use a more user-friendly SRS algorithm. The simple-to-explain SRS interval system with cute names like “Adept” or “Expert” seems pretty baked into the site. And it’ll meet user’s expectations coming from WaniKani. (I probably will be posting another such rant over there, where I have a similar problem after spending 270 days post level 60 with ~800 items bouncing between levels 4 and 8.)
But if anyone else is in a similar situation to me, where sometimes they finish a Bunpro session and just feel depressed and discouraged about all the things they’ve forgotten, I want to provide a little bit of an explanation. It’s not your fault: you’re not bad at Japanese, or bad at studying. You’re just using a SRS system which is poorly designed and will inevitably fail at helping you remember the harder grammar points, since it doesn’t customize to your review history and it doesn’t have a good re-learning strategy to react to forgetting a higher-level grammar point.
And maybe writing this essay will finally give me the courage to do what I should have done some time ago, and give up on Bunpro and reallocate that daily time toward more reading practice and native content consumption. It’s really hard for me, since I enjoyed using Bunpro as a learning tool so much. I’m intrinsically motivated by pushing things along progress bars, and I think working with specialized SRS web apps like WaniKani and Bunpro is a great low activation energy way of learning Japanese. But if Bunpro is not actually helping me remember the things it taught me, then what’s the point?
Thanks for listening.
I considered scrapping the site to create a (private) deck several times. Not to stop paying and keep on using Bunpro, I would have continued to sub, but to save time while doing reviews (I know it’s probably against the ToS, I’m just mentioning that to illustrate the frustration that comes with the SRS).