Bunpro's bad SRS algorithm is discouraging

I am sometimes discouraged, but I don’t think it’s the timing that causes it.
When my reviews are low it’s usually

  1. interference, I get similar grammar points mixed up
  2. I do a bunch of really overdue reviews

Jake’s fixes would help with the second.

For the first, I don’t think the SRS is the solution. If something isn’t sinking in, you do need some kind of outside study.

Bunpro has lot’s of good info on the grammar point page. I think some kind of automatic redirect to the grammar point for troubled grammar would be useful, but might feel condescending or intrusive.

I think if you pulled anki’s SRS algorithm and put it directly into bunpro, you would be equally discouraged because bunpro reviews are harder than anki reviews. And it doesn’t fix problem 1.

I think ‘troubled grammar’ is the same as a leech.

If you want to fix this, your ‘learning steps’ and ‘relearning steps’ don’t count as lapses. So I looked up a table and hard coded in all the beginner steps as learning steps. (15m, 1d, 3d, 7d, 16d) going straight into ‘mature’ at 30d

You can also pick any number.

2 Likes

I have to admit, I don’t quite get it either. Also I might be mistaken, but I feel there’s a misunderstanding somewhere in this conversation.

How would this makes a difference from an SRS to another ?
No rhetoric intended, I’m really asking. To me, the algorithm would consider a single grammar point as the thing to learn, to reproduce if you want, cycling through all associated sentences, just as it is now. I mean switching to FSRS would only adjust the timing a grammar point (not a single sentence) is presented to an individual, depending on is ability to reproduce it. Nothin less, nothing more.


2 Likes

One of the main differences is in the goal. The goal of Bunpro is to learn, not to memorize. For the sake of this explanation, I’ll just give these two terms a quick description of how I am using them.

To learn - To understand and then be able to apply knowledge in unfamiliar situations.
To memorize - To be able to recite previously seen information, regardless of understanding.

If the sentence presented is always the same (as with most cases in Anki), one is able to memorize the sentence, which is a big tip-off as to what the grammar is going to be. We actually have people mentioning this as a negative thing quite frequently, especially when it comes to ghosts. It’s the ‘I feel like I am just memorizing sentences, not actually understanding why to use certain grammar’ effect. With Anki it is the same. If you just memorize the sentence, yes you will get the question correct more often, but have you actually learned anything? No, you have memorized it.

In Bunpro, you are shown a new sentence at each new SRS level. This takes away the possibility of memorizing the answer, and forces critical analysis of the sentence in order to retrieve the information. This cannot be done unless something is actually learned, due to the things needed to take into consideration about the question. For example, who is talking to who, what is the style of writing, are there any relevant conjugations before/after that will eliminate some possibilities, does the sentence have a certain level of politeness, etc etc. All of these analysis steps become completely irrelevant if you see ‘昨日、図書館で借りた本を------。’ and then think to yourself ‘Oh I remember this sentence, the answer is XYZ’.

This Reddit post (and the accompanying scientific article is a good discussion on the topic. https://www.reddit.com/r/Anki/comments/w4pui6/i_think_the_spaced_repetition_community_should_up/

The top commenter actually also brings up another very good point. Longer spacing = better retention, but more wrong answers, so it can be demotivating. This raises the question of what is more important, to feel good about yourself, or to actually learn (not memorize) the content. I am not the person to make that call, that is for each individual to decide.

To test whether the SRS is truly bad for you as an individual on Bunpro, I would recommend changing your review style to ‘reading’ temporarily. If you retain the information/can understand the meaning from sight much more consistently, then it is a sign that you’re probably rushing through the reviews without thinking about why the grammar is being used in a certain sentence in the cloze style.

That is correct. The only thing that we are cautious about here is the difference between memorization and learning. It is one thing to get something right more often. It is something else entirely get something right only because you memorized it.

Anyway I am not an expert on this, I just think that it’s not as simple as just changing it. Especially if there’s a possibility that the results for users ends up being something like a pain-killer (memorization) for the underlying problem (learning).

Disclaimer - This really only applies to grammar I would say. Vocab is a different thing where rote memorization is likely to be much more helpful. We will continue to seriously research this topic on our end and try to find the best solution for everyone.

13 Likes

Oh I edited my post while you were typing, I’ll repost here for the sake of clarity.:


Anyway, thank’s for elaborating. As for the grammar side, I think I assumed that the sentence-cycling part of Bunpro would kind of cancel the “memorizing instead of learning effect”. It already does that, to a certain point.

Adding more sentences would also reduce the chance of memorizing and help with practice. I mean, what’s another 5000, right? :upside_down_face:

6 Likes

I’d like to support this too. There are other improvements and innovations that the team could be working on instead. E.g. finding a new trick or study mode for us to finally conquer all those pesky quasi-synonyms. Or QoL for vocab (I’m not a user of vocab features yet though, so cannot say too much).

The topic has an aggressive title, and OP is very convinced about decades of research or whatnot, but in the end no studies with experiment setup relevant to Bunpro’s grammar points were provided. I don’t think we at the stage were all work has to be dropped immediately to fsrs.

4 Likes

Yes, let’s keep it civil or everybody loses when the subject is no longer considered seriously :sweat_smile:

Having not done super deep research, as far as I can tell FSRS does have a notable benefits w/ increased retention and less time spent on each card, and I think at the very least it should be a genuine contender for vocab SRS.

Also, tho it’s been mentioned before, the distinction between “recall” and “remind” and so on seems mostly semantic; whether you do reveal & grade or cloze fill, you are fundamentally recalling several things such as reading and meaning each time ᵘⁿˡᵉˢˢ ʸᵒᵘ ʳᵉᵛᵉᵃˡ ᵃⁿˢʷᵉʳˢ ʷ/ᵒ ᵏⁿᵒʷⁱⁿᵍ ᵗʰᵉᵐ

You could even implement a feature to export vocab and even full decks (incl. current progress, ideally) to Anki, so Bunpro users may continue to benefit from the many sample sentences and such they pay for. I can understand a reluctance on your part, but it’s worth considering.

2 Likes

I’m note sure it’s the case, but if you’re talking about this part :

The absence of clear distinction between production and reading abilities are sabotaging the whole vocab section.

It’s is not what I meant.
For exemple I’ve added なつかしい to my reviews a month ago or so. Ask me to produce “nostalgic”, no problem. If I hear it in a conversation, no problem.
And then yesterday the word 懐かしい showed up in another sentence and although I try to pay attention to the kanji while filling cloze questions, I could not read it at all.

That’s why I wish we had an option to have both types of questions mixed while reviewing vocab.
Maybe with a separate SRS if needed (not sure it is).
Without it, I would just level up なつかしい in cloze style, then I’d have to manually reset it to be tested the other way around ?

I feel all the content is there (up to N3 at least) and it would just be a matter of not overcomplicated implementation.

3 Likes

I was thinking of an earlier post about Bunpro being “output” and Anki being “input” or smth like that, which I think is entirely semantic and depends more on your personal approach if anything, regardless of platform.

What I meant by this

whether you do reveal & grade or cloze fill, you are fundamentally recalling several things such as reading and meaning each time

is that you’re always recalling at least one aspect of a word with either type, not that you get everything with either.

As to your point tho, that’s why I’m using reveal & grade for vocab and hide all furigana so I have to read the kanji and recall the meaning before revealing, but it would be a nice option to be quizzed on both for each vocab.
In fact I almost wrote that at the end of my last post but didn’t wanna bloat it more than I already had. xD

2 Likes

Slightly diverging, but I am all for mixed recognition + recall vocab (and even grammar) as well. They’re completely different types of learning and both are very important.

6 Likes

I like the SRS system. The early levels are there to see if you remember, and the later levels are there to see if you know. Especially with the safety system of being able to redo a point if you did actually know you just blanked and tried something else or typed it wrong.

But if you don’t know something then the excuse of “well I haven’t seen it in a year” doesn’t matter much. You just don’t know it. And if you are actively studying and exposing yourself and you still haven’t seen it in all that time, then who cares. Personally I don’t really care that much about 100% completing any of the items on bunpro.

Mind you I just use this and study as a hobby, if someone has grander aspirations or requirements their needs might not be comparable to mine.

Also I’ve always kind of thought that just one srs system for all the points isn’t a great idea. I kinda thought it should have multiple systems with multiple weights. But I won’t try to think of a better wheel when their are plenty of engineers out there who are gladly doing that for me already.

3 Likes

I want to add something to this. I just want to note that I had this feedback a loooong time ago, and was told that it wasn’t possible.

The idea of ghosts is good. I want to be able to practice things that I get incorrect. However, they are bad in practice. Instead of being drilled on the grammar, I’m drilled on the sentence I got wrong. This, in my mind, is not good. I find myself memorizing sentences instead of grammar. This has gotten even worse now for me - I took a break for a few months (only from Bunpro, not other study materials) and now I can barely remember anything. SRS as it stands is not working for me.

6 Likes

This is called “overfitting” and it means your brain adapted too well to the problem. Those memories will eventually be attached once you add more context. I completely see where you’re coming from in terms of Bunpro, but contrarily I think the problem is heavily mitigated by reading native books (or manga, etc) or chatting online.

2 Likes

I’d like to contribute a few notes to recent arguments:

  • For users like me using only recall (reading questions), Bunpro is really similar to other things I do on Anki, where I have to recognize the meaning of an expression, especially since so many grammar points are vocab points in disguise. In this context, I’m quite convinced that FSRS will be a huge improvement to the current algorithm
  • For users that use Bunpro the traditional way (typing the answer), FSRS is for me still applicable in the sense that FSRS is only a means to derive intervals given users’ learning patterns: the nature of the learning doesn’t matter too much as long as the usual decay observed in any retention-related activity is still observed. Given the nature of what we learn on Bunpro (nothing is conceptual, it’s basic conjugation and vocabulary), I don’t see anything too far away from the usual stuff that is learned on Anki. It happened to me a few times to get to a new point on Bunpro that corresponds to a word I mined during immersion: the frontier is extremely blurry.
  • Compared to right now where the intervals are the same whether you do recall, audio or generation, FSRS will very likely provide much more suitable intervals given the review type a user selects. E.g., when I switched to reading to save a lot of time during Bunpro reviews, my error rate dropped but the intervals remained the same, even though the questions are now easier because during recall I don’t have to tackle synonym hell or to get every detail right during generation, I just have to recall
  • People mentioning that at some point, for high intervals, you should just get exposition and learn through native material, are in my opinion missing the point of a SRS: if you did see the material a lot through exposition the reviews will be very easy, quick, and will take maybe a few seconds of your time 3 or 4 times before the review interval is 2 years+. SRS is there for the rest. I definitely get some very simple reviews wrong on Anki, even for words that are really common (like top 1.5k), even though I read every day and have 6k+ vocab cards by now. SRS helps me for them.
7 Likes

I’m happy to see this!

I imported my WaniKani vocab to Bunpro as Extra Practice* and I often have to use “Show Answer” during reviews because I can’t work out the intended response. Even so, I’m honestly stoked that I can recall multiple similar-but-not-what-was-being-asked-for words if even I don’t quite understand the correct context for them yet.

And when it comes to grammar, I’m again just excited that I remembered <submitted grammar point> is a thing, even if it isn’t the exact grammar point being asked for. Either way, I’m learning!

For vocab, I find remembering sentences, phrases or clauses is a good thing, as that means I’m essentially remembering the various use cases for the word. So… Bunpro is improving my ability to recall the word and often teaching me something new about it every time I jump an SRS level.

*I seriously question the wisdom of importing for Extra Practice when I was already at WK level 21 (~2k words, oh god), but there’s no going back now. :weary: On the other hand, I’ve reset so many words because I memorized but didn’t learn them through WK, to add to @Asher’s earlier point from several days ago. Maybe I can actually learn them now!

3 Likes

I was thinking about the cram feature but haven’t used it yet because wouldn’t that mess up the SRS? Because by then the computer would think that you haven’t seen the grammar point in a month while you actually saw it during your cram session earlier that week. So when you get the review right, it’ll give you an even bigger interval. But at that point you don’t actually know it and would be even more likely to get it wrong later

1 Like

well you are technically correct if you consider the SRS to be the core of your study but I think it shouldn’t be so it isn’t an issue at all in my opinion.
It is exactly the same as what would happen with exposure and of course you wouldn’t reduce it for the sake of bunpro’s SRS, the more you get the better.

The SRS is not so accurate that it should be taken religiously.
As far as I am concerned, I even adjust manually the SRS levels stepping down regularly if I feel a difficulty with a specific grammar point. I am fine with it my aim is not bunpro fast success but japanese solid learning. I’d just encourage to do so whenever you feel you may need more testing on a grammar point.

5 Likes

TLDR: I study by deck because it’s got some filters like cram and adjusts the SRS appropritly

So there are 3 ways to reschedual after cram

Bunpros’ don’t reschedual at all.
This is good if you want straight up extra practice.

jpdb: modified schedual, if you do a card early it is reschedualed to longer of schedualed due date and time elapsed between reviews
If you have a card due in a month, and you do it in a week, it will still be due 3 weeks after your early review

if you do a card late it is reschedualed
if you have a card the that was going to come back the next day, but you did it a week later, it’s reschedualed another week later, because appartly you can remember it for a whole week

(a slightly more complicated answer is “It uses 1.2* actual interval instead of the schedualed interval if you don’t do you reviews on time.” )

OLD anki (I’m still using 2.1.48, I heard in bunpro’s forums 2.3 has a different schedualer)
You can toggle between not reschedualling at all like bunpro or

Longest reschedual: push your revews foward by the same about of time as if you did them on time.
This is bad if you like to cram stuff you just did while it’s still fresh, your doubling the length of time between your reviews each time you cram

1 Like
  1. I agree I shouldn’t stress about SRS as much as I do. Ooops
  2. I’ve been studying Japanese for 11 years now the best part of SRS is it comes back eventually.
  3. Sure coming back to 3000 reviews is scary, but better than coming back to lingo deer with 300 lessons done, not sure which ones I still remember
  4. I’ve never wanted *more * reviews. I understand why you do, to get more practice. I don’t find reviews fun.
  1. This is the SRS readjusting. Probably you crammed a point because you thought you would get it wrong now. getting it wrong now or later is they same on net. You mess up the srs if you cram so regularly that you ‘never’ get a review wrong. SRS is built on the assumtion of 70-95% accuracy if you are outside that SRS can get out of sync.
1 Like

Hi there !

Unrelated to the recent discussion, but I’m curious about your experiments with FSRS and the vocab section. Any updates ? Could we expect any changes in the near future? And what about mixed production/recognition reviews? @Jake :thinking:

4 Likes

@Asher @Jake

I appreciate your thoughtful responses to @d11 's urging on moving to FSRS algo, and I understand your responsibility is to the entire community of users, which makes making core changes difficult to do, especially if they are potentially divisive.

Looks like the conversation broke down above when it turned into a bit of back-and-forth when asking for linked research and then whether or not that research was valid for this use case, but if I may, I’d like to bring it back up, and hopefully steer it in a more helpful direction.

To be clear, the only thing in question here is how the timing of the next review is set. The method of input/recall as compared to other decks/platforms, nor the fact that Bunpro uses multiple examples per “point” (card), shouldn’t distract from this. Those points matter, and are what makes Bunpro special, but the time-to-next-review question is the one that is being asked.

So, given the research and proven effectiveness of FSRS in similar contexts (of course there is no published research on different spacing algorithms as applied to specifically Bunpro – that would be rather absurd to expect), I wonder:

  • what is the fear / expected downside of using that algorithm? (Do you think it will hurt some users, and if so why)

And, given that the current algorithm is so basic (it just gets longer every time and you go back a step if you get it wrong, as far as I can tell):

  • what evidence/reasoning would there be to expect that this specific approach is more effective than FSRS?

And lastly, assuming it is not a technical limitation, since the platform is quite customizable already, and that Jake said he would have something available for beta testing after only a short time after OP landed on the forum:

  • Is there any reason not to implement FSRS and make it available behind a settings flag (even if only beta, or not set as default, so that power users can find it)?

Finally, just will interject my opinion into the mix. I think that a modified version of FSRS that does allow for “mastery” after enough correct answers is much better (or a button to manually mark it). I would argue against the OPs statement that “statistically, 10% of the time [you’ll forget]”, as that doesn’t take into account any real-world usage. I’d wager there are zero users on Bunpro who don’t use Japanese at all in daily life, reading, or media, unless they are tire-kickers in the first weeks of trying before giving up. It’s highly doubtful N4-N1 speakers are going to forget 10% of their N5 grammar points that they encounter every time they use Japanese outside of Bunpro.

7 Likes