Bunpro's bad SRS algorithm is discouraging

@Asher I’d like to suggest a YouTube video about an optimal SRS timing that is individual to each learner specifically, using something called FSRS inside of Anki!

Since I’ve switched to FSRS my retention has increased drastically. It will require a “bed-in” period for the values to adjust correctly as the system learns based on a multitude of factors. It’s only a suggestion, and I’d love to see an option for FSRS inside of BunPro one day!

6 Likes

Sure, here you go: Butler, A. (2010). Repeated Testing Produces Superior Transfer of Learning Relative to Repeated Studying. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36, 1118–1133.

Experiments 1a and 1b also explored Would spaced repetition memory systems perform better with varied question texts? with a varied-questions condition but found a null result.

This should provide some evidence for the hypothesis that Bunpro’s system is not doing anything so special that it somehow can ignore decades of memory science research. Which should be the default assumption anyway! If you have any science showing support for this surprising claim, I’d love to read it!

3 Likes

you kinda sound rude. for being constructive you sure have a rude way to go about it. real redditor energy there.

7 Likes

Here is a link to the paper itself; I just skimmed through it and the findings do not seem relevant at all to the long term learning of grammar and the use of SRS. The information being recalled consists of simple facts. The variability in question types is extremely simple. The spacing and testing is also over a very short period (a week). The students have no relationship or context for the information outside of the test (which people studying languages would have). I don’t think any useful conclusions about language learning and SRS can be drawn from this study, personally. The authors of the paper are also not interested in that and do not draw any conclusions about it themselves. The main finding is that active recall (contextualised testing of knowledge/ability) is more effective for learning than simple studying with no active recall (testing). This is already extremely well established and the basis of why people use active recall based SRS to begin with.

Of papers I have read myself, I have never seen a good long term study on SRS and language learning. Anything close is focused on a relatively short period of time (months, not years), focuses on simple vocab lists of only a hundred or so words, and does not take into account all the ways students learn languages outside of the SRS itself.

If you know of any solid long term research about SRS and grammar acquisition of foreign langauges, especially English speakers acquiring Japanese outside of classroom environments, then I would love to read about it.

12 Likes

It seems someone beat me to replying while I was reading it, but yeah, the full paper here seems to convey quite different results to what you’re claiming.

https://www.researchgate.net/publication/46094320_Repeated_Testing_Produces_Superior_Transfer_of_Learning_Relative_to_Repeated_Studying

The retrieval method that they used is quite different from what we do here at Bunpro. Also the whole experiment happened over the space of a week, so there is no real indication of any long term retention being validated.

I don’t think that this conversation can go anywhere constructive if it’s just a back and forth of competing for being right. It’s not about who is right, it’s about finding what is right.

17 Likes

Ahh…I was really enjoying reading this thread. Anything that would improve BunPro’s SRS algorithm would be great. We even saw the author of FSRS chime in here and everything. Hopefully it doesn’t go down the drain.

9 Likes

I am sometimes discouraged, but I don’t think it’s the timing that causes it.
When my reviews are low it’s usually

  1. interference, I get similar grammar points mixed up
  2. I do a bunch of really overdue reviews

Jake’s fixes would help with the second.

For the first, I don’t think the SRS is the solution. If something isn’t sinking in, you do need some kind of outside study.

Bunpro has lot’s of good info on the grammar point page. I think some kind of automatic redirect to the grammar point for troubled grammar would be useful, but might feel condescending or intrusive.

I think if you pulled anki’s SRS algorithm and put it directly into bunpro, you would be equally discouraged because bunpro reviews are harder than anki reviews. And it doesn’t fix problem 1.

I think ‘troubled grammar’ is the same as a leech.

If you want to fix this, your ‘learning steps’ and ‘relearning steps’ don’t count as lapses. So I looked up a table and hard coded in all the beginner steps as learning steps. (15m, 1d, 3d, 7d, 16d) going straight into ‘mature’ at 30d

You can also pick any number.

2 Likes

I have to admit, I don’t quite get it either. Also I might be mistaken, but I feel there’s a misunderstanding somewhere in this conversation.

How would this makes a difference from an SRS to another ?
No rhetoric intended, I’m really asking. To me, the algorithm would consider a single grammar point as the thing to learn, to reproduce if you want, cycling through all associated sentences, just as it is now. I mean switching to FSRS would only adjust the timing a grammar point (not a single sentence) is presented to an individual, depending on is ability to reproduce it. Nothin less, nothing more.


2 Likes

One of the main differences is in the goal. The goal of Bunpro is to learn, not to memorize. For the sake of this explanation, I’ll just give these two terms a quick description of how I am using them.

To learn - To understand and then be able to apply knowledge in unfamiliar situations.
To memorize - To be able to recite previously seen information, regardless of understanding.

If the sentence presented is always the same (as with most cases in Anki), one is able to memorize the sentence, which is a big tip-off as to what the grammar is going to be. We actually have people mentioning this as a negative thing quite frequently, especially when it comes to ghosts. It’s the ‘I feel like I am just memorizing sentences, not actually understanding why to use certain grammar’ effect. With Anki it is the same. If you just memorize the sentence, yes you will get the question correct more often, but have you actually learned anything? No, you have memorized it.

In Bunpro, you are shown a new sentence at each new SRS level. This takes away the possibility of memorizing the answer, and forces critical analysis of the sentence in order to retrieve the information. This cannot be done unless something is actually learned, due to the things needed to take into consideration about the question. For example, who is talking to who, what is the style of writing, are there any relevant conjugations before/after that will eliminate some possibilities, does the sentence have a certain level of politeness, etc etc. All of these analysis steps become completely irrelevant if you see ‘昨日、図書館で借りた本を------。’ and then think to yourself ‘Oh I remember this sentence, the answer is XYZ’.

This Reddit post (and the accompanying scientific article is a good discussion on the topic. https://www.reddit.com/r/Anki/comments/w4pui6/i_think_the_spaced_repetition_community_should_up/

The top commenter actually also brings up another very good point. Longer spacing = better retention, but more wrong answers, so it can be demotivating. This raises the question of what is more important, to feel good about yourself, or to actually learn (not memorize) the content. I am not the person to make that call, that is for each individual to decide.

To test whether the SRS is truly bad for you as an individual on Bunpro, I would recommend changing your review style to ‘reading’ temporarily. If you retain the information/can understand the meaning from sight much more consistently, then it is a sign that you’re probably rushing through the reviews without thinking about why the grammar is being used in a certain sentence in the cloze style.

That is correct. The only thing that we are cautious about here is the difference between memorization and learning. It is one thing to get something right more often. It is something else entirely get something right only because you memorized it.

Anyway I am not an expert on this, I just think that it’s not as simple as just changing it. Especially if there’s a possibility that the results for users ends up being something like a pain-killer (memorization) for the underlying problem (learning).

Disclaimer - This really only applies to grammar I would say. Vocab is a different thing where rote memorization is likely to be much more helpful. We will continue to seriously research this topic on our end and try to find the best solution for everyone.

14 Likes

Oh I edited my post while you were typing, I’ll repost here for the sake of clarity.:


Anyway, thank’s for elaborating. As for the grammar side, I think I assumed that the sentence-cycling part of Bunpro would kind of cancel the “memorizing instead of learning effect”. It already does that, to a certain point.

Adding more sentences would also reduce the chance of memorizing and help with practice. I mean, what’s another 5000, right? :upside_down_face:

6 Likes

I’d like to support this too. There are other improvements and innovations that the team could be working on instead. E.g. finding a new trick or study mode for us to finally conquer all those pesky quasi-synonyms. Or QoL for vocab (I’m not a user of vocab features yet though, so cannot say too much).

The topic has an aggressive title, and OP is very convinced about decades of research or whatnot, but in the end no studies with experiment setup relevant to Bunpro’s grammar points were provided. I don’t think we at the stage were all work has to be dropped immediately to fsrs.

4 Likes

Yes, let’s keep it civil or everybody loses when the subject is no longer considered seriously :sweat_smile:

Having not done super deep research, as far as I can tell FSRS does have a notable benefits w/ increased retention and less time spent on each card, and I think at the very least it should be a genuine contender for vocab SRS.

Also, tho it’s been mentioned before, the distinction between “recall” and “remind” and so on seems mostly semantic; whether you do reveal & grade or cloze fill, you are fundamentally recalling several things such as reading and meaning each time ᵘⁿˡᵉˢˢ ʸᵒᵘ ʳᵉᵛᵉᵃˡ ᵃⁿˢʷᵉʳˢ ʷ/ᵒ ᵏⁿᵒʷⁱⁿᵍ ᵗʰᵉᵐ

You could even implement a feature to export vocab and even full decks (incl. current progress, ideally) to Anki, so Bunpro users may continue to benefit from the many sample sentences and such they pay for. I can understand a reluctance on your part, but it’s worth considering.

2 Likes

I’m note sure it’s the case, but if you’re talking about this part :

The absence of clear distinction between production and reading abilities are sabotaging the whole vocab section.

It’s is not what I meant.
For exemple I’ve added なつかしい to my reviews a month ago or so. Ask me to produce “nostalgic”, no problem. If I hear it in a conversation, no problem.
And then yesterday the word 懐かしい showed up in another sentence and although I try to pay attention to the kanji while filling cloze questions, I could not read it at all.

That’s why I wish we had an option to have both types of questions mixed while reviewing vocab.
Maybe with a separate SRS if needed (not sure it is).
Without it, I would just level up なつかしい in cloze style, then I’d have to manually reset it to be tested the other way around ?

I feel all the content is there (up to N3 at least) and it would just be a matter of not overcomplicated implementation.

3 Likes

I was thinking of an earlier post about Bunpro being “output” and Anki being “input” or smth like that, which I think is entirely semantic and depends more on your personal approach if anything, regardless of platform.

What I meant by this

whether you do reveal & grade or cloze fill, you are fundamentally recalling several things such as reading and meaning each time

is that you’re always recalling at least one aspect of a word with either type, not that you get everything with either.

As to your point tho, that’s why I’m using reveal & grade for vocab and hide all furigana so I have to read the kanji and recall the meaning before revealing, but it would be a nice option to be quizzed on both for each vocab.
In fact I almost wrote that at the end of my last post but didn’t wanna bloat it more than I already had. xD

2 Likes

Slightly diverging, but I am all for mixed recognition + recall vocab (and even grammar) as well. They’re completely different types of learning and both are very important.

6 Likes

I like the SRS system. The early levels are there to see if you remember, and the later levels are there to see if you know. Especially with the safety system of being able to redo a point if you did actually know you just blanked and tried something else or typed it wrong.

But if you don’t know something then the excuse of “well I haven’t seen it in a year” doesn’t matter much. You just don’t know it. And if you are actively studying and exposing yourself and you still haven’t seen it in all that time, then who cares. Personally I don’t really care that much about 100% completing any of the items on bunpro.

Mind you I just use this and study as a hobby, if someone has grander aspirations or requirements their needs might not be comparable to mine.

Also I’ve always kind of thought that just one srs system for all the points isn’t a great idea. I kinda thought it should have multiple systems with multiple weights. But I won’t try to think of a better wheel when their are plenty of engineers out there who are gladly doing that for me already.

3 Likes

I want to add something to this. I just want to note that I had this feedback a loooong time ago, and was told that it wasn’t possible.

The idea of ghosts is good. I want to be able to practice things that I get incorrect. However, they are bad in practice. Instead of being drilled on the grammar, I’m drilled on the sentence I got wrong. This, in my mind, is not good. I find myself memorizing sentences instead of grammar. This has gotten even worse now for me - I took a break for a few months (only from Bunpro, not other study materials) and now I can barely remember anything. SRS as it stands is not working for me.

6 Likes

This is called “overfitting” and it means your brain adapted too well to the problem. Those memories will eventually be attached once you add more context. I completely see where you’re coming from in terms of Bunpro, but contrarily I think the problem is heavily mitigated by reading native books (or manga, etc) or chatting online.

2 Likes

I’d like to contribute a few notes to recent arguments:

  • For users like me using only recall (reading questions), Bunpro is really similar to other things I do on Anki, where I have to recognize the meaning of an expression, especially since so many grammar points are vocab points in disguise. In this context, I’m quite convinced that FSRS will be a huge improvement to the current algorithm
  • For users that use Bunpro the traditional way (typing the answer), FSRS is for me still applicable in the sense that FSRS is only a means to derive intervals given users’ learning patterns: the nature of the learning doesn’t matter too much as long as the usual decay observed in any retention-related activity is still observed. Given the nature of what we learn on Bunpro (nothing is conceptual, it’s basic conjugation and vocabulary), I don’t see anything too far away from the usual stuff that is learned on Anki. It happened to me a few times to get to a new point on Bunpro that corresponds to a word I mined during immersion: the frontier is extremely blurry.
  • Compared to right now where the intervals are the same whether you do recall, audio or generation, FSRS will very likely provide much more suitable intervals given the review type a user selects. E.g., when I switched to reading to save a lot of time during Bunpro reviews, my error rate dropped but the intervals remained the same, even though the questions are now easier because during recall I don’t have to tackle synonym hell or to get every detail right during generation, I just have to recall
  • People mentioning that at some point, for high intervals, you should just get exposition and learn through native material, are in my opinion missing the point of a SRS: if you did see the material a lot through exposition the reviews will be very easy, quick, and will take maybe a few seconds of your time 3 or 4 times before the review interval is 2 years+. SRS is there for the rest. I definitely get some very simple reviews wrong on Anki, even for words that are really common (like top 1.5k), even though I read every day and have 6k+ vocab cards by now. SRS helps me for them.
7 Likes

I’m happy to see this!

I imported my WaniKani vocab to Bunpro as Extra Practice* and I often have to use “Show Answer” during reviews because I can’t work out the intended response. Even so, I’m honestly stoked that I can recall multiple similar-but-not-what-was-being-asked-for words if even I don’t quite understand the correct context for them yet.

And when it comes to grammar, I’m again just excited that I remembered <submitted grammar point> is a thing, even if it isn’t the exact grammar point being asked for. Either way, I’m learning!

For vocab, I find remembering sentences, phrases or clauses is a good thing, as that means I’m essentially remembering the various use cases for the word. So… Bunpro is improving my ability to recall the word and often teaching me something new about it every time I jump an SRS level.

*I seriously question the wisdom of importing for Extra Practice when I was already at WK level 21 (~2k words, oh god), but there’s no going back now. :weary: On the other hand, I’ve reset so many words because I memorized but didn’t learn them through WK, to add to @Asher’s earlier point from several days ago. Maybe I can actually learn them now!

3 Likes