Community Decks - An Analysis

Edit: changed title from “Postmortem” to something less dramatic sounding.

Today it has been one month since the official release of the Community Decks feature on Bunpro. It has been available longer in beta (about nine months ago?), but I think now’s the perfect time to see how the community is responding and reacting to this feature.

As of this writing, there are currently 119,232 users on Bunpro. This is found by looking this page and adding all of the users. I tested after making an account, it automatically gets added to the page.

That’s a lot of users! But how many of them are active subs? This is where things get tricky as I don’t have the data, but I can speculate. Edtech apps usually hover between 20-50% so that would put the active sub count between 24k-60k users. That’s a huge gap!

Sources, as this shows 27% retention and this one has it as low as 2.1%! However this source shows 30-day retention rates between 27-43% and higher-performing apps at 32-66%

Boy, that’s not helpful at all! But let’s stick with 20-50%, if Bunpro would share their numbers, this would of course paint a more accurate picture.

Let’s dive into the decks!

Lookin’ pretty good eh?

I manually counted 155 community decks at the time of writing this, so I will use that for the epic math crunch later.

But let’s break things down. Those numbers in that image from earlier? Looking pretty impressive…but that’s misleading. Let’s go into the numbers.

There is one deck that has 500+ users. I assume at minimum 500 users. I think the range is between 500-999 users?

Congrats…@asher…wait…decks from Bunpro staff counts as a community deck? Okay. That’s cool. But what about the others?

There are 6 decks that have 100+ users, again assuming minimum 100 users and a max of 499.

Uh, congrats again @Asher! You must be quite the popular man.

There are 11 decks that have 50+ users. 50 minimum and a max of 99 users. Don’t worry I won’t post another picture! Let’s just go through the rest of this!

57 decks with 10+ users. 10 minimum and a max of 49 users.

And lastly… 71 decks, about half of all community decks with less than 10 users. SAD! Side note, I found 9 decks with nothing mentioned. My assumption that it’s 0 users, so I will add it to the <10 side making it 80 total.

What does this mean?

Let’s do some number crunching!

Category Number of Decks Min Users per Deck Max Users per Deck Min % (24k Active) Max % (24k Active) Min % (60k Active) Max % (60k Active)
500+ users 1 500 999 2.08% 4.16% 0.83% 1.67%
100+ users 6 100 499 0.42% 2.08% 0.17% 0.83%
50+ users 11 50 99 0.21% 0.41% 0.08% 0.17%
10+ users 57 10 49 0.04% 0.20% 0.02% 0.08%
<10 users 80 0 9 0.00% 0.04% 0.00% 0.02%

What a lovely table, eh? I hope I did the math right!

Wow that is a long tail. Half of all decks have between 0.00-0.04% of active users. Even lower if we account all users on the platform!

Slightly, and I do mean slightly better with the ~55 decks with 10+ users. But we have about 135 or so decks with less than half of one percent of all active users using this function.

However, on the bright side, I think it’s admirable that there are users who spent time out of their day to make these decks with the intent on helping out fellow learners. Not to mention one deck has between 0.83-4.16% of active users on it. Pretty impressive!

Overall, if I was Bunpro, I would pay close attention to the adoption rates of these decks and see how much value they are providing to users. The data…isn’t encouraging, but with tweaking and curating, it could move the needle in a positive direction.

Perhaps consider weekly spotlights to highlight some users or reaching out to deck authors directly to get feedback on the process of making them.

It’s an idea that can work, but must be tread carefully. My instincts tell me that Bunpro-curated decks are the way to go and to axe the community feature entirely, but I would like to be proven wrong!

4 Likes

I use decks without adding them. Reason is their rigidity. If a deck has lots of A/E words without sentences, I don’t want them in my queue the same way I want N5-1 A1-2 words with sentences.

So I never add a community deck to queue. Instead I bookmark them and add some terms to my queue by selecting them.

So on the stats it seems like I’m largely ignoring this feature. I’m not though.

Long story short, the publicly available data points aren’t enough to estimate usage.

10 Likes

I think you bring up an interesting point about the shortcomings of this observation. I feel like this would be more of a fringe case and I wonder how many actually use that when accounting for all active users. Maybe it’s not as fringe as I claim and I am wrong. Boy, that would be embarrassing.

The spread of 24k-60k active users shows how hard it is to paint a full, accurate picture, and may give one explanation why there is such a long tail distribution for the data.

Overall, I think publicly available data points are good enough to paint a “rough” estimate to gauge if this feature is working or not. There would need to be a suite of metric tracking tools Bunpro could implement to be thorough.

Actually, this brings up a valid UX situation. If you added a Community Deck to your queue, and you had the option to drop words in that deck, would you use the feature?

From my experience, there’s many decks with weird word choices that I would prefer to skip. This might be worth seriously looking into. :thinking:

1 Like

I don’t see the point in removing a feature, however niche, when it’s already been made and works. Worst case, just leave it for those of us who do use it.

4 Likes

I don’t think that is the point he is trying to make. It makes no sense to remove something that is already there. A postmortem is usually done after a big mistake, failure, error, or crash, at least in the dev world. So he is saying the feature failed and suggests some improvements, but not many. I still don’t understand why he did this analysis, though. First of all, the feature was in beta and was recently released, hasn’t been marketed heavily, and there might be some version 2 features in the work that we don’t know of.

I think overall the feature is not bad, but definitely needs some additional tools, like vocab within sentences, and I don’t know, some Anki users probably have a better picture of what you could include here

2 Likes

I know that wasn’t the only point they made, but it was their final one and I just wanted to chime in that even if someone may think it’s a pointless feature, there are still those using it and Bunpro going out of their way to remove something that’s already there and working seems needlessly destructive.

As a feature it can certainly be improved, but then again that’s true of most things.

Edit: Imagine if hitting the reply button would make my post a reply to the post to which I was trying to reply xD @mersadajan

2 Likes

I feel like one point to make is that the people who are active on bunpro probably already have a lot on their SRS plate. I know I’m already spread pretty thin with wanikani,anki and bunpro. For now it’s probably the case that people start using bunpro when they are already regular anki users or after being regular wanikani users and anki users (my case). I feel like it’s definitely a useful feature for those who don’t want to use anki, which is not most bunpro users.

6 Likes

Same here. I like fill-in exercises so I would use community decks more if there was an easy way of ignoring unwanted words (like A/E) without marking them as mastered or so. This is probably the only reason why official Bunpro decks remain my main ones.

Or if you could treat A/E words with self study sentences as if they were official sentences. Now I have sentence less words with self study sentences in my queue but never added the actual word.

Or if you could have words without sentences show up in a English → recall Japanese mode since I want to practice recall and not recognition

1 Like

Oh yeah, having such fallback scenarios like “if possible use this review type, else if possible use that, and otherwise that” would be so ideal I didn’t even think about mentioning it. Same as treating self study sentences like official ones. Both your ideas are something I’d love to have on Bunpro

It’s an interesting point I see often. If you look at Bunpro’s Twitter account, you will see they post frequently, but get super low engagement. Some posts get 1-2 likes and less than 100 views. That would make it super niche, right?

Would it be in the best interest of the company and the users they serve to continue spending money on making these posts despite the super low engagement? This is a problem I have encountered a ton when working at a startup. Usually you change strategies or decide to pull out entirely and attempt to provide value in another aspect.

At least in the realm of social media, there’s a huge amount of opportunity they can tap into, so drastically changing strategy could provide long-term success.

I think cutting features shouldn’t be done without deep consideration. Not many users pay attention to N1 grammar points. Should that be cut? Of course not! But maybe the question should be “how do you get more users to access N1 grammar points?” Similar logic can be applied to this scenario.

A community showcase per week or tighter curation could help fix the lack of engagement with Community Decks. Or maybe just better UX with the ability to drop/ignore A/E words in decks.

In the context of this, I am looking at “postmortem” from a perspective of critique, not implying failure. Edit: Upon further reflection, I decided to change the title of the thread.

The feature has been on the site formally for a month, and longer than that in beta. I think it’s absolutely fair to dig into it. Choosing to not market it heavily, if Bunpro decided upon that is worthy for discussion.

I share that opinion as well and the intent behind my post was to pivot the conversation around improving the service. My instinct is to cut things, but that should be done as an absolute last resort!

If I was Bunpro, I would try to see what is the most efficient method of learning vocab. You can try to see what are some really efficient Anki decks from users who crossed the fluency line, hire educators as advisors, whatever it is to get as much “truth” on your side. Then make adjustments to learning vocab on Bunpro to reflect those changes but in an easy to use and engaging manner. (Easier said than done :upside_down_face:)

That’s a very insightful point! Vocab was a very late addition to Bunpro. There’s already prominent SRS-based vocab apps that it can feel like just something to ignore if they already have a system going.

In reality, it needs to take a lot of convincing to get Anki/Wanikani users to move over to the Bunpro system permanently. That’s a huge challenge and it seems to me, that Bunpro’s strat is to lean towards Anki. I don’t agree with that pivot.

I think the best short-term strategy they can take is to make the system as easy and engaging for absolute beginners for them to stay, and then refine the tool to be overwhelmingly good to get the Wanikani/Anki users to convert. You already see this attempt as you can import Wanikani progress and Anki cards. But so far, the implementation is…a half measure.

2 Likes

I PLAN to use the decks, once I reach 50-60% vocab organically (from core decks). I’d love to use them to help jump start my listening practice. But since I can’t learn everything at once, I’m focusing on the core vocab first.

It seems like too much effort not enough reward to grind some of them up from 10%, when I could be learning more general purpose words.

But if I could start them from 60-70%? To then walk in to a show knowing all the vocab? That seems awesome!

Also hitting the refresh button and seeing how much closer I have become to being able to watch a particular show for listening practice is exciting.

7 Likes

I feel like the community decks aren’t meant to be a mainstream feature or heavily promoted. To me they seem to mostly be an advanced feature for people who would like to learn more specific things. Most people will probably just go through the basic decks instead and maybe later down their journey switch to community decks if they want more specific vocab, like if they have a certain interest or use certain other learning materials.
Personally, I don’t see a need to push community decks down people’s throats. If people are looking for them, they should be able to find them, but as others have said, most people already have plenty to do the way things are.
Personally, I only use them a little bit but I also love being able to refresh my percentage on decks and see how far I’ve come :slight_smile:
As others have mentioned, I’d also be happy if there was an option to only get the “good” vocab from decks, i.e. vocab with example sentences, etc.

4 Likes

I think good questions to ask are: What is the value per user this feature will give? Is this value per user high enough to invest time and resources developing and maintaining it?

Look at this deck. 5850 words for this deck. That’s a lot! Realistically, how many people will go through this and learn the entire deck? Well, less than ten based on what we see right now, but out of those, how many?

Is there value in having this? If so, how much?

Taking a look into it, this user got the game’s script, converted it to a CSV and then uploaded it. By the way, I want to praise this user for taking time out of their day to even do this, and this is clearly a selfless and caring move on their end.

Nearly 6000 words is…well, frankly a lot! A significant portion of being fluent in vocabulary just in this deck.

If I were to do this 100 times for 100 different games, would this be valuable? Maybe I do this for 1000 games?

I am not posing these questions in a judgmental or rhetorical sense, but to overall try to understand the intent behind this feature and in a roundabout way explaining a point. The point in being this: each feature, no matter how small needs to be considered on how it works with the whole. (On a separate note, I have seen this repeatedly in multiple failures in proper UX on this site, so I feel like this is part of a deeper trend that must be explored.)

If this isn’t mainstream, then why implement it if they are already churning new users heavily? When looking at the bigger picture, it seems confusing and I have to ask why.

I must apologize, as most of this message was targeted towards the staff (as they are most likely reading this), but I thought your comment was an appropriate avenue to go on a bit of a tangent.

To be honest I don’t understand what the issue seems to be.

The most successful SRS platform in the world has custom decks, and the ability to share custom deck. It is called Anki.
Several other commercial platforms (I don’t want to advertise specific names here) have custom decks related to anime/manga/etc too.

Yes, I personally believe it is valuable to have a custom deck for a piece of content I’m reading at the moment. But I’m not included in that statistic because I already have a different workflow for vocabulary.

Custom decks are (obviously?) the default way to learn anything outside of school-mandated tests.
Sharing custom decks is (obviously?) a great way to engage community and save staff effort.

If there is another feature where you’d like development effort to be spent, why not just advocate for that specific feature in the feature suggestion thread?

2 Likes

I don’t use decks because of what you’re saying. That’s indeed why I made a couple of custom decks that filter out vocabs that are not “Bunpro-mature” yet.

1 Like

I understand this post from a developer/“start-up” POV but I don’t think it takes into consideration the language learning market overall.

Out of the general population, a fraction is actively learning a foreign language. Of that fraction, I’d estimate the vast majority are children undergoing general (school) or secondary (university) education. Of adults that learn a foreign language, I’d assume most are doing it in a classroom setting and aren’t going out of their way to find online resources.

Of those that are looking for online resources, regardless of the language, if you want a free resource, Anki is king. Anki is also both free and community-driven.

Being a paid resource, for a specific language (Japanese), targeted at English speakers, that essentially does what Anki does “but better”, I’d imagine is a hard sell for most people.

WaniKani took its leading position in the Japanese language learning market by being quite specifically about learning kanji. One could say WaniKani has a monopoly on that niche. That’s why it doesn’t really make sense for BunPro to “convert” WaniKani users.
I’ve started out as a WaniKani user and got into BunPro back when it was still only about grammar, because back then I’ve had kanji covered, I’ve had vocabulary covered (Anki, Torii SRS), and this was basically the one missing element.

When BunPro added vocabulary to its system, I largely stopped using Torii SRS as BunPro basically just did the same thing but better. Targeting Anki users for conversion makes sense because BunPro already has the infrastructure for doing it and mostly just has a problem of quantity of quality content that’s slowly but surely getting resolved over time.

The addition of community decks seems to me like a move in the right direction when it comes to trying to get Anki users to come over. Why isn’t it being marketed currently? If we are to assume a general lack of marketing is not the problem and that waiting is a deliberate choice, it’s probably because marketing BunPro Community Decks when a lot of the vocabulary works “like Anki but less customizable” is bad optics. BunPro has to resolve the content issue first.

As far as retention itself goes I don’t think it’s a BunPro issue as much as it is a “Japanese language retention” issue, as seen from the threads where people are getting oneshot by the language and end up with mental health issues or calculate that learning at a sustainable to them pace means it will take them over a decade to be fluent in Japanese and just drop it entirely.

As someone who uses Community Decks basically any time I’m on BunPro, here’s my take on the situation:

  1. The vocabulary in official Vocab decks needs to be fixed first so there’s no instances of vocabulary without accompanying sentences or with non-ideal sentence translations (examples where the entire sentences is translated but then there’s the meanings of the word in brackets after it).

  2. Core 2K/4K decks should be given the next priority. This is quite a popular category on Anki and the only reason why it’s at a 100+ members currently despite being made by a staff member is that it’s still counted as a community deck and still largely suffers from the aforementioned problem of missing sentences to go with them.

  3. There desperately needs to be an ability to filter content in decks by their readiness or whether they appear in official N5 to N1 decks or not. A setting to include only Nx items, also include Ax items, or include everything.
    Doing it by how “ready” an item is might be harder if BunPro doesn’t have an existing flag on those items in their database as it’s a lot of manual work to put such a flag in place at this point.
    A filter such as “only include Nx items” however should, even with the assumption that the codebase MIGHT be an absolute mess, take no more than a work week.

If anything, item 3. is probably the cheapest and fastest way to make Community Decks more viable.

1 Like

Decided to inspect the website to see if there isn’t an API that can be leveraged to achieve this before waiting for them to implement it and it looks like they generate their components server-side.

What is this even, React Server Components? Is this Next.js or something?

Same except I’m not as organized and I just have my “temp deck” where I dump stuff into.

Until I made myself an “addon” (= a grease monkey script) where I can just cherry pick words from any deck or search result to generate a learn queue on the go. Saves me a bunch of clicks and page loads per vocab item compared to a temp deck.



I will post it on the forums once it doesn’t have some annoying bugs and doesn’t look like garbage anymore.

Before my holiday I also started making an addon that just takes a deck and mimicks the queue function without having to cherry pick but skips all words without sentences. But that doesn’t work yet.

2 Likes