Feature Discussion: Bunpro AI Assistant

[PowerAWBS]

I’m looking on Kanshudo’s website, I learned about it from this thread.

This would be an innovative feature for Bunpro, but the use case I was thinking on how AI intersects with Bunpro is that for users who fall off the wagon, the AI can adjust, or recommend like “maybe you’re going too fast, and then like chatgpt, rank those suggestions as helpful or not helpful”

Bunpro needing better onboarding is something that I have been thinking about and others have mentioned it too, so this would be one tool to help retain users at Bunpro.

What do you think?

[airbus29]

I personally have found the grammar points to be pretty self explanatory, but i think it would be neat if on each grammar point there was an AI assistant that was fed the text of the grammar point and the example sentences, and then you could ask the assistant itself any questions you have regarding the grammar point.

idk if i would use it tho and if it would be worth the time it would take to implement. I think that implementing ai just for the sake of implementing ai isnt really super productive. If there was a way ai could be used to help grammar acquisition that would be cool tho.

[EdBunpro]

An updated onboarding is coming this year, that will have better explanations for main functions of the dashboard, review page and lesson page. Nothing set in stone by any means. We’ve made adjustments in the past to give guidance on how many hours of studying per day correlates to X points learned per day. The big issue is that a lot of people coming from WK and Anki for vocab reviews still treat grammar like vocab (despite previously mentioned onboarding warning) and then some obvious frustration ensues. I don’t mean that as a knock against them by any means, it’s still on us to tinker things to avoid this in the first place. Users should never feel frustrated by the product at any stage.

Also included in onboarding discussions has been improved warnings that occur in specific instances within first week - two weeks, improved forecasting, and some small items I can’t remember right now.

Regarding AI though I personally think that since grammar is such an ever moving complexity from point to point, it may not be effective. Some points are frankly a lot more difficult than others and I don’t know if AI could take that into consideration. At least, not at its current state. I still believe that if we do onboarding properly, the user will know after a week or so what the outcome will be based on how often they are adding points.

Regardless of application used, users will usually go through periods where they go “okay I’ve added too much recently, let’s slow it down a couple days till I get back on track.” I know people will be able to figure that out when the time comes, it’s just making sure that new users are able to navigate that first-time hurdle when it pops up.

Sorry for the scattered post my brains all over the place while I cook dinner lmao. Good discussion nonetheless that I’d be interested in hearing from others what their initial experience was like in terms of pacing. We’re excited to improve the new user experience in 2024!

[PowerAWBS]

Thank you for the in-depth and quick reply!

Per the onboarding, perhaps I missed where it was discussed in the forums but basically you’d advocate instead of just going thru wanikani reviews like no tomorrow, needing to slow down for Bunpro, maybe even just a couple of grammar points a week?

Onboarding is one thing, but like re-onboarding, like users who’ve used Bunpro in the past but then return, especially lifetime members, like maybe it’d help to add like certain use case scenarios to the FAQ, like “i’ve fallen off the wagon, how do i get back on?”

I’ve been posting in my study log thread trying to figure out how to make Bunpro work for me, and maybe I’m getting the hang of it, needing to re-read the grammar point summary blurb + resource links to remember my rusty N4 grammar that are basically mostly turning into ghosts/leeches at this point.

To some extent I think that asking people for help is necessary and I don’t expect learning Japanese to continue to be a smooth journey, there’ll be times I get stuck, but yea re-onboarding (or whatever term to use) has been honestly a pain.

I already think Bunpro does so much better than Wanikani when it comes to features (i don’t need a bazillion scripts that might break, it just comes vanilla) – and Wanikani’s workflow is kinda straightforward – “let the mnemonics shock you and you try to recall, and then remember them”

• but Bunpro’s workflow, now that I’m thinking about it, maybe sounds more like “… just read the resources / the textbook you’re following, and then use Bunpro as a primer”

Example: In the gacha game Genshin Impact, returning users who have not been active in the past 6 months receive a “returning banner for the game.” (the game calls it Stellar Reunion)

they get extra in-game currency (I guess Bunpro could adjust this to like exp) and get some exp for re-acquainting themselves with features of the game, like
ex. (do one review, and get bonus exp, and here’s a hyperlink)
ex. (do one cram session, get bonus link, here’s a hyperlink on the banner to literally take you there)

You even get exp for logging in.

I understand that Bunpro to some extent is gamified with the exp system, and the financial incentives are different: Genshin Impact is powered by the micro-transactions that players purchase. So maybe there’s little to no financial incentive to provide this for the main benefit of lifetime members, but from what I understand Bunpro isn’t a company solely focused on financial profit.

I acknowledge this post is not organized as well as it could’ve been, please do let me know your thoughts~

[wct]

@PowerAWBS

the use case I was thinking on how AI intersects with Bunpro is that for users who fall off the wagon, the AI can adjust, or recommend like “maybe you’re going too fast,

I was thinking about this issue, as well. Not just ‘maximizing efficiency’ all the time, but also taking account of ‘psychological’ aspects of learning – especially such a sustained, ongoing project such as learning a whole language like Japanese – such as ‘energy level’, motivation, enjoyment/fun/entertainment, focus level (in particular for folks who might have issues, like how I have ADHD, for example), daily routine, weekly/monthly scheduling, planned vacations, etc. etc.

I mean, it doesn’t have to start out handling all those various factors. It could start out as something relatively simple. But ‘the future of learning apps’ will probably end up dealing with this ‘psychological side’ of things one way or another.

@PowerAWBS

Bunpro needing better onboarding is something that I have been thinking about and others have mentioned it too, so this would be one tool to help retain users at Bunpro.

Definitely agree with this. The BP team definitely has the right perspective/attitude, in the sense that I’m sure they’d agree with us that these are areas that could use improvement. But seeing the potential value in something, compared with having a strong strategy for accomplishing that thing, can often be two quite different things. Seeing a competitor like this with such a (seemingly) comprehensive on-boarding system could perhaps serve to spur the BP team to explore developing their own more-comprehensive onboarding system/strategy. (I generally see competition as a good thing and an opportunity for innovation.)

[nminer]

I’ll be honest, one of the things I love about Bunpro is that it’s not gamified and there aren’t a bunch of extra features bogging it down. I’ve tried using other apps like Duolingo and have taken a peek at others like MaruMori and actually felt more unmotivated to study after using their systems. I think for me at least, studying Japanese in and of itself is its own reward, so I don’t need an AI popping up every time I do my reviews to congratulate me (or make me feel guilty for not doing them).

I’m definitely in favour of having a better onboarding, and perhaps some kind of reboarding, although I think I’d prefer if this was an optional thing people could dismiss if they aren’t interested in it.

(P.S. Not going to lie, I know AI helpers have come a long way, but I couldn’t help but picture Clippy jumping around when reading this thread )

[Jake]

Just to chime in on the AI aspect of the discussion within the past 6-12 months, the team has discussed various ideas we had on how we might add/incorporate ChatGPT in a way that isn’t just a gimmick but actually adds something of value to the site. Two of the ideas that we felt had promise were:

1. Related to what @airbus29 mentioned, we had discussed a ChatGPT integration for reviews where after you gave your answer, assuming it wasn’t one we accepted or one we anticipated and gave a warning for, you could click a button to have the AI access your answer and give some feedback.

   Here were a few examples:
   

   

   

   

   In general, it gives a pretty decent but not always straight forward breakdown. I’m sure we could tweak the prompt to find the best option.

   Here is an example of where it is the problem (needing to use the verb stem) in less clearly explained:
   

2. The other idea we have explored is adding contextualization and dialogue to existing example sentences on the site with the goal being to give you a few different scenarios in which the sentence makes sense and show you what the question/answer/dialogue surrounding that sentence might look like. Here were a few examples:

   For the sentence: 田中さんは、まだ私の家にいます。
   

   

   For the sentence この店は美味しいですか。
   

   

   The challenge with this one is that based on our experience with ChatGPT, it is far from perfect when it comes to Japanese.

   At the end of the day, we wouldn’t ever want to put content for learning on the site that has wonky Japanese in it so all of these extra context sentences would need to be checked by the team first. That being said, we do feel like this implementation could offer a lot of value for learners.

[imsamuka]

I don’t know the considerations on the implementation of EXP on Bunpro, but they are directly related to the amount of work you’re putting into your learning. If I could gain it just by opening the site, it would feel even more meaningless than it already is.

EXP on games are important because you need it to grow and the more the better, but here, it’s a gimmick to make you feel more accomplished. Making easier to increase it, only decreases it’s value. At least on my opinion.

I also completely agree with @nminer. I would go further and say we don’t need to be overly addictive like duolingo-like apps, because this is only a means, and not the end itself. I’m already guilty myself of spending way too much time on Bunpro/JPDB/SRS/Forums instead of spending time on actually “using” the language. It’s way easier to be on controlled simulated environments than to actually adventure on the wild.

[wct]

Honestly, I think the ‘AI’ thing is a bit of a red herring with Kanshudo, and thus with a proposed Bunpro ‘assistant’.

Just my own take on things, but I don’t see a lot of market value (at this point in time, anyway) in having some advanced ChatGPT-esque AI thing. That’s far too complicated, IMO, far beyond the kind of thing I’m imagining.

What I’m thinking of is more along the lines of ‘machine learning’. In other words, a simpler, domain-specific model of ‘how people use Bunpro’ with the purpose of ‘learning’ things like: Which items (grammar, vocab, any other types of items yet to be implemented) are better to learn before/after other items? (In general, but also for a particular user.) What pacing of items is good for different kinds of users? What are a particular user’s strengths and weak points? Etc.

A lot of this could be modelled with something much simpler than a huge thing like a generalized language model like ChatGPT. Even just keeping a bunch of usage statistics and using them wisely to predict better user experience could probably go a long way.

[Sidgr]

This sounds like a reskinned AI Manager/ Storyteller analogous to rim world or left for dead. I am for that idea.

[nminer]

@Jake

The other idea we have explored is adding contextualization and dialogue to existing example sentences on the site with the goal being to give you a few different scenarios in which the sentence makes sense and show you what the question/answer/dialogue surrounding that sentence might look like. Here were a few examples

I’m really intrigues by this idea, and feel like it would be very valuable especially at the higher levels when nuance starts to come more into play. That being said, I’d be hesitant to rely too much on a ChatGPT solution for the reasons you mentioned about wonky Japanese. But if the content team ever expands, I’d love to see this added to their roadmap

[soundjona]

If I may revive this topic to give some more ideas, I’ve started recently using chatgpt to compare different grammar points and give examples sentences when I thought the bunpro grammar explanation where a bit ambiguous when it came to compare those points.

Example :

Or for example, in Bunpro it’s not really explained how じゃあ or さて differ. At least, with chatgpt I have some sense on how

I completely agree on the fact that this content should not be exposed just like any other content, some disclaimers should be there informing the user that this is AI generated and thus, it’s more pointers than actual grammar rules.

However, I do think that LLM AI, even if they can be wonky for a lot of use case, when it comes to language, they do actually perform well to give some concrete examples. But always with a pinch of salt, of course

[jrmr50]

AI that mutates the review sentences into using more vocab that I know would be nice. Rarely understand an N2 sentence due to this.

[casual]

An explanation of “why is my answer not accepted”, when there’s not one manually written yet, would probably be a great feature, if it’s short, clear and 99% correct. This is a regular and reasonable question from every learner. For now the alternative, at least for me, is to just move on and hope I’ll figure it out from immersion later.
One typical issue with LLMs is they love giving overlong roundabout generic non-answers (well, some humans do it too).
Another is, I suspect if you ask an LLM “what is wrong?” it will try very hard to come up with a reason, even if nothing is wrong and the answer is a completely natural synonym.
There’s also a question of costs of integrating the latest and greatest model every year.

Specifically for clarifying questions like “how is X different from Y”, wouldn’t a native human answer be much preferable? There are already places to ask those in the forums. All that’s left is for staff to find time to regularly follow-up, clarify/rewrite grammar points, and cross-link to related answers so they are easier to find. Sounds like a more productive use of time to me.

[Zenyth]

Thank you for being cautious about the use of generative AI and not just jumping on the bandwagon but actually taking time to think about if and how this could make bunpro better.

Personally, I would not use the first type of idea given and if it was implemented I would seriously consider leaving bunpro because generative AI like chatgpt are not at all reliable when it comes to giving explanations as they (especially chatgpt) are prone to making stuff up when there isn’t an easy answer available.

There is also the issue of power and water. Chatgpt and similar programs take up a lot of power and in order to cool down the computers generating this content, it takes a lot of water to keep things cool to not overheat. There are some instances where various AI systems could do serious good in the world such as AI being trained to recognize cancer cells, but in most cases we have to ask ourselves in this day and age where the climate and responsible stewardship of the planet has become a major issue how much physical resources are we willing to put into AI. In this case, if you do choose to incorporate generative AI, I would hope that you consider alternatives to chatgpt that have a lower environmental footprint.

Overall, a lot of the time, people choose to use AI because of its supposed “ability” to come up with new solutions to novel situations. However, AI cannot actually do this. All it can actually do is run the algorithm it was trained on to a new data set. This means that it is not an effective tool for generating answers as all answers should be double checked by an expert which defeats the point of it in the first place. I personally would not be comfortable with AI generated explanations for wrong answers or AI generated example sentences as I would not be able to trust that the information is correct. Adding this onto the fact that generative AI like chatgpt have not been optimized to use less power and thus has a significant environmental impact, I do not see the point of using a tool that has so many cons and very little pros.

I would instead propose that it would be more reliable to have specific threads for each grammar point that are directly linked to their page so that if you have questions you can just ask in the threads. People already do this, so streamlining it would be welcomed and also encourage people to engage with their learning more thus making better long term pathways.

[Rukifellth]

Ya kinda went way off track there about halfway through for no real reason.

While I don’t fully disagree with the sentiment that generative AI is bad, or LLM’s are bad, it is better than nothing. Many people cannot afford a tutor or have a great amount of difficulty understanding certain grammar points. it’s quite easy to make machines like X’s Grox or Bing’s AI really break down how certain grammar points into realistic examples or even use it in the context of your favorite TV shows, if you prompt engineer well enough. It is a lot like the old internet, you have to know “how” to search for something to get an accurate response.

I’m sure over time things will get more efficient and more accurate, but as of now it is probably better to wait until we have more filters against the poor responses that one can get from AI sometimes.

[nekoyama]

@Rukifellth

better than nothing

I think this is the key issue: If it’s wrong too often, it’s not better than nothing, it’s worse. And that goes especially for learners who have nothing else to turn to, because that’s exactly the people who won’t realize when the LLM is hallucinating.

And I think this is something that has to be re-evaluated pretty much constantly. I have records from last year where ChatGPT was insisting that 日曜日 is read hiyoubi. Obviously a huge red flag. But it doesn’t do this anymore, and the same probably applies to a lot of other red flags. At some point it will probably be good enough. That point might even be in the past now. Progress in LLMs has been ridiculously fast. But evaluating this properly is a ton of work.

[Noxsora]

There’s already form posts for each topic, but bunpro has a necromancy warning (against old reviving old topics)
If bunpro forums decreases pings for discussions in the grammar catigory so only replys ping - not new posts
then n00bs get pinged when someone offers a solution, but senpais don’t get pinged because some has a question 3 years later

I might repost this in suggestions. It would make the grammar points catigory にぎやか

[Noxsora]

As someone who actually has used Kanshudo-
It’s a group of educational mini games. Reminds me of Edutainment games like on PBSKid’s
click here for flash cards, there to read a text book, over here practice handwriting or over there do a reading passage
The “AI companion” Is just how Power discribes it in Gaishen Impact. It’s a to do list. Bunpro is straight foward compared to Kanshudo so it doesn’t need more than the current do reviews and learn new
Maybe add the readings to the workflow
“congratulations, you have learned all of L#, would you like to try Reading?” pop up when you finnish the learn step

calling Kanshudo an “AI” is like calling minecraft mobs AI. It’s totally diffrent than an LLM, we just use the same name for some reason.

if you press A n=n1.8
if you press B n=1.3n
C n=0.9*n
D n=1
Is my Anki “Algorithm”, it’s just really easy multication

[Rukifellth]

@nekoyama

I think this is the key issue: If it’s wrong too often, it’s not better than nothing, it’s worse. And that goes especially for learners who have nothing else to turn to, because that’s exactly the people who won’t realize when the LLM is hallucinating.

The thing is, it’s not always wrong. Teachers too are often wrong, after all. Heck, try to explain the usage of “for” to a japanese student, there’s a good chance you’ll miss a nuance or specific usecase. Imagine AI like a teacher that sometimes comes in drunk, yeah most of the time it’s right but sometimes it is. I’ve also seen japanese people misread kanji as well, so that’s hardly a point against AI. Ofcourse, you’re always going to be better off having a (physical) dictionary even if you are in a classroom or self studying.