Just checked the stats! I got sidetracked by this whole song and dance.
Thank you to everyone who gave it a try, and especially to those who sent feedback 
it doesn’t follow from the same schema being used during eval that the test would be any more or less accurate
In the previous paragraph, the subject was “it” (the eval framework), and answers were never mentioned. So this brings up at least two questions. 1. What is the nature of these “answers” 2. What is “the live config”. This is also abruptly introduced, and we have to assume that you mean some kind of configuration for the live application that would give similar “answers”.
Let’s also notice now that this concept of “pairs” has yet to be mentioned or defined, so we have to assume that it is some component of the aforementioned “schema”. The phrase “seeded off the pair” so re-runs are stable is absolutely meaningless, maybe it could say “seeded off the pair ID so re-runs are stable”, but then we have to take all of our previous assumptions into account.
The fact that “the judge can return a tie or mark both bad” does not seems to have anything to do with this flipping process, unless the reader further assumes that the judging process is marking the “pairs” of “answers” as better or worse than one another. (Where one of them is the human input? As a reader we have no idea.)
I could continue, but I think you will notice the following about the rest of the sentences in the paragraph.
-
A concept is introduced without any prior context or relation to the sentences that came before (in the following sentences these include “the furigana”, and “turns”).
-
There is some attempt at making one portion of the sentence appear as a logical substantiation for some other part of the sentence, when in fact there is no such substantiation, or the opposite can be inferred.
A large chunk of my waking hours are now spent reading AI generated English writing with these problems, and asking the author (human proxy) to fix the clear lack of foundation or true exposition in the seemingly information-dense arguments that come back from their AI.
I can only assume that AI generated Japanese text has a similar writing style, and I have no interest in learning from it.
3 Likes
Totally fine! If you have other cheap ways to improve your Japanese conversation skills, go for it. In fact, I would love to know what you’re using. I made Unsubtitled because I couldn’t find an existing app that does free-form conversation with corrections and is fun to use.
Like I said a few times already in this chat, I’m not here to have a philosophical debate on the merits of using AI for learning (although I feel like I got baited into getting off topic). Not here to get the app’s learning methodology peer-reviewed either lol. Sorry (not sorry) that I can’t tell you exactly how I built it… that’s what I do for my day job, and I’ve got it covered. But thank you for taking the time to respond, I suppose, even if you’ve decided you’re not a fan before even trying it.
I’m excited that other people are having fun with it now! 
1 Like
So, not only were you keeping a spreadsheet of errors across multiple Japanese learning apps (which… why?), you also counted the number of sentences you encountered (to determine the error-to-sentence ratio)? And you did this with multiple apps shortly after they launched?
1 Like
Yep. For about two years, it was all Japanese all the time - I was studying about four hours a day. I worked at a Japanese language learning company, traveled there frequently for work, and had a timeline to move there. I used pretty much every available resource during that period to learn from multiple angles and solidify my knowledge as quickly as possible. I made a schedule to complete everything on time, and if anything didn’t make sense to me, even a little, I wrote it down for later review. It was more about keeping myself on pace than doing free QA for everyone, but that’s kinda what it turned into.
And then life went in a different direction. I don’t have much opportunity to speak Japanese day-to-day now, so I’ve gotta put more effort into practicing on my own.
2 Likes
But why were you counting the number of sentences you consumed on those apps? Logging errors only collects half of the data need to determine an error rate.
Because I was tracking how much material I needed to get through on each platform every day!
1 Like
So I did a little searching to find what scholarly research is actually out there. If people are actually interested in this topic in good faith, this looks good to follow:
There are also studies out of Chinese universities doing ESL education, which I won’t paste here because it would just be a ChatGPT text dump. TL;DR though:
- Complexity, Accuracy, and Fluency are not the same thing
- Chatbots seem to help on the complexity and fluency dimensions, not so much on accuracy so far
- Chatbots need to let some mistakes slide (human teachers definitely do this too)
2 Likes
the link doesn’t even work
A bunch of people are using it now, so it definitely does work, at least on iOS and Android.
What kind of phone are you using?
no phone, just chrome on a laptop
Gotcha, thanks for the info. I can’t see that anything is wrong, and people have been actively using it yesterday and today.
Maybe it was due to this partial Cloudflare outage - Services & Sites - Cloudflare Status
Hey there!
I just started using it, and love the voice of Mei.
The voices are so expressive! Just curiosity, what voice generator are you using? 
I will be using it everyday this month (subscribed), so perhaps at the end i’ll have more feedback if I remember to post it.
Cheers
1 Like
Thanks, so glad you’re enjoying it!
I evaluated many voice options and ultimately chose Cartesia (Cartesia | AI Voices) to start. I feel like they do a good job balancing realism, expressiveness, cost, and latency.