AI based Japanese grammar checker

Hey everyone!

I’m a ~N2 level Japanese learner and currently doing my Masters in Comp Sci & AI. I couldn’t find any decent free Japanese grammar checker online, so I decided to make an AI model which can highlight grammatical errors in Japanese!

It’s totally free to use, there are no ads or anything, so if you wanna check it out:

https://bunpo-check.com/

How to use it

Just type your text in, hit check, and like magic, underlines will appear under any problematic parts of your sentence. Be aware there’s a limit of 100 sentences, after which you’ll have to wait a while before checking again (to prevent the servers getting overloaded).

How accurate is it?

Will it always get everything right? Nope - do note, this is still experimental and a work in progress. Don’t use this to check your Japanese resume, but if you’re wondering whether a sentence sounds weird or might be off, this can help you figure out which part is wrong.

Open source

If you’re interested in using the model for your own projects, it’s free for download under an MIT license (so you can do whatever you like with it), alongside code to help use it or generate pre-training. All can be found at my github:

Update: 8 April 2021

:relaxed: No more model feedback needed!

Thanks to everyone who gave feedback to the model through the mechanism on the site! Thousands of sentences were sent as feedback which was really helpful to me in seeing where the model was going wrong. Since the performance is now pretty good, I’ve removed the feedback mechanism which means, no more annoying messages asking you for feedback! Should make checking things much quicker. As always though, feel free to send me your thoughts or problems with the site, either here or at the email address on the website.

:lipstick: Make-over!

I decided to re-design the site with a brighter theme, since some people couldn’t read the text clearly. I’ve also set the default language to English.


Thanks for all your interest, almost 10,000 sentences have now been checked on 文法ーCHECK! :open_mouth:

35 Likes

Does the model also check for proper vocabulary usage? For example, I put in this sentence that I had asked Japanese people to correct (so I know what’s wrong with it):

Patreonとかないんですか?こんなにいいコンテンツがあるのに、財布がいっぱいなんですよ。こんなくだらない事態から手伝ってもらってもいいかな。

The biggest mistake was that I should have used 救う instead of 手伝う. When I put it in the checker, it highlighted から as the main problem, but when I changed 手伝う to 救う, it passed. So it seems like it’s deciding there’s a mistake based on the vocabulary, but highlighting something else in that case. (EDIT: actually, it makes sense that it underlined から, because it doesn’t go with 手伝う, awesome catch!)

If anyone is curious, the native-approved revision was the below (differences: use やる and switch 手伝う to 救う or 助け出す)

Patreonとかやってないんですか?こんなにいいコンテンツがあるのに、財布がいっぱいなんですよ。こんなくだらない事態から救ってもらってもいいかな。

The corrected Japanese-approved sentence also passed your checker! Seems like it’s working well with identifying whether there is a problem or not, and for that, it already seems valuable! Bookmarked.

I’m gonna try some more tests and report back here.

EDIT: I just went through the majority of HiNative sentences that I wrote and got corrected on.

All the Japanese natives’ sentences that I put in passed without errors! So it rarely has a false positive (says it’s wrong when it’s not).

I also put in a couple of my own sentences that I never got corrected on and was happy to see that most of them passed! (Might be false negatives, but it increased my self confidence a bit :D)

But I also put this piece in:
A: もしもし
B: あっ、忙しい?運転中みたいんだ。
A: えぇ。気分転換ドライブだけだ。
B: えっ!運転しながら電話するのは交通違反でしょ。
A: 大丈夫だよ。全然警察に気づかれないから。
B: バカ、このままじゃ他人を危険にさらすよ。罰金取れよ。

and was told that there’s an issue in みたい and 転換ドライブ. I put each of those sentences on their own (without "A: " or "B: ") just to make I wasn’t messing with the parser, and still got a highlighting in those respective places.

But two Japanese natives looked over the dialogue and didn’t point out that there was an issue in either of those sentences. However, they both pointed out that it should have been 罰金を取られろよ, so that was a false negative.

Looking back on it, there was indeed an issue: it should have 運転中みたいなんだ. So it caught a sentence natives overlooked :D. When I corrected that, it passed.

Overall, I’d say this is a nice tool. The fact I haven’t yet seen it fail a fully written out sentence made by a native is a really good sign. I still don’t see the issue with 気分転換ドライブだけだ, but since the tool rarely has a false positive, I’ll keep thinking about it

3 Likes

Hi and welcome on this forum.
I am not (yet) skilled enough to check your app as NickavGnaro did, but thanks for sharing this very useful tool. It may really help when no native or high-level non-native is available.

3 Likes

This tool is awesome! I tried to throw some curveballs at it and it handled everything without issue.
I even tried some nuanced grammatical mistakes ie “バナナなんか大好きだよ” which is correctly marked as wrong and “バナナなんか大嫌いだよ” which it understood to be correct.
This definitely has potential to become some sort of Grammarly for Japanese.
I really look forward to seeing what you do with this project!

4 Likes

Really interesting, would love to see the code for how you trained the model.

2 Likes

Very nice idea for an AI project, congrats.

3 Likes

Thanks for the feedback everyone! Happy to hear you’ve had a good experience with it so far and that it’s done pretty well even with some difficult sentences, that’s encouraging.

@NickavGnaro with regards to your points specifically, you’ve figured out the false positive rate is quite low through experimentation, which is correct! The false positive rate is actually quite a bit better than the false negative rate (although neither is bad). I think there might just be no issues with your sentence there, and it could still definitely be a false positive. The model is not perfect and it will get things wrong some of the time. But yeah hopefully if it flags something up, it can bring your attention to it, so that you can look into that part.

4 Likes

That’s awesome, great work!

With the false negative rate being sufficiently low, if it passes without noticing any mistakes, then at least it’s probably not a disaster of a sentence. But even better is that with the false positives being this low, I basically know that if it notices a potential mistake on something, I should really think long and hard about where I messed up on xD. So it’s really ideal for making improvements with confidence, and having some guarantee that what you put in there is pretty decent Japanese, even if it could still be made slightly more natural. I’m gonna get a lot of use out of this :smiley:

4 Likes

Update

A minor update today!

:relaxed: No more model feedback needed!

Thanks to everyone who gave feedback to the model through the mechanism on the site! Thousands of sentences were sent as feedback which was really helpful to me in seeing where the model was going wrong. Since the performance is now pretty good, I’ve removed the feedback mechanism which means, no more annoying messages asking you for feedback! Should make checking things much quicker. As always though, feel free to send me your thoughts or problems with the site, either here or at the email address on the website.

:lipstick: Make-over!

I decided to re-design the site with a brighter theme, since some people couldn’t read the text clearly. I’ve also set the default language to English.

Thanks for all your interest, almost 10,000 sentences have now been checked on 文法ーCHECK! :open_mouth:

3 Likes

If only I had a magic wand to have the AI automatically suggest changes to fix the incorrect grammar points :upside_down_face: I remember bookmarking the website when you first posted it; completely forgot about it, but going to start using it again!

2 Likes

Ahah yup that’s the next step! It’s a pretty big undertaking but I’ll give it a go when I have more time .

2 Likes

All the best! Something of that nature that was even remotely successful in suggesting fixes would be absolutely monumental for everyone learning the language.

2 Likes

wow I, have no words except maybe I love you? I am so lit about this, I screamed. What a generous human; we don’t deserve you

4 Likes

:pleading_face: :disappointed_relieved: maybe i love you too !!! :heart:

2 Likes

You are exactly who I wish I was! This is so inspiring!

1 Like

If you would ever be interested in talking about this (or anything else) to an engineer in training please private message me.

1 Like

Thanks for sharing your work!

2 Likes

Very cool!!

1 Like

Just came here to say a heartfelt “Thank you!” As for now I’m studying on my own and I really need to somehow check my answer without using a key. So your grammar checker just the thing. You’re the best! Thanks!

1 Like

This is great! Do you know if the hugging face models support part of speech tagging? I would find that very useful as I study reading material, https://jisho.org/ has some part of speech tagging but it’s not the best.

I think an improved part of speech tool would be helpful from a learning perspective. I am pretty familiar with the English variations of these models and would be interested in putting in some work to make something like that.

Let me know if you think it could fit in!

ありがとう!

1 Like