AI based Japanese grammar checker (文法ーCHECK)

Japanese
#1

Hey everyone!

I’m a ~N1 level Japanese speaker, living in Tokyo and working as an AI researcher. I couldn’t find any decent free Japanese grammar checker online, so I decided to make one using AI!

Here it is:

https://bunpo-check.com/

How to use it

Just type your text in, hit check, and like magic, underlines will appear under any problematic parts of your sentence. Tap an underlined section to see the suggestion for fixing it. Click the speech bubble, and the AI will give you an explanation for the suggestion.

How accurate is it?

Will it always get everything right? Nope - this is still experimental and a work in progress.


Update: March 10th 2024 :partying_face:

文法ーCHECK has been updated today with a brand new, modern UI. In addition to giving suggestions for fixing your sentence, you can now also get explanations, as well as have general conversations about Japanese. I’ve also added a tone check feature for converting text to casual / normal / keigo politeness, too.

Note that this is still a work in progress and might give incorrect results as always. :slight_smile: Any and all feedback is welcome. I’m going to continue to work to make the accuracy better and better over time.


Thanks for everyone’s support until now! :heart: hope everyone enjoys the new features.

53 Likes
#2

Does the model also check for proper vocabulary usage? For example, I put in this sentence that I had asked Japanese people to correct (so I know what’s wrong with it):

Patreonとかないんですか？こんなにいいコンテンツがあるのに、財布がいっぱいなんですよ。こんなくだらない事態から手伝ってもらってもいいかな。

The biggest mistake was that I should have used 救う instead of 手伝う. When I put it in the checker, it highlighted から as the main problem, but when I changed 手伝う to 救う, it passed. So it seems like it’s deciding there’s a mistake based on the vocabulary, but highlighting something else in that case. (EDIT: actually, it makes sense that it underlined から, because it doesn’t go with 手伝う, awesome catch!)

If anyone is curious, the native-approved revision was the below (differences: use やる and switch 手伝う to 救う or 助け出す)

Patreonとかやってないんですか？こんなにいいコンテンツがあるのに、財布がいっぱいなんですよ。こんなくだらない事態から救ってもらってもいいかな。

The corrected Japanese-approved sentence also passed your checker! Seems like it’s working well with identifying whether there is a problem or not, and for that, it already seems valuable! Bookmarked.

I’m gonna try some more tests and report back here.

EDIT: I just went through the majority of HiNative sentences that I wrote and got corrected on.

All the Japanese natives’ sentences that I put in passed without errors! So it rarely has a false positive (says it’s wrong when it’s not).

I also put in a couple of my own sentences that I never got corrected on and was happy to see that most of them passed! (Might be false negatives, but it increased my self confidence a bit :D)

But I also put this piece in:
A: もしもし
B: あっ、忙しい？運転中みたいんだ。
A: えぇ。気分転換ドライブだけだ。
B: えっ！運転しながら電話するのは交通違反でしょ。
A: 大丈夫だよ。全然警察に気づかれないから。
B: バカ、このままじゃ他人を危険にさらすよ。罰金取れよ。

and was told that there’s an issue in みたい and 転換ドライブ. I put each of those sentences on their own (without "A: " or "B: ") just to make I wasn’t messing with the parser, and still got a highlighting in those respective places.

But two Japanese natives looked over the dialogue and didn’t point out that there was an issue in either of those sentences. However, they both pointed out that it should have been 罰金を取られろよ, so that was a false negative.

Looking back on it, there was indeed an issue: it should have 運転中みたいなんだ. So it caught a sentence natives overlooked :D. When I corrected that, it passed.

Overall, I’d say this is a nice tool. The fact I haven’t yet seen it fail a fully written out sentence made by a native is a really good sign. I still don’t see the issue with 気分転換ドライブだけだ, but since the tool rarely has a false positive, I’ll keep thinking about it

3 Likes
#3

Hi and welcome on this forum.
I am not (yet) skilled enough to check your app as NickavGnaro did, but thanks for sharing this very useful tool. It may really help when no native or high-level non-native is available.

3 Likes
#4

This tool is awesome! I tried to throw some curveballs at it and it handled everything without issue.
I even tried some nuanced grammatical mistakes ie “バナナなんか大好きだよ” which is correctly marked as wrong and “バナナなんか大嫌いだよ” which it understood to be correct.
This definitely has potential to become some sort of Grammarly for Japanese.
I really look forward to seeing what you do with this project!

4 Likes
#5

Really interesting, would love to see the code for how you trained the model.

2 Likes
#6

Very nice idea for an AI project, congrats.

3 Likes
#7

Thanks for the feedback everyone! Happy to hear you’ve had a good experience with it so far and that it’s done pretty well even with some difficult sentences, that’s encouraging.

@NickavGnaro with regards to your points specifically, you’ve figured out the false positive rate is quite low through experimentation, which is correct! The false positive rate is actually quite a bit better than the false negative rate (although neither is bad). I think there might just be no issues with your sentence there, and it could still definitely be a false positive. The model is not perfect and it will get things wrong some of the time. But yeah hopefully if it flags something up, it can bring your attention to it, so that you can look into that part.

4 Likes
#8

That’s awesome, great work!

With the false negative rate being sufficiently low, if it passes without noticing any mistakes, then at least it’s probably not a disaster of a sentence. But even better is that with the false positives being this low, I basically know that if it notices a potential mistake on something, I should really think long and hard about where I messed up on xD. So it’s really ideal for making improvements with confidence, and having some guarantee that what you put in there is pretty decent Japanese, even if it could still be made slightly more natural. I’m gonna get a lot of use out of this :smiley:

4 Likes
#9

Update

A minor update today!

:relaxed: No more model feedback needed!

Thanks to everyone who gave feedback to the model through the mechanism on the site! Thousands of sentences were sent as feedback which was really helpful to me in seeing where the model was going wrong. Since the performance is now pretty good, I’ve removed the feedback mechanism which means, no more annoying messages asking you for feedback! Should make checking things much quicker. As always though, feel free to send me your thoughts or problems with the site, either here or at the email address on the website.

:lipstick: Make-over!

I decided to re-design the site with a brighter theme, since some people couldn’t read the text clearly. I’ve also set the default language to English.

Thanks for all your interest, almost 10,000 sentences have now been checked on 文法ーCHECK! :open_mouth:

3 Likes
#10

If only I had a magic wand to have the AI automatically suggest changes to fix the incorrect grammar points :upside_down_face: I remember bookmarking the website when you first posted it; completely forgot about it, but going to start using it again!

3 Likes
#11

Ahah yup that’s the next step! It’s a pretty big undertaking but I’ll give it a go when I have more time .

2 Likes
#12

All the best! Something of that nature that was even remotely successful in suggesting fixes would be absolutely monumental for everyone learning the language.

2 Likes
#13

wow I, have no words except maybe I love you? I am so lit about this, I screamed. What a generous human; we don’t deserve you

4 Likes
#14

:pleading_face: :disappointed_relieved: maybe i love you too !!! :heart:

2 Likes
#15

You are exactly who I wish I was! This is so inspiring!

1 Like
#16

If you would ever be interested in talking about this (or anything else) to an engineer in training please private message me.

1 Like
#17

Thanks for sharing your work!

2 Likes
#18

Very cool!!

1 Like
#19

Just came here to say a heartfelt “Thank you!” As for now I’m studying on my own and I really need to somehow check my answer without using a key. So your grammar checker just the thing. You’re the best! Thanks!

1 Like
#20

This is great! Do you know if the hugging face models support part of speech tagging? I would find that very useful as I study reading material, https://jisho.org/ has some part of speech tagging but it’s not the best.

I think an improved part of speech tool would be helpful from a learning perspective. I am pretty familiar with the English variations of these models and would be interested in putting in some work to make something like that.

Let me know if you think it could fit in!

ありがとう！

2 Likes