From my experience with a US region PS4, changing the language settings only affects the system interface. I would imagine that carried over to the PS5. Changing the language on the Switch does change the game text (for some games) so I was hoping that would work for PS games. It did not.
As far as I can tell, the only way to play games in Japanese on non-Japanese systems is to either obtain a Japanese copy of the game or to create another PS account and set the region to Japan and buy games from the PS store. I haven’t actually tried it myself, so I don’t know how annoying the process is.
I hoped that helped at all!