Was going to add something similar. You’re not wrong at all, I think really the only difference in many cases is the nuance, and several particles can be used correctly.
父に会う - You met your dad (focus on your action).
父と会う - You and your dad met (both people doing the same action).
家族で会う - You met somebody with your family.
Note that the last one doesn’t have 父, as で is only used for ‘with’ when the word that comes before it is a group that the speaker is a part of. 父 is not a group that the speaker is a part of, it is one person.
It can come across this way, or it can come across simply that 父 simply had no control over it. That means it’s either a coincidence, it was unplanned, or that the only person that planned it was the doer. 父 himself is nothing more than the destination. You’d need other context to really tell which of these meanings is true.
Specifically for 会う, there are several different kanji that are all used relatively frequently that have slightly different meanings. 合う, 遭う, 遇う, 逢う. However, they all just mean ‘to come together’, either physically or metaphorically/rhythmically, etc.