Its (usually) not a substitution at all.
I mean I guess its a substitution the same way replacing any particle with any other particle is “substituting” it.
But at the end of the day, its just a completely different particle, doing its job as a particle, with the nuance of that particle.
First I thought this post was talking about the actual case of が being substituted by の. As we know, in older grammar, が literally substituted for の; “我が家” = “我の家”.
One difference in the sentences lies in the predicates which are being referred to to embed the larger clause into one main and one relative-clause:
A (⓪が)(私が食べた)ケーキ(である)
B (⓪が)私の(食べたケーキ)(である)
In B, の refers to the ケーキ (or rather the omitted copula). We are saying “It” (⓪が) is (である/だ) my (私の) eaten-cake (食べたケーキ); Here, 食べたケーキ is the relative (aka embedded) clause. Note that the Subject (who has eaten the cake) is implied; “I” am not the Subject but may be implied to be. Whether the cake was eaten by me or by someone else remains ambigious. Thats very typical for Japanese.
Although note that based on pronunciation, 私の(食べたケーキ) can also function as (私の食べた)ケーキ in which case this is closer to a substitution of が. Though the difference is subtle, we are still not declaring “I” as the Subject. If I had to translate, it would be like saying “The cake eaten by me (Lit. “The cake of my eating”)”, and not “The cake I ate”.
In A, 私が refers to the verb 食べた and builds a relative clause with it;
“It” (⓪が) is (である/だ) a I(Subject)-have-eaten (私が食べた) cake (ケーキ). Here, the Subject is clear but of course we do not mention possession–so whether it is “my” cake or not remains unknown.