Hey sorry for the delay!
I tried to write in a way as clear as possible, I hope it will be understandable.
Yes, ほかの modifies a noun, and ほかに modifies adjectives and verbs(adverb).
In this case, ほかに refers to the book from the previous sentence, since it can’t really go by itself(though like in the description, if we know what we are talking about it can be omitted).
I like to think of ほかに as “other than”, and ほかの as “other”.
So we are really saying:
この本だけですか。この本のほかに、ほしいものはありますか。
Clearly said by clerk. In addition to that, is there something you want?
この本だけですか?ほかのものはありますか?
“Only this book? Do you have other(different) ones?”.
Clearly said by a client, which doesn’t seem to like the book shown to him.
If ほかにほしいものはありますか. was said alone, by itself - it would be unnatural, since person hearing it would think “other than what??”.
That is why some say that ほかに has inclusive nature, and ほかの the exclusive nature.
このリンゴがすき。ほかに赤いのが欲しい
(in grocery shop)I like this apple. I want red one other than that(in addition to that).
(and the first apple is probably not red, most likely green or something)
このリンゴは好きじゃない。ほかの赤いのが欲しい。
I don’t like this apple. I want other(different) red apple(it also suggests that the apple we are talking about was red).
他に質問はありませんか?
Any questions other than that?(in addition to those you asked)
他の質問はありませんか.
Do you have other question?(different one, because we can’t answer the one you asked second ago, something like that).
And like you thought, saying この本のほかのもの would sound strange, something like another thing of this book.