Hey sorry for the delay!
I tried to write in a way as clear as possible, I hope it will be understandable.
Yes, ほかの modifies a noun, and ほかに modifies adjectives and verbs(adverb).
In this case, ほかに refers to the book from the previous sentence, since it can’t really go by itself(though like in the description, if we know what we are talking about it can be omitted).
I like to think of ほかに as “other than”, and ほかの as “other”.
So we are really saying:
Clearly said by clerk. In addition to that, is there something you want?
“Only this book? Do you have other(different) ones?”.
Clearly said by a client, which doesn’t seem to like the book shown to him.
If ほかにほしいものはありますか. was said alone, by itself - it would be unnatural, since person hearing it would think “other than what??”.
That is why some say that ほかに has inclusive nature, and ほかの the exclusive nature.
(in grocery shop)I like this apple. I want red one other than that(in addition to that).
(and the first apple is probably not red, most likely green or something)
I don’t like this apple. I want other(different) red apple(it also suggests that the apple we are talking about was red).
Any questions other than that?(in addition to those you asked)
Do you have other question?(different one, because we can’t answer the one you asked second ago, something like that).
And like you thought, saying この本のほかのもの would sound strange, something like another thing of this book.