How does signing the hash of the public key show that the signer knew the private key?
That's what signatures do. They prove that a private key must have been known for a particular public key that the verifier checks there has been a successful signature for. You can sign the message "hello" and that'd still prove you knew the private key for the public key that the recipient is expecting the message to have been signed for.
It explains it a little better further down in that document. It says:
A pedersen commitment can be proven to be a commitment to a zero by
just signing a hash of the commitment with the commitment as the
public key. Using the public key in the signature is required to
prevent setting the signature to arbitrary values and solving for the
commitment. The private key used for the signature is just the
So your commitment to zero will be P = xG + aH, and since a=0, that simplifies to P = xG. Therefore the public key is P and the private key is x.
So you sign the message H(P) using the private key x. The recipient will know P, and will check the signature, which will prove x was known. Proving x was known proves that the commitment must have been a commitment to zero, because the private keys for Pedersen Commitments to non-zero values can never be known (because H=yG where y is unknowable).
Using H(P) as the message prevents attacks.