When you are signing a Monero transaction, do you sign the transaction itself, or do you hash the transaction and then sign that?
In the interest of pedantry (and since I can't comment -_-):
You sign the hash of the transaction prefix. In Monero that is everything but the signatures.
(at the above answer) Monero relies on ed25519, not EdDSA. EdDSA is a particular signature system (completely absent in Monero). Monero's ring signatures are presently the Fujisaki-Suzuki variety, implemented in the ed25519 curve.
One additional quibble: in Monero's ring signatures, the hash doesn't actually have "mathematical" transformations applied--it is rather hashed again along with a bunch of other data--so one actually could "sign" an arbitrary-length message instead of a hash of said message.
All¹ asymmetric signature protocols calculate a hash of the message and then apply the “mathematical” transformation to this hash. This includes EdDSA which Monero relies on. The reason is that all those mathematical transformations work on fixed-size numbers (of about a few hundred bits). Hashing condenses the information into this fixed size, with the property that it is effectively impossible² to have two messages (here, two transaction logs) with the same hash, hence a signature only validates one particular message.
The use of a hash (or at least a function with some of the properties of a hash: condensing information while being effectively injective) is intrinsic in any form of blockchain, where the signature of a transaction also validates all the history of the system that made this transaction possible (e.g. a debit has to validate all the transactions that credited the amount to the debited account) but the signature size should not grow as the history grows.
¹ At least all the ones I've heard of, and doing otherwise would be surprising.
² I.e. if it's possible then the hash function is considered badly broken.