"transferFee" should be set to 0, as it's an obsolete setting: the wallet will calculate the fee for this transaction based on its size.
"minPayment" is the amount of monero a miner should reach before a payment is made.
"denomination" is the granularity of the payment made. That is, payments will be made in multiples of that number (most likely leaving some unsent small amount in the miner's pending balance).
Note that all amounts above are expressed in atomic units, of which there are 1e12 per monero. For example, 1 monero is 1000000000000, 0.1 monero is 100000000000, etc.
To give a more real world example of minPayment and denomination, you could compare it to a scheme where you pay someone, but only using fairly large coins. For instance, with 0.10 coins, if the person you're paying has a balance of 0.46, you can pay 0.4 (4 coins), and the remaining still unpaid balance is 0.06.
With RingCT, denomination does not matter anymore, since amounts are not decomposed in several outputs anymore, so there is no dust problem. Therefore, denomination can be set to 1.