For blocks above the median block size of the last 100 blocks but below the 2x the median block size dynamic cap a quadratically increasing penalties is applied to miners.
How is this penalty calculated?
Can miners avoid the penalty by refusing to mine any transactions that would exceed the threshold where the penalty exceeds the transaction fees earned for including those transactions?
For blocks above the median block size of the last 100 blocks but below the 2x the median block size dynamic cap a quadratically increasing penalties is applied to miners.
How is this penalty calculated?
Current block reward penalty is given by:
P = R * ((B / M) - 1) ^ 2,
where R is the base block reward, B the block size and M the median block size of the last 100 blocks. The penalty doesn't come in effect unless B > M_0, where M_0 = 300000 bytes is the minimum penalty-free block size. Maximum allowed block size is 2 * M, at which the full base block reward is withheld.
Can miners avoid the penalty by refusing to mine any transactions that would exceed the threshold where the penalty exceeds the transaction fees earned for including those transactions?
Yes, but actually that is not the most economic decision, as they will earn more for a slightly bigger block. To understand this, we must also see how dynamic fees are calculated:
F_min = (R / M) * (W_0 - 1),
Where W_0 = 1.0012 is what we can call the block expansion factor and R and M have the same meaning as above.
If everyone is paying the minimum fee, and a miner mines a block with W_0 expansion factor (1.2% bigger than median), his total fees earned will be the same as if he mined a 100% full block. With the minimum fee, anything between 0% and 1.2% increase will earn him more (0.6% increase is the optimum for the min fee)! Of course, there must be enough transactions in the TXpool for him to be able to actually fill the block with fee-paying transactions.
Note that he has no incentive to stuff the block with his own transactions to try to keep the median artificially high because he earns the most at 100%+ block capacity utilization. This is because the wallets auto-adjust their fees as the block size changes. For example, if the median is 400kB, but there are only transactions to fill 300kB blocks, he'd earn more if the median was dropped to 300kB as it would increase the fee paid by wallet so those 300kB would then come with more fees.
So we see how the min. fee gives some incentive to increase the blocks. But there are options to offer a multiplier of the minimum fee and there is an optimum block size increase for each wallet setting, as we can see below:
The "sweet spot" for the miner is where the optimum curve crosses any of the fee curves. That means, that in case of heavy network usage any backlog could be resolved: the more people compete with their fees, the faster it will be resolved.
Of course, due to how median works, in case of block propagation problems etc, 51% miners could veto any median increase even if it would be economic. They'd just have to keep mining blocks at 100%, and the median wouldn't move regardless of what others do.
When a block is built a block larger than the median size of the last 100, then a quadratic penalty is imposed on the miner such that:
penalty = base subsidy * ((block size / median size of last 100 blocks) - 1)²
subsidy = base subsidy - penalty
Miners can avoid this fee altogether by refusing to include transactions in blocks that would lead to exceeding the median block size, although this is likely not the optimal practice as including such transactions will often still be profitable.
Miners can create their own criteria for including transactions in blocks by balancing transaction fees collected against the quadratic miners penalty that will be applied.
For example for all blocks exceeding the median blocksize they could enforce minimum transaction fees with a formula that scales fee requirements to compensate for the quadratic penalty.
