Since the penalty function is inherited from CryptoNote, we must look at the whitepaper for the "original thought" on the subject.
A miner still has the ability to stuff a block full of his own zero-fee transactions up to its maximum size 2 * M_b. Even though only the majority of miners can shift the median value, there is still a possibility to bloat the blockchain and produce an additional load on the nodes. To discourage
malevolent participants from creating large blocks we introduce a penalty function:
NewReward = BaseReward * (BlkSize / M_N - 1) ^ 2
This rule is applied only when BlkSize is greater than minimal free block size which should be close to max(10kb, M_N * 110%). Miners are permitted to create blocks of “usual size” and even exceed it with profit when the overall fees surpass the penalty. But fees are unlikely to grow quadratically unlike the penalty value so there will be an equilibrium.
We can see that Monero deviates from this as the penalty is triggered for any %. I'm not sure what the reasoning was, but one could assume it had to do with preventing a spam attack. With a free 10% increase, any >0 fee transactions would be economic for the miner and someone could cheaply keep the blocks growing all by himself for almost no cost. The do-not-relay minimum fee could maybe alone counter this, but I don't have sufficient knowledge to know whether it would be safe to rely on it alone.
I'd say the whole idea is to throttle the rate at which the block size can grow, while allow some pressure to build up and push them upwards should there be a rational need. But why does the growth rate need throttling? Let's see the example given.
Let's say there are 100 transactions waiting to be included in the blockchain, I can't see the drawback of including all of them in the next block.
That would mean a 1.3MB block. We could probably get away with that. But that would mean a 2067% increase! What if the median was 1.3MB and you had 10000 low-priority transactions? Would you still say it's ok to mine a 130MB block? How would the network deal with that? I'd argue it's common sense not to allow for such shocks. The growth rate must be limited in some way, because the growth rate of everything else is limited by some other mechanisms. Sure, one day we will have 130MB as the norm, but the hardware will be there as well and the network will have grown into it.
These mechanisms are there to ensure smooth functioning of everything while we get there.
It's not just about that. We want some fee pressure. If every TX would get in immediately why would anyone ever raise the fee to "priority"? If the fee was ultra-low, how would you make people use the network rationally? Making the network "work" for someone sending a value of 0.001$ hardly seems rational and long term it could make the system collapse. I wouldn't bend over to pick up 0.1$ and the entire network should store multiple copies of a TX spending 0.001$ and pass it around forever? As we can't know the actual amounts involved, the only way to keep the usage rational is to make it cost something as a kind of "proof of intent".
I'd say the problem is not in the formulas themselves, but just in the starting parameters which were kind of unintentionally changed with introduction of RCT.
