Because it's not economic for the pool or miner. There's also a "hard" cap (which goes up with median) at 2x the median. If a miner mines a too big of a block, he loses some base block reward (emission) and for it to be economic to him, the fees must be high enough to compensate and earn a little more.
Each wallet fee level corresponds to a % increase incentive for the miner. Bigger the fees, faster the increase. Bigger the increase, cheaper the fees =)
Low priority (min fee) - 1x - 0.6% increase is the "sweet spot" (economic up to 1.2%, after that it's negative earnings)
Normal priority (default fee) - 4x - 2.4% increase is the "sweet spot" (economic up to 4.8%, after that it's negative earnings)
High priority 20x - 12% increase is the "sweet spot" (economic up to 24%, after that it's negative earnings)
Ultra priority 166x - 100% increase is the "sweet spot". Can't make a bigger block than this, anyway :)
When a miner fills the block from highest to smallest fee, it will be the fee of last TX to get into a block which determines the optimal increase. So, it's not just enough to have 1TX in the pool to hasten an increase. There must be many of those, enough to make 100 blocks full of high-fee TX-es to lift the median quickly.
This is why a huge backlog can never develop.