I'm going to do my best to answer your questions but I'm afraid I'm no ArticMine, who I'm sure could answer far more eloquently.
I'm hoping to find a current reference that explains how the dynamic block size/fee calculation works? ... how does Monero set the fee during normal operation?
Beyond the code (e.g. source for getting a fee estimate), A note on fees is the most recent explanation I can find. And whilst there have been tweaks since the addition of Bulletproofs, it is largely still relevant/accurate.
The biggest change to the various calculations when Bulletproofs were introduced, was a need to factor in the huge space savings Bulletproofs introduced. Therefore, instead of simply using size as weight, the notion of a "clawback" was introduced to offset this space saving when calculating a txs weight. This can be observed in commit 5ffb2ff9. Therefore, whilst the terms size and weight were previously the same thing (e.g. size), weight no longer strictly equals size.
Another recent change can be observed in commit b8787f43, quoting the commit message:
ArticMine's new block weight algorithm
This curbs runaway growth while still allowing substantial
spikes in block weight
Original specification from ArticMine:
here is the scaling proposal
Define: LongTermBlockWeight
Before fork:
LongTermBlockWeight = BlockWeight
At or after fork:
LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight)
Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time.
Define: LongTermEffectiveMedianBlockWeight
LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight))
Change Definition of EffectiveMedianBlockWeight
From (current definition)
EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight))
To (proposed definition)
EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight)
Notes:
1) There are no other changes to the existing penalty formula, median calculation, fees etc.
2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork.
3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
Note: the long term block weight is stored in the database, but not in the actual block itself,
since it requires recalculating anyway for verification.
It also seems like this would leave the Monero vulnerable to moderate spam attacks ... what safeguards (price disincentives) are in place to protect the blockchain against a spam attack?
Now we need to take a step back. A spender putting too much data in a tx, they pay higher tx fees. Send lots of txs, they have to pay the fees on each. But, in general, a spam attacker wants to get these txs onto the blockchain, so now we need to also look from the miners perspective. A miner is incentivized to produce blocks of optimum size based on normal network usage. Trying to stuff the blocks with too many txs based on past/recent usage, they get penalized. For an attacker to incentivize miners to create larger than normal blocks, fees would need to go up, to compensate for the reward penalty. Therefore to flood the blockchain with spam txs, the more one deviates from normal usage, the more it's going to cost. This is why fees are calculated not just on tx weight, but also median block weight.