Let's say I have all the words to a mnemonic seed. I scramble them in an order that only I know. Can someone take that scrambled up order of words and unlock my wallet? Or does the phrase have to be exactly in the same order that I first saw it?
Yes, the word order matters. Somebody else can provide the correct math behind this answer, but the short version is that words out of order will have different results.
For example, if I generate this random seed using Luigi1111's generator:
down nurse broken pager phrases owner shuffled titans pavements zapped fungal balding ruffled swagger noodles duties haystack yearbook navy akin woes glass pram punch haystack
it results in this address:
But if I switch the first two words (placing "nurse" in front of "down") the generator errors:
"Your private key could not be verified, please try again"
Long story short, the word order matters. The one change you can do is shorten each word to its first three letters. So instead of "down nurse broken ..." you could write "dow nur bro ..."
Edit: Good input by Jolly Mort below. The generator errored, not because that mnemonic seed is bad, but because the 25th word in the seed (haystack) is a checksum of the other 24, so it will error if any of the previous words are changed. If the checksum word was recalculated after switching down and nurse, the address would give a legit address, but it would not be the same one as before. So the word order still matters.
The mnemonic seed is constructed by splitting the private key in 32 bit chunks, then interpreting these chunks as a number, each of which translates to a set of three words in the list you are using (each list contains 1626 words, and 1626^3 is slightly over 32 bits). This implies that if you move some words around, the reconstructed bitstring (and thus private key) will be different.
So, that means yes.