I'll talk about monero-wallet-cli and monero-wallet-gui, since I don't know about other wallets.
It does not download the whole blockchain from the daemon. There are two things that get cut out:
- first, all blocks up to (a little bit before) the wallet creation height are not downloaded at all.
- second, for those blocks which are downloaded, tx signatures and related things are not downloaded
If you're creating a new wallet, this means about 1.4 million blocks aren't downloaded. For those later blocks which get downloaded, most of the block size is taken by range proofs, which are part of the data that's not downloaded.
So, in short, monero-wallet-cli does not download anywhere near the entire blockchain.
Now, monerod may be running anywhere you want. Either on the same machine/VM as the wallet, or elsewhere. If you're using the wallet on a phone, then you're probably going to run monerod either on a home machine, or a VPS, and connect your wallet to it. monerod will obviously get the whole blockchain, so you get the benefit of checking it for validity.
Last, MyMonero is being changed to that anyone will be able to run a mymonero node (at home or on a VPS, again), and you'll connect to that from your phone wallet. This will be even lighter than the current wallets, since only your own transactions (to/from you) will be downloaded, as the scanning will be offloaded to the MyMonero style server you'd be running. This is a work in progress that is not yet ready, however.