No, a remote node cannot determine your wallet address. 

However, using a 3rd party remote node, you are trusting the remote node operator to not: log your usage (e.g. IP analysis), trusting that they will not censor your transactions (e.g. broadcast them), trusting they are on the correct chain and trusting they are not doing any kind of transaction tracing. 

The risk of IP analysis can be lessened by using the remote node over tor/i2p/vpn. Checking they are running the correct chain / not censoring your transactions can, with a little investigation, be checked/verified. But you cannot mitigate against the problem that the remote node knows *when* you are spending and this information can be very useful if your are doing any kind of blockchain analysis / output tracing.

Bottom line, the moment you start using a 3rd party remote node, you are trusting them to some degree and leaking some information.

So whether to use a 3rd party remote node or not for any particular transaction boils down to the level of risk and trust that is acceptable to you. 

Running your own local or remote node is always preferable. Of course there's nothing stopping you having a couple of different wallets. For example, one which is quick and fast (like some of the mobile wallets which use remote node / centralized servers) for buying your coffees, and a separate wallet, using your own node, for transaction that require a higher level of privacy.