1. Can ZCoin protect the privacy of transaction amounts like Monero will do with RingCT
No, it cannot. All transaction amounts are visible on the ZCoin blockchain. As of the block 1,220,517 v4 fork in early January, RingCT transactions will be allowed on the Monero network. RingCT transactions hide all Monero transaction amounts but will not become mandatory until the v5 fork in September, 2017.
ZCoin (formerly known as Moneta) developers are independent from the academics that created the Zerocoin or the more recent Zerocash protocol. Some key distinctions, as clarified by Zcash developer, Daira Hopwood:
BTW, in case anyone is confused, Zcoin is a renaming of Moneta. It would have been much clearer if the article had said that. It's based on development of the original Zerocoin prototype code, and has nothing to do with Zcash.
Zerocoin/Moneta/Zcoin uses RSA-based ZK proofs. Unlike Zcash, separate parameters are needed for each denomination of coin, and there is information leakage about the denominations used. Proofs are about 45 KiB (not even taking into account the need for multiple proofs for the denominated coins) as opposed to 296 bytes for the raw proof in Zcash.
More background can be found in this AMA with Moneta/ZCoin co-founder Gary Le.
2.What are the key security risks associated with each design? Does ZCoin require a trusted setup or is it trustless like Monero?
ZCoin required a trusted setup and relied on the parameters created by the RSA Factoring Challenge in 1991. According to the parameter generation discussion on the ZCoin website:
At projected computing capacity, it will be safe to use for many more decades. By then, Zcoin can port its parameters to a new cryptographic scheme. Admittedly, the RSA has not been a great company in recent years, with revelations of the RSA collaborating with the NSA. But the keys to the RSA Factoring challenge were generated in 1991, early in the RSA days when the creators of the RSA algorithm still had a high amount of control over their company.
When asked why ZCoin relied on this RSA accumulator setup and hoping that the parameter keys created by someone else (RSA Laboratories) where destroyed, instead of creating their trusted setup similar to Zcash, Gary Le said
There is this thing called a trustless RSA setup but it is very inefficient and would lead essentially to blocktimes that are hours long... In order to avoid that and make this actual cryptocurrency that you can send transactions with we had to use this previously generated trustless[ed] setup by RSA
To say the least, its not the best solution, but it is not no worse and no better than Zcash trusless[ed] setup.
The above "trustless" quotes (instead of trusted) is misleading just like the language on the Zcoin website:
On the other hand, there is a strong mitigating factor in the unlikely scenario of a compromised Zcoin setup: everybody could still see that Zcoin’s money supply checks out.
The above quote seems to falsely imply that a compromised setup could not impact the money supply, like it could with Zcash. A more accurate statement based on the Zcoin whitepaper would be:
"On the other hand, there is a mitigating factor in the scenario of a compromised Zcoin setup: everybody can still check at any time to see whether or not Zcoin’s money supply matches its planned emission curve or if additional Zcoin's have been created as a result of the compromise"
3.How does the PoW algorithm of ZCoin compare to that of Monero?
ZCoin uses Lyra2 and Monero uses CryptoNight both of which are meant to be ASIC resistant.