Yes, it's possible to mine on XMR-Stak with CPU, AMD GPUs and Nvidia GPUs in the same system at the same time. Take a look at Monero Mining Wiki on Reddit for more info on building a Monero mining rig and how to configure software.
In your case, the problem isn't mATX board, but how many PCI-E slots on it. Supposing that you're using the only full-size PCI-E x16 slot, maybe you have some PCI-E x1 slots free and you can use them with risers to add other cards to your rig.
If you plan to go big in mining, also a motherboard with many PCI-E slots could be helpful and you will need some risers. Get the ones that fit in a PCI-E x1 slot, since they also fit in the larger PCI-E x16 slots. Lastly, Nvidia 1070 and AMD RX 580 are nice cards for Monero mining and a good starting point for sure, I will remain with them.