Monero transactions create outputs, which are where funds are stored. Each output has a public key, also known as a one-time public key and sometimes as a "stealth address".
If Monero was like Bitcoin, then to spend the output you would demonstrate knowledge of the private key associated with that public key. This is achieved using a signature.
But Monero prevents observers from knowing which output is really being spent. Therefore instead of providing a signature proving knowledge of the private key corresponding to one particular public key, a ring signature is provided which proves knowledge of one private key for an explicitly stated list of possible public keys.
Therefore each ring signature explicitly references all of the possible public keys whose outputs could potentially have been spent.
It is not necessary to do any kind of random "trying" of public keys. The list of possible public keys is listed, and no further information about which public key is actually being spent can be known.
Ring signatures are a set of data that can be validated by performing a set of mathematical operations on that data. The operations prove that a private key must have been known for one of the public keys referenced, but it cannot be mathematically determined which. The math is detailed on page 20 onwards of this document: https://github.com/UkoeHB/Monero-RCT-report/blob/master/Zero-to-Monero-1-0-0.pdf