Complete extra field structure (standard interpretation)

Question

The documentation on transactions' 'extra' field is spread out and not completely clear. This is the original cryptonote description. Here is a different question that reveals some more of the concepts.

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how is it stored in transaction data, and what are the rules and assumptions/conventions surrounding common tags (i.e. how the field gets parsed)?

Answer 1

Things to keep in mind:

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2). Its content is verified by neither the blockchain protocol nor the network consensus, so transaction authors can include anything they want in any order. This answer focuses on how the core implementation parses the field.
3. It contains information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Best practice: wallet designers should understand the rules for all known tags, in order to mitigate interference with the expectations and behavior of other wallets (especially the core wallet). In particular, non-standard tags and other information should be placed after any standard information.
6. Bytes to be interpreted as 'length' or 'amount' are considered varints. Varints can theoretically have unlimited bytes, but in Monero transactions are limited to 9 bytes long, and the subsequent byte is included only if the current byte's most significant bit is 1 (the end of the varint is either the byte with MSB not set or the 9th byte). It is an integer with up to 63 bits of information. See src/common/varint.h.
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field began with a varint equal to the field's number of elements. Monero does not do that.
8. The standard implementation interprets non-standard extra fields to the best of its ability. The parser works like this. It looks at the first byte. If it is a known tag then if the following bytes obey the tag's rules, save the tag's bytes for use elsewhere and skip ahead to the next available byte and check if it's a known tag. If an examined byte is not a known tag (specifically 0x00, 0x01, or 0x04), then according to this it will interpret the next byte(s) as a varint, skip that number of bytes, and examine the next available byte.

Basic/standard tags (in hexidecimal format):

• 0x00: padding, ignore and move to the next byte; the core implementation does not add any of these to the extra field (there seems to be a 255 consecutive padding byte limit, although not clear how this affects the core parser)
• 0x01: public key, saves the next 32 bytes if they can be converted into an elliptic curve point; most transactions will have one of these, and it corresponds to the transaction public key; when going through an extra field, the wallet will save every valid public key it finds in order to use them to test for output ownership
• 0x02: extra nonce, the next byte is its length (not a varint, just an 8bit unsigned integer; may be up to 255); this is like an extra field within the extra field, and is often used by miner pools to prevent nonce overlap, and can be used to extend the proof of work nonce size to larger than 32 bits (although unlikely to be used, as a typical miner can only get through around 1/1000th or less of the available nonce values each block before resetting; see this site for current hash speeds); if the transaction type is non-miner (RCTTypeNull) the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below); if the transaction type IS RCTTypeNull, there is no consistent or standard useage
• 0x03: merge mining tag; this doesn't seem to be used by anyone any more
• 0x04: additional public keys, the next byte is the number of additional keys (must be convertible to elliptic curve points, 32 bytes each), and the byte after that is the most significant byte of the first additional key; usually only one of these sets per transaction, and there should typically be either 1 or #outputs total amount of public keys per transaction (standard single transaction public key, or at-least-one-ouput-is-a-subaddress case with #outputs transaction public keys)
• 0xDE: mysterious minergate tag

Note: the core implementation sorts tag fields for transactions being constructed in this order [0x01, 0x04, 0x02, 0x03, 0xDE, 0x00] (see function sort_tx_extra()). It does not support other tag fields or random data, although alternative implementations can and do edit the sort function (if they use it at all) to accommodate their own tags.

Special extra nonce tags (tags within the extra nonce section, likely only relevant for non-miner/coinbase transactions):

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID (note: the core implementation adds a dummy encrypted payment ID to transactions with two outputs if one isn't specified, to improve transaction indistinguishability); only the first one of these found will be used and reported to the user if they happen to own an output in the same transaction (my understanding of best practice is encrypted payment IDs should only be included in 2-output transactions, and that the 'change'-output will ignore payment IDs)

An example from a transaction in block 2004288 (transaction hash 3033917f822512e7b9dab4eb3477cd9ec61052224723623c1353f4d160e8dd68):

"extra":[1,240,78,249,111,229,39,232,253,137,61,146,133,192,190,13,58,96,239,78,21,116,133,204,111,251,40,170,155,126,208,23,244,2,9,1,253,234,238,74,23,163,141,203] = [1 (transaction public key TAG), ...32 bytes..., 2 (extra nonce TAG), 9 (length of extra nonce), 1 (extra nonce encrypted payment ID TAG), ...8 bytes... ]

P.S. Thanks to jtgrassie for his patience assisting with this topic.

Answer 2

0x01: transaction public key...

Incorrect. 0x01 is a public key. Not specifically a transaction public key. The wallets may stuff other public keys in extra, as an example, for multi-sig needs.

Open question(s): If e.g. there are two transaction public keys with tag 0x01, how will the core implementation interpret this?

Tag 0x01 is not defined as "transaction" public key, just as public key. Thus, when anything needs the transaction public key, it has to try each public key found in extra until it finds one that works for them.

Will it ignore the first, or second one, or try to use both of them? What about other tag multiples?

No. The field extra is essentially freeform. You don't even have to tag data in it, this is just how the wallets are using it. Therefore anything that depends on data stuffed into extra has to deal with this fact - the core wallets included.

...merge mining tag, how does this one work?

It's not currently used by anything (that I'm aware of at least).