Return to Question

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how is it stored in transaction data, and what are the rules and assumptions/conventions surrounding common tags (i.e. how the field gets parsed)? What follows is my current understanding.

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2).
3. It contains information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Wallet designers should understand the rules for all known tags, in order to mitigate interference with the expectations and behavior of other wallets (especially the core wallet).
6. Bytes to be interpreted as 'length' or 'amount' are considered varints.
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field was prefixed with a varint (a normal varint occupies up to 8 bytes, with up to 57 bits of information) equal to the field's number of elements. Monero does not do that.
8. The standard implementation interprets non-standard extra fields to the best of its ability. The parser works like this. It looks at the first byte. If it is a known tag then if the following bytes obey the tag's rules save the tag's bytes for use elsewhere and skip ahead to the next available byte and check if it's a known tag. If an examined byte is not a known tag, look at the next byte.

Basic tags (in hexidecimal format):

• 0x00: padding, ignore and move to the next byte; there can be up to 255 of these in an extra field
• 0x01: public key, saves the next 32 bytes if they can be converted into an elliptic curve point; most transactions will have one of these, and it corresponds to the transaction public key; when going through an extra field, the wallet will save every valid public key it finds in order to use them to test for output ownership
• 0x02: extra nonce, the next byte is its length (may be up to 255); this is like an extra field within the extra field, and the utility of that is unclear to me; the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below)
• 0x03: merge mining tag; this doesn't seem to be used by anyone any more
• 0x04: additional public keys, the next byte is the number of additional keys (must be convertible to elliptic curve points, 32 bytes each), and the byte after that is the most significant byte of the first additional key; usually only one of these sets per transaction, and there should typically be either 1 or #outputs total amount of public keys per transaction (standard single transaction public key, or at-least-one-ouput-is-a-subaddress case with #outputs transaction public keys)
• 0xDE: mysterious minergate tag

Special extra nonce tags (tags within the extra nonce section):

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID (note: the core implementation adds a dummy encrypted payment ID to transactions with two outputs if one isn't specified, to improve transaction indistinguishability); only the first one of these found will be used and reported to the user if they happen to own an output in the same transaction (my understanding of best practice is encrypted payment IDs should only be included in 2-output transactions, and that the 'change'-output will ignore payment IDs)

An example from a transaction in block 2004288 (transaction hash 3033917f822512e7b9dab4eb3477cd9ec61052224723623c1353f4d160e8dd68):

"extra":[1,240,78,249,111,229,39,232,253,137,61,146,133,192,190,13,58,96,239,78,21,116,133,204,111,251,40,170,155,126,208,23,244,2,9,1,253,234,238,74,23,163,141,203] = [1 (transaction public key TAG), ...32 bytes..., 2 (extra nonce TAG), 9 (length of extra nonce), 1 (extra nonce encrypted payment ID TAG), ...8 bytes... ]

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how is it stored in transaction data, and what are the rules and assumptions/conventions surrounding common tags (i.e. how the field gets parsed)?

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how it is stored in transaction data, and the rules surrounding common tags.

Qualifying answers may verify, correct, or add to the following information (note: the answer should include all good information from below; once answered, I will edit the question down to just the question).

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2).
3. It contains information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Wallet designers should understand the rules for all known tags, in order to mitigate interference with the expectations and behavior of other wallets (especially the core wallet).
6. Bytes to be interpreted as 'length' or 'amount' are considered varints.
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field was prefixed with a varint (a normal varint occupies up to 9 bytes, with up to 63 bits of information; see section 1.2) equal to the field's number of elements. Monero does not do that.
8. The standard implementation interprets non-standard extra fields to the best of its ability. In other words, it will scan the entire extra field until it finds the tag it wants, and if the following bytes obey pertinent rules then it considers the tag's information field 'found'. Different tags have different rules pertaining to 'multiple information fields with the same tag'. More details below.

• 0x00: padding, ignore and move to the next byte; there can be up to 255 of these in an extra field
• 0x01: public key, the next 32 bytes is that key (an elliptic curve point) in byte format (it can be converted to a group element); most transactions will have one of these, and it corresponds to the transaction public key; when scanning a transaction for owned outputs, a wallet will find every possible public key within the extra field and use it to test each of the transaction's outputs (i.e. for every byte equal to 0x01, if there are at least 32 bytes after it that can be interpreted as a public key the wallet will test it)
• 0x02: extra nonce, the next byte is its length (may be up to 255); this is like an extra field within the extra field, and the utility of that is unclear to me; the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below); only the first one of these, where the second byte is less than or equal to the remaining number of bytes, will be interpreted by the wallet
• 0x03: merge mining tag; this doesn't seem to be used by anyone any more
• 0x04: additional public keys, the next byte is the number of additional keys (elliptic curve points in bytes, 32 bytes each), and the byte after that is the most significant byte of the first additional key; usually only one of these sets per transaction, and there should typically be either 1 or #outputs total amount of public keys per transaction (standard single transaction public key, or at-least-one-ouput-is-a-subaddress case with #outputs transaction public keys); wallet will look for all legitimate patterns, and test all available public keys when searching for owned outputs
• 0xDE: mysterious minergate tag

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet, i.e. treated as padding
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID; wallet will only recognize the first one of these in a transaction (note: the core implementation adds a dummy encrypted payment ID to transactions where one isn't specified, to improve transaction indistinguishability)

"extra":[1,240,78,249,111,229,39,232,253,137,61,146,133,192,190,13,58,96,239,78,21,116,133,204,111,251,40,170,155,126,208,23,244,2,9,1,253,234,238,74,23,163,141,203] = [1 (transaction public key TAG), ...32 bytes..., 2 (extra nonce TAG), 9 (length of extra nonce), 1 (extra nonce encrypted payment ID TAG), ...8 bytes... ]

Open question(s): If e.g. there are two transaction public keys with tag 0x01, how will the core implementation interpret this? Will it ignore the first, or second one, or try to use both of them? What about other tag multiples?

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how is it stored in transaction data, and what are the rules and assumptions/conventions surrounding common tags (i.e. how the field gets parsed)? What follows is my current understanding.

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2).
3. It contains information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Wallet designers should understand the rules for all known tags, in order to mitigate interference with the expectations and behavior of other wallets (especially the core wallet).
6. Bytes to be interpreted as 'length' or 'amount' are considered varints.
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field was prefixed with a varint (a normal varint occupies up to 8 bytes, with up to 57 bits of information) equal to the field's number of elements. Monero does not do that.
8. The standard implementation interprets non-standard extra fields to the best of its ability. The parser works like this. It looks at the first byte. If it is a known tag then if the following bytes obey the tag's rules save the tag's bytes for use elsewhere and skip ahead to the next available byte and check if it's a known tag. If an examined byte is not a known tag, look at the next byte.

• 0x00: padding, ignore and move to the next byte; there can be up to 255 of these in an extra field
• 0x01: public key, saves the next 32 bytes if they can be converted into an elliptic curve point; most transactions will have one of these, and it corresponds to the transaction public key; when going through an extra field, the wallet will save every valid public key it finds in order to use them to test for output ownership
• 0x02: extra nonce, the next byte is its length (may be up to 255); this is like an extra field within the extra field, and the utility of that is unclear to me; the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below)
• 0x03: merge mining tag; this doesn't seem to be used by anyone any more
• 0x04: additional public keys, the next byte is the number of additional keys (must be convertible to elliptic curve points, 32 bytes each), and the byte after that is the most significant byte of the first additional key; usually only one of these sets per transaction, and there should typically be either 1 or #outputs total amount of public keys per transaction (standard single transaction public key, or at-least-one-ouput-is-a-subaddress case with #outputs transaction public keys)
• 0xDE: mysterious minergate tag

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID (note: the core implementation adds a dummy encrypted payment ID to transactions with two outputs if one isn't specified, to improve transaction indistinguishability); only the first one of these found will be used and reported to the user if they happen to own an output in the same transaction (my understanding of best practice is encrypted payment IDs should only be included in 2-output transactions, and that the 'change'-output will ignore payment IDs)

"extra":[1,240,78,249,111,229,39,232,253,137,61,146,133,192,190,13,58,96,239,78,21,116,133,204,111,251,40,170,155,126,208,23,244,2,9,1,253,234,238,74,23,163,141,203] = [1 (transaction public key TAG), ...32 bytes..., 2 (extra nonce TAG), 9 (length of extra nonce), 1 (extra nonce encrypted payment ID TAG), ...8 bytes... ]

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation. In other words, how it is stored in transaction data, and the rules surrounding common tags. Qualifying answers may verify, correct, or add to the following information (note: the answer should include all good information from below; once answered, I will edit the question down to just the question).

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2).
3. It is composed of information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Wallets should understand the rules for all known tags, in order to correctly parse the ones they care about.
6. Bytes to be interpreted as 'length' or 'amount' are considered varints (section 1.2).
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field was prefixed with a varint (up to 63 bits) equal to the field's number of elements. Monero does not do that.

• 0x00: padding, ignore and move to the next byte; there can be up to 255 of these
• 0x01: transaction public key, the next 32 bytes is that key (an elliptic curve point) in byte format (it can be converted to a group element); one of these per transaction
• 0x02: extra nonce, the next byte is its length (may be up to 255); this is like an extra field within the extra field, and the utility of that is unclear to me; the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below); one of these per transaction
• 0x03: merge mining tag, how does this one work?
• 0x04: additional public keys, the next byte is the number of additional keys (elliptic curve points in bytes, 32 bytes each), and the byte after that is the most significant byte of the first additional key; one of these sets per transaction, and there should be either 1 or #outputs total amount of transaction public keys per transaction (standard, or at-least-one-ouput-is-a-subaddress cases)
• 0xDE: mysterious minergate tag, the next byte is its length

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; one of these per transaction (deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet)
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID; one of these per transaction (the core implementation will add a dummy encrypted payment ID to transactions where one isn't specified, to improve transaction indistinguishability)

I would like to devote this question to listing all information about the 'extra' field, and its standard interpretation and implementation in core software v0.15.x.x. In other words, how it is stored in transaction data, and the rules surrounding common tags. 

Qualifying answers may verify, correct, or add to the following information (note: the answer should include all good information from below; once answered, I will edit the question down to just the question).

1. The field's location within transaction data is defined by the transaction's structure. Currently (protocol v12) the transaction version is v2, and 'extra' lives between 'vout' (outputs) and 'rct_signatures' (ringct data).
2. The field is interpreted as an array of bytes with little endian order (indexed 0, 1, 2).
3. It contains information sections, which each begin with a 'tag' byte. A tag defines how the bytes after it are interpreted. Wherever relevant, the leftmost byte (lower index) is considered most significant.
4. The field's core implementation can be found here: src/cryptonote_basic/tx_extra.h
5. Wallet designers should understand the rules for all known tags, in order to mitigate interference with the expectations and behavior of other wallets (especially the core wallet).
6. Bytes to be interpreted as 'length' or 'amount' are considered varints.
7. The field may have unlimited elements, up to the functional limit of its transaction's maximum weight. Note: in the original CryptoNote specification the extra field was prefixed with a varint (a normal varint occupies up to 9 bytes, with up to 63 bits of information; see section 1.2) equal to the field's number of elements. Monero does not do that.
8. The standard implementation interprets non-standard extra fields to the best of its ability. In other words, it will scan the entire extra field until it finds the tag it wants, and if the following bytes obey pertinent rules then it considers the tag's information field 'found'. Different tags have different rules pertaining to 'multiple information fields with the same tag'. More details below.

• 0x00: padding, ignore and move to the next byte; there can be up to 255 of these in an extra field
• 0x01: public key, the next 32 bytes is that key (an elliptic curve point) in byte format (it can be converted to a group element); most transactions will have one of these, and it corresponds to the transaction public key; when scanning a transaction for owned outputs, a wallet will find every possible public key within the extra field and use it to test each of the transaction's outputs (i.e. for every byte equal to 0x01, if there are at least 32 bytes after it that can be interpreted as a public key the wallet will test it)
• 0x02: extra nonce, the next byte is its length (may be up to 255); this is like an extra field within the extra field, and the utility of that is unclear to me; the byte after its length is an 'extra nonce tag' interpreted according to special rules (see below); only the first one of these, where the second byte is less than or equal to the remaining number of bytes, will be interpreted by the wallet
• 0x03: merge mining tag; this doesn't seem to be used by anyone any more
• 0x04: additional public keys, the next byte is the number of additional keys (elliptic curve points in bytes, 32 bytes each), and the byte after that is the most significant byte of the first additional key; usually only one of these sets per transaction, and there should typically be either 1 or #outputs total amount of public keys per transaction (standard single transaction public key, or at-least-one-ouput-is-a-subaddress case with #outputs transaction public keys); wallet will look for all legitimate patterns, and test all available public keys when searching for owned outputs
• 0xDE: mysterious minergate tag

• 0x00: payment ID, the next 32 bytes are a payment ID in clear text; deprecated as of core implementation v0.15, which means it will be ignored/skipped over by the main wallet, i.e. treated as padding
• 0x01: encrypted payment ID, the next 8 bytes are an encrypted payment ID; wallet will only recognize the first one of these in a transaction (note: the core implementation adds a dummy encrypted payment ID to transactions where one isn't specified, to improve transaction indistinguishability)