How does the payment id appear in a transaction?

Question

I'm having troubles finding the payment ID in a transaction info. I'm using the daemon RPC API to get the transaction info. According to the official documentation the extra field is used for the payment ID. However, It seems like this byte array is not even the size of a valid payment ID (8 or 32 bytes).
Here are two examples:

1. tx hash: c748fedd286bdd5a34e223316593625cd4482d7666b262cef51f8bc0e1694b26
   'extra' field (in hex): 012772e8e0893433644b93401067a804415d86ddde4c91154a95c84a365111fb86022100fb7ef3f11937667a9242ed4d252ed97c4401079bc11d87a17a581032bcb02769 (68 bytes)
2. tx hash: b9678d96071be7d31484c324d2009848872744e79ddd85eb94b71fb867b543cb
   'extra' field (in hex): 01ed37fc961021099a74c227fa414c29a7213e6678215abf5f37a1c25ed4bd26a2 (33 bytes)

I have another transaction without payment ID (shows 0000000000000000 in the cli wallet) but the extra field in this transaction is not filled with 0's, but actually looks random (I'm not posting its details for privacy reasons).

So what exactly is in the extra field, and how do I find the payment ID in the transaction info?

Answer 1

The txextra field can contain several subfields.

Start by reading a single byte. This is the subfield tag.

Tags: 
0 = padding
1 = the next 32 bytes is the transaction public key
2 = a nonce. Read a VarInt[1] to determine the nonce length. Then read a byte to determine the nonce type

Nonce types:
0 = 32 byte unencrypted payment id 
1 = 8 byte encrypted payment id

If there is an encrypted payment id, you need to compute the shared secret for the transaction in order to decrypt it.

To decrypt an encrypted payment id, you need to XOR it with the payment id key. Start with the shared secret, by doing scalar multiplication of your private view key with the transaction public key located in the txextra and then multiplying that by 8. Concatenate the byte representation of the shared secret elliptic curve point with the hex "8d". Take a keccak(256) hash of this, and you have the XOR decryption key for the encrypted payment id.

[1] The VarInt integer representation is defined in section 1.2 of https://tukaani.org/xz/xz-file-format.txt

Also see https://cryptonote.org/cns/cns005.txt and https://github.com/monero-project/monero/blob/1a4298685aa9e694bc555ae69be59d14d3790465/src/cryptonote_basic/tx_extra.h