3

Synchronization is stuck at block height 1200001 (98.5%).

Running monerod in version monero.win.x64.v0-10-1-0 it says You are now synchronized with the network. However, I am getting the following in the logs saying the node is 24 days behind, followed by a bunch of output that says key does not exist. Is this related to the upcoming hardfork or what? It seems as if it has been stuck at 24 days behind. What are these exceptions flooding the logs?

Edit: I updated from version 9.4 to 10.1, skipping version 10.0. It had to update the LMDB blockchain data, which seemed to work fine. Then it synchronized about two months of blockchain data via the net. Then it got stuck at block 1200001.

Here is what I get with log-level=1: 

2017-Jan-06 00:53:11.382858 [P2P9][XXX.XXX.XXX.XXX:18080 OUT]Sync data returned a new top block candidate: 1200001 -> 1217764 [Your node is 17763 blocks (24 days) behind] 
SYNCHRONIZATION started
2017-Jan-06 00:53:11.382858 [P2P9]Remote blockchain height: 1217764, id: <314feaf707ed226c391801daf4aab47719868bb7eb3c2d0828d384cb9acdac13>
2017-Jan-06 00:53:11.382858 [P2P9][XXX.XXX.XXX.XXX:18080 OUT] COMMAND_HANDSHAKE INVOKED OK
2017-Jan-06 00:53:14.097263 [P2P8][XXX.XXX.XXX.XXX:18080 OUT]-->>NOTIFY_REQUEST_GET_OBJECTS: blocks.size()=200, txs.size()=0requested blocks count=200 / 200
2017-Jan-06 00:53:15.017665 [P2P9][XXX.XXX.XXX.XXX:18080 OUT]Got NEW BLOCKS inside of handle_response_get_objects: size: 200
2017-Jan-06 00:53:15.017665 [P2P9]block_batches: 100
2017-Jan-06 00:53:25.266883 Attempting to get output pubkey by global index (amount 9, index 11629, count 11617), but key does not exist
2017-Jan-06 00:53:25.266883 EXCEPTION: Attempting to get output pubkey by global index (amount 9, index 11629, count 11617), but key does not exist
2017-Jan-06 00:53:25.282483 Attempting to get output pubkey by global index (amount 60, index 10128, count 10127), but key does not exist
2017-Jan-06 00:53:25.282483 EXCEPTION: Attempting to get output pubkey by global index (amount 60, index 10128, count 10127), but key does not exist
2017-Jan-06 00:53:25.344883 Attempting to get output pubkey by global index (amount 5000, index 10756, count 10737), but key does not exist
2017-Jan-06 00:53:25.344883 EXCEPTION: Attempting to get output pubkey by global index (amount 5000, index 10756, count 10737), but key does not exist
2017-Jan-06 00:53:25.360483 Attempting to get output pubkey by global index (amount 20000, index 15171, count 15139), but key does not exist
2017-Jan-06 00:53:25.360483 EXCEPTION: Attempting to get output pubkey by global index (amount 20000, index 15171, count 15139), but key does not exist
2017-Jan-06 00:53:25.376083 Attempting to get output pubkey by global index (amount 10000, index 14664, count 14630), but key does not exist
2017-Jan-06 00:53:25.376083 EXCEPTION: Attempting to get output pubkey by global index (amount 10000, index 14664, count 14630), but key does not exist
2017-Jan-06 00:53:25.376083 Attempting to get output pubkey by global index (amount 30000, index 14814, count 14791), but key does not exist
2017-Jan-06 00:53:25.376083 EXCEPTION: Attempting to get output pubkey by global index (amount 30000, index 14814, count 14791), but key does not exist
2017-Jan-06 00:53:25.376083 Attempting to get output pubkey by global index (amount 40000, index 15076, count 15056), but key does not exist

It continues with similar messages until: 

2017-Jan-06 00:53:26.748885 EXCEPTION: Attempting to get output pubkey by global index (amount 1000000000000000, index 10464, count 9509), but key does not exist
2017-Jan-06 00:53:26.748885 Attempting to get output pubkey by global index (amount 10000000000000000, index 670, count 650), but key does not exist
2017-Jan-06 00:53:26.748885 EXCEPTION: Attempting to get output pubkey by global index (amount 10000000000000000, index 670, count 650), but key does not exist
2017-Jan-06 00:53:28.823689 [P2P9]transaction with hash 84ae0c3c70ec65b11dc174f340341a1cfe2e9f2af07beeb87ded9f1e5a977b2c not found in db
2017-Jan-06 00:53:28.839289 [P2P5][XXX.XXX.XXX.XXX:18080 OUT]Got NEW BLOCKS inside of handle_response_get_objects: size: 200
2017-Jan-06 00:53:28.839289 [P2P9]Additional outputs needed: 1
2017-Jan-06 00:53:28.839289 [P2P9]Attempting to get output pubkey by global index (amount 7000000000, index 231993, count 201901), but key does not exist
2017-Jan-06 00:53:28.839289 [P2P9]Output does not exist! amount = 7000000000
2017-Jan-06 00:53:28.839289 [P2P9]Failed to get output keys for tx with amount = 0.007000000000 and count indexes 4
2017-Jan-06 00:53:28.839289 [P2P9]Failed to check ring signature for tx <84ae0c3c70ec65b11dc174f340341a1cfe2e9f2af07beeb87ded9f1e5a977b2c>  vin key with k_image: <751c67c732b9713ae468bf79665aa226ae7db0fd73a7cca66e3c0e154adecb87>  sig_index: 0
2017-Jan-06 00:53:28.839289 [P2P9]Block with id: <9df518e013515cb091a21f568a3152b6126b4e4e9d64380d36ccbcd920a72fc9> has at least one transaction (id: <84ae0c3c70ec65b11dc174f340341a1cfe2e9f2af07beeb87ded9f1e5a977b2c>) with wrong inputs.
2017-Jan-06 00:53:28.839289 [P2P9]BLOCK ADDED AS INVALID: <9df518e013515cb091a21f568a3152b6126b4e4e9d64380d36ccbcd920a72fc9>
, prev_id=<fa7d13a90850882060479d100141ff84286599ae39c3277c8ea784393f882d1f>, m_invalid_blocks count=1
2017-Jan-06 00:53:28.839289 [P2P9]Block with id <9df518e013515cb091a21f568a3152b6126b4e4e9d64380d36ccbcd920a72fc9> added as invalid because of wrong inputs in transactions
2017-Jan-06 00:53:28.854889 [P2P9]Additional outputs needed: 1
2017-Jan-06 00:53:28.854889 [P2P9]Attempting to get output pubkey by global index (amount 7000000000, index 231993, count 201901), but key does not exist
2017-Jan-06 00:53:28.854889 [P2P9]Output does not exist! amount = 7000000000
2017-Jan-06 00:53:28.854889 [P2P9]Failed to get output keys for tx with amount = 0.007000000000 and count indexes 4
2017-Jan-06 00:53:28.854889 [P2P9]Failed to check ring signature for tx <84ae0c3c70ec65b11dc174f340341a1cfe2e9f2af07beeb87ded9f1e5a977b2c>  vin key with k_image: <751c67c732b9713ae468bf79665aa226ae7db0fd73a7cca66e3c0e154adecb87>  sig_index: 0
2017-Jan-06 00:53:28.854889 [P2P9]  *pmax_used_block_height: 0
2017-Jan-06 00:53:28.854889 [P2P5]block_batches: 100
2017-Jan-06 00:53:28.854889 [P2P5]Skipping prepare blocks. Blocks exist.
2017-Jan-06 00:53:28.854889 [P2P5]Block recognized as orphaned and rejected, id = <069cfbd5364f72888fd0cc5d380dd519f21661f7edf75b6cb503fb43ee39a0fe>
2017-Jan-06 00:53:28.854889 [P2P5][XXX.XXX.XXX.XXX:18080 OUT]Block received at sync phase was marked as orphaned, dropping connection
2017-Jan-06 00:53:28.854889 [P2P5]IP XXX.XXX.XXX.XXX fail score=1
2017-Jan-06 00:53:28.870489 [P2P9][XXX.XXX.XXX.XXX:18080 OUT]Block verification failed, dropping connection
2017-Jan-06 00:53:28.870489 [P2P9]IP XXX.XXX.XXX.XXX fail score=1
2017-Jan-06 00:53:32.224495 [P2P3]Connecting to XXX.XXX.XXX.XXX:18080(white=0, last_seen: d0.h0.m1.s2)...
2017-Jan-06 00:53:35.781301 Failed to invoke command 1001 return code -3
2017-Jan-06 00:53:35.796901 [XXX.XXX.XXX.XXX:18080 OUT]COMMAND_HANDSHAKE invoke failed. (-3, LEVIN_ERROR_CONNECTION_DESTROYED)
2017-Jan-06 00:53:35.796901 [P2P3][XXX.XXX.XXX.XXX:18080 OUT]COMMAND_HANDSHAKE Failed
2017-Jan-06 00:53:35.796901 [P2P3][XXX.XXX.XXX.XXX:18080 OUT]Failed to HANDSHAKE with peer XXX.XXX.XXX.XXX:18080

Running with default log-level I get these missed transaction errors: 

You are now synchronized with the network. You may now start monero-wallet-cli.

Please note, that the blockchain will be saved only after you quit the daemon with "exit" command or if you use "save" command.
Otherwise, you will possibly need to synchronize the blockchain again.

Use "help" command to see the list of available commands.
**********************************************************************
2017-Jan-06 01:50:18.282661 [P2P9]ERROR C:/msys64/DISTRIBUTION-BUILD/src/cryptonote_core/blockchain.cpp:1481 Error retrieving blocks, missed 1 transactions for block with hash: <5a210d06d04e82e18a2da4b85a37d6302516a9b131ea702cb015f65b05490bef>

Edit: Getting a whole lot of this in the logs. Really hoping I can skip this block or fix this without having to sync the entire blockchain again.

SYNCHRONIZATION started
2017-Jan-07 01:52:35.054848 [P2P6]Remote blockchain height: 1218478, id: <c5ef2e523c0f0761992199d8c5e8f8ab033349df6b2c5114e32927a86b86804a>
2017-Jan-07 01:52:35.054848 [P2P6][XXX.XXX.XXX.XXX:18080 OUT]requesting callback
2017-Jan-07 01:52:35.054848 [P2P6][XXX.XXX.XXX.XXX:18080 OUT] COMMAND_HANDSHAKE INVOKED OK
2017-Jan-07 01:52:35.054848 [P2P5][XXX.XXX.XXX.XXX:18080 OUT]callback fired
2017-Jan-07 01:52:35.054848 [P2P5][XXX.XXX.XXX.XXX:18080 OUT]-->>NOTIFY_REQUEST_CHAIN: m_block_ids.size()=31
2017-Jan-07 01:52:35.054848 [P2P5][XXX.XXX.XXX.XXX:18080 OUT] post N10cryptonote20NOTIFY_REQUEST_CHAINE -->
2017-Jan-07 01:52:35.054848 [P2P8][XXX.XXX.XXX.XXX:18080 OUT]CONNECTION HANDSHAKED OK.
2017-Jan-07 01:52:35.054848 [P2P8]Selected peer: 17286354698914956588 XXX.XXX.XXX.XXX:18080[white=1] last_seen: d0.h0.m4.s21
2017-Jan-07 01:52:35.054848 [P2P8]Connecting to XXX.XXX.XXX.XXX:18080(white=1, last_seen: d0.h0.m4.s21)...
2017-Jan-07 01:52:35.054848 [P2P8]connections_ size now 5
2017-Jan-07 01:52:35.070448 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]Block verification failed, dropping connection
2017-Jan-07 01:52:35.070448 [P2P7]IP XXX.XXX.XXX.XXX fail score=1
2017-Jan-07 01:52:35.413649 [P2P8][XXX.XXX.XXX.XXX:18080 d7849c01-da74-ef6d-4863-3836c1fe31e2 OUT] NEW CONNECTION
2017-Jan-07 01:52:36.552451 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]REMOTE PEERLIST: TIME_DELTA: 6, remote peerlist size=85
2017-Jan-07 01:52:36.552451 [P2P7][XXX.XXX.XXX.XXX:18080 OUT] COMMAND_HANDSHAKE INVOKED OK
2017-Jan-07 01:52:36.552451 [P2P8][XXX.XXX.XXX.XXX:18080 OUT]CONNECTION HANDSHAKED OK.
2017-Jan-07 01:52:36.552451 [P2P8]Selected peer: 5095727181020846244 XXX.XXX.XXX.XXX:8180[white=0] last_seen: d0.h0.m20.s20
2017-Jan-07 01:52:36.552451 [P2P8]Connecting to XXX.XXX.XXX.XXX:8180(white=0, last_seen: d0.h0.m20.s20)...
2017-Jan-07 01:52:36.552451 [P2P8]connections_ size now 6
2017-Jan-07 01:52:36.739651 [P2P8][XXX.XXX.XXX.XXX:8180 26474ee4-683a-eb18-0515-7a9282c40a25 OUT] NEW CONNECTION
2017-Jan-07 01:52:36.786451 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]NOTIFY_REQUEST_CHAIN: m_block_ids.size()=25
2017-Jan-07 01:52:36.817651 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]-->>NOTIFY_RESPONSE_CHAIN_ENTRY: m_start_height=25434, m_total_height=1200001, m_block_ids.size()=10000
2017-Jan-07 01:52:36.817651 [P2P7][XXX.XXX.XXX.XXX:18080 OUT] post N10cryptonote27NOTIFY_RESPONSE_CHAIN_ENTRYE -->
2017-Jan-07 01:52:36.911251 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]NOTIFY_RESPONSE_CHAIN_ENTRY: m_block_ids.size()=10000, m_start_height=1200000, m_total_height=1218478
2017-Jan-07 01:52:36.958051 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]-->>NOTIFY_REQUEST_GET_OBJECTS: blocks.size()=200, txs.size()=0requested blocks count=200 / 200
2017-Jan-07 01:52:36.958051 [P2P7][XXX.XXX.XXX.XXX:18080 OUT] post N10cryptonote26NOTIFY_REQUEST_GET_OBJECTSE -->
2017-Jan-07 01:52:37.020452 [P2P4]Failed to invoke command 1007 return code -6
2017-Jan-07 01:52:37.020452 [P2P4][XXX.XXX.XXX.XXX:18080 OUT]COMMAND_REQUEST_SUPPORT_FLAGS invoke failed. (-6, LEVIN_ERROR_CONNECTION_HANDLER_NOT_DEFINED)
2017-Jan-07 01:52:38.939255 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]NOTIFY_RESPONSE_GET_OBJECTS
2017-Jan-07 01:52:38.986055 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]Got NEW BLOCKS inside of handle_response_get_objects: size: 200
2017-Jan-07 01:52:38.986055 [P2P7]block_batches: 100
2017-Jan-07 01:52:38.986055 [P2P7]Skipping prepare blocks. New blocks don't belong to chain.
2017-Jan-07 01:52:38.986055 [P2P7]tx <80d5129ded5de64a51c507d146035ce9a14cd32037034549a8a5de9a7900e5c9>already have transaction in tx_pool
2017-Jan-07 01:52:38.986055 [P2P7]Block recognized as orphaned and rejected, id = <069cfbd5364f72888fd0cc5d380dd519f21661f7edf75b6cb503fb43ee39a0fe>
2017-Jan-07 01:52:38.986055 [P2P7][XXX.XXX.XXX.XXX:18080 OUT]Block received at sync phase was marked as orphaned, dropping connection
2017-Jan-07 01:52:38.986055 [P2P7]IP XXX.XXX.XXX.XXX fail score=1
2017-Jan-07 01:52:41.747260 [P2P7][XXX.XXX.XXX.XXX:8180 OUT]Timeout on invoke operation happened, command: 1001
2017-Jan-07 01:52:41.747260 [P2P7]Failed to invoke command 1001 return code -4
2017-Jan-07 01:52:41.747260 [P2P7][XXX.XXX.XXX.XXX:8180 OUT]COMMAND_HANDSHAKE invoke failed. (-4, LEVIN_ERROR_CONNECTION_TIMEDOUT)
2017-Jan-07 01:52:41.747260 [P2P8][XXX.XXX.XXX.XXX:8180 OUT]COMMAND_HANDSHAKE Failed
2017-Jan-07 01:52:41.747260 [P2P8][XXX.XXX.XXX.XXX:8180 OUT]Failed to HANDSHAKE with peer XXX.XXX.XXX.XXX:8180
2017-Jan-07 01:52:41.747260 [P2P8]Selected peer: 5761827740902656532 XXX.XXX.XXX.XXX:18080[white=0] last_seen: d0.h0.m3.s24
2017-Jan-07 01:52:41.747260 [P2P8]Connecting to XXX.XXX.XXX.XXX:18080(white=0, last_seen: d0.h0.m3.s24)...
2017-Jan-07 01:52:41.762860 [P2P8]connections_ size now 7
Improve this question
4
  • 1
    Seems like some invalid blocks in your chain. I think that the key doesn't exist would imply a recent block is referring to some output in an old block which doesn't exist on your copy so it fails verification and is stuck. I have no idea how this could have happened. You could get the blockchain.raw from getmonero.org and import anew from scratch if you want to save bandwidth.
    – JollyMort
    Commented Jan 8, 2017 at 12:02
  • The full background is that I upgraded from version 0.9.4 and it updated the db from version 0 to 1. That seemed to be successful, but it was still over 100 days behind the blockchain and it was able to sync from the net up until block 2100001. I skipped version 0.10.0 and went straight to 0.10.1 Does that give you any ideas how it got mixed up? I ended up just deleting the lmbd data and syncing the entire blockchain from the net. It's currently about halfway done.
    – iyrin
    Commented Jan 8, 2017 at 12:11
  • @JollyMort I found a comment from someone who faced the same issue with being stuck on block 1200001. I've also noticed since starting over that it is Loading precomputed blocks: 1200001. So is it likely that this is related to precomputed blocks from the older version 9.4?
    – iyrin
    Commented Jan 10, 2017 at 10:35
  • Dunno, maybe @user36303 would know the answer.
    – JollyMort
    Commented Jan 10, 2017 at 10:59

1 Answer 1

Reset to default
2

It's not clear whether this is OK or not.

It could be a corrupt blockchain, as mentioned by others.

However, it could be that a transaction really was made referencing invalid outputs, and thus retrieving them fails. This might even plausibly happen when there is a large reorganization (such large ones are pretty rare, though).

In addition, when syncing 200 blocks at a time, this can also happen if spending outputs which are not on the current blockchain, but which are on an earlier of those 200 blocks (ie, spending outputs that were just received).

If this happens and your node fails to continue syncing with the network, then you know something's amiss. However, these messages do not by themselves show 100% evidence of something being wrong.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.