Skip to content

Scrape fails after 46,147 blocks - receipt_data is NoneType #62

@whydna

Description

@whydna

Describe the bug
Setup went smoothly using a local Geth node. Ran ether_sql scrape_block_range and everything seemed to be working smoothly. It now seems to be stuck on block 46,147 with the following error every time I try to run it:

(venv) user1@full-node-1:~/ether_sql$ ether_sql scrape_block_range
[2021-06-25 13:45:37,600][INFO][ether_sql.session] Connected to the db ether_sql
[2021-06-25 13:45:37,604][INFO][ether_sql.session] Connected to Geth node
[2021-06-25 13:45:37,604][INFO][ether_sql.globals] Pushing the session DefaultSettings in local thread
[2021-06-25 13:45:39,128][INFO][ether_sql.cli] 0 missing blocks detected
Traceback (most recent call last):
  File "/home/user1/venv/bin/ether_sql", line 33, in <module>
    sys.exit(load_entry_point('ether-sql', 'console_scripts', 'ether_sql')())
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/cli/__init__.py", line 91, in scrape_block_range
    mode=mode)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 66, in scrape_blocks
    add_block_number(block_number)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/local.py", line 193, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 143, in add_block_number
    timestamp=iso_timestamp)
  File "/home/user1/ether_sql/ether_sql/models/receipts.py", line 68, in add_receipt
    receipt = cls(transaction_hash=to_hex(receipt_data['transactionHash']),
TypeError: 'NoneType' object is not subscriptable

If I skip that block by running: scrape_block_range --start_block_number 46148

It will scrape a few more blocks and then fail on the next one without receipt data:

[2021-06-25 13:49:25,923][INFO][ether_sql.tasks.scrapper] Commiting block: 46162 to sql
[2021-06-25 13:49:26,020][INFO][ether_sql.tasks.scrapper] Commiting block: 46163 to sql
[2021-06-25 13:49:26,094][INFO][ether_sql.tasks.scrapper] Commiting block: 46164 to sql
[2021-06-25 13:49:26,267][INFO][ether_sql.tasks.scrapper] Commiting block: 46165 to sql
[2021-06-25 13:49:26,334][INFO][ether_sql.tasks.scrapper] Commiting block: 46166 to sql
[2021-06-25 13:49:26,414][INFO][ether_sql.tasks.scrapper] Commiting block: 46167 to sql
[2021-06-25 13:49:26,465][INFO][ether_sql.tasks.scrapper] Commiting block: 46168 to sql
Traceback (most recent call last):
  File "/home/user1/venv/bin/ether_sql", line 33, in <module>
    sys.exit(load_entry_point('ether-sql', 'console_scripts', 'ether_sql')())
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/cli/__init__.py", line 91, in scrape_block_range
    mode=mode)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 66, in scrape_blocks
    add_block_number(block_number)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/local.py", line 193, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 143, in add_block_number
    timestamp=iso_timestamp)
  File "/home/user1/ether_sql/ether_sql/models/receipts.py", line 68, in add_receipt
    receipt = cls(transaction_hash=to_hex(receipt_data['transactionHash']),
TypeError: 'NoneType' object is not subscriptable

To Reproduce
Steps to reproduce the behavior:
Just run the the scraper.

Expected behavior
It should handle the case where receipt_data is missing?

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari
  • Version [e.g. 22]
  • Node [e.g. Parity, Infura, Geth]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions