- Python versions 3.8 and 3.9 are no longer supported.
bcd
datasource has been removed.- Two internal tables were added,
dipdup_contract_metadata
anddipdup_token_metadata
. - Some methods of
tzkt
datasource have changed their signatures and behavior. - Dummy
advanced.oneshot
config flag has been removed. - Dummy
schema approve --hashes
command flag has been removed. docker init
command has been removed.ReindexingReason
enumeration items have been changed.
- Ensure that you have a
python = "^3.10"
dependency inpyproject.toml
. - Remove
bcd
datasources from config. Usemetadata
datasource instead to fetch contract and token metadata. - Update
tzkt
datasource method calls as described below. - Run the
dipdup schema approve
command on every database you use with 5.0.0. - Update usage of
ReindexingReason
enumeration if needed.
Chain reorgs have occurred much recently since the Ithaca protocol reached mainnet. The preferable way to deal with rollbacks is the on_rollback
hook. But if the logic of your indexer is too complex, you can buffer an arbitrary number of levels before processing to avoid reindexing.
datasources:
tzkt_mainnet:
kind: tzkt
url: https://api.tzkt.io
buffer_size: 2
DipDup tries to remove backtracked operations from the buffer instead emitting rollback. Ithaca guarantees operations finality after one block and blocks finality after two blocks, so to completely avoid reorgs, buffer_size
should be 2.
Better Call Dev API was officially deprecated in February. Thus, it's time to go for bcd
datasource. In DipDup, it served the only purpose of fetching contract and token metadata. Now there's a separate metadata
datasource which do the same thing but better. If you have used bcd
datasource for custom requests, see How to migrate from BCD to TzKT API article.
Historically, most TzktDatasource
methods had a page iteration logic hidden inside. The quantity of items returned by TzKT in a single request is configured in HTTPConfig.batch_size
and defaulted to 10.000. Before this release, three requests would be performed by the get_big_map
method to fetch 25.000 big map keys, leading to performance degradation and extensive memory usage.
affected method | response size in 4.x | response size in 5.x |
---|---|---|
get_similar_contracts |
unlimited | max. datasource.request_limit |
get_originated_contracts |
unlimited | max. datasource.request_limit |
get_big_map |
unlimited | max. datasource.request_limit |
get_contract_big_maps |
unlimited | max. datasource.request_limit |
get_quotes |
first datasource.request_limit |
max. datasource.request_limit |
All paginated methods now behave the same way. You can either iterate over pages manually or use iter_...
helpers.
datasource = ctx.get_tzkt_datasource('tzkt_mainnet')
batch_iter = datasource.iter_big_map(
big_map_id=big_map_id,
level=last_level,
)
async for key_batch in batch_iter:
for key in key_batch:
...
Starting with 5.0 you can store and expose custom contract and token metadata in the same format DipDup Metadata service does for TZIP-compatible metadata.
Enable this feature with advanced.metadata_interface
flag, then update metadata in any callback:
await ctx.update_contract_metadata(
network='mainnet',
address='KT1...',
metadata={'foo': 'bar'},
)
Metadata stored in dipdup_contract_metadata
and dipdup_token_metadata
tables and available via GraphQL and REST APIs.
This version introduces initial Prometheus integration. It could help you set up monitoring, find performance issues in your code, and so on. To enable this integration, add the following lines to the config:
prometheus:
host: 0.0.0.0
* {{ #summary config/prometheus.md }}
* {{ #summary deployment/prometheus.md }}
- config: Added
custom
section to store arbitrary user data. - metadata: Added
metadata_interface
feature flag to expose metadata in TzKT format. - prometheus: Added ability to expose Prometheus metrics.
- tzkt: Added ability to process realtime messages with lag.
- tzkt: Added missing fields to the
HeadBlockData
model. - tzkt: Added
iter_...
methods to iterate over item batches.
- config: Fixed default SQLite path (
:memory:
). - prometheus: Fixed invalid metric labels.
- tzkt: Fixed pagination in several getter methods.
- tzkt: Fixed data loss when
skip_history
option is enabled. - tzkt: Fixed crash in methods that do not support cursor pagination.
- tzkt: Fixed possible OOM while calling methods that support pagination.
- tzkt: Fixed possible data loss in
get_originations
andget_quotes
methods.
- tzkt: Added
offset
andlimit
arguments to all methods that support pagination.
- bcd: Removed
bcd
datasource and config section. - cli: Removed
docker init
command. - cli: Removed dummy
schema approve --hashes
flag. - config: Removed dummy
advanced.oneshot
flag.
- dipdup: Use fast
orjson
library instead of built-injson
where possible.