StreamingFast Substreams module types
Substreams uses two types of modules,
mapmodules are functions receiving bytes as input and output. These bytes are encoded protobuf messages.
storemodules are stateful, saving and tracking data through the use of key-value stores.
storemodules write to key-value stores.
Note: To ensure successful and proper parallelization can occur,
storemodules are not permitted to read any of their own data or values.
Stores declaring their own data types expose methods capable of mutating keys within the
- Do not save keys in stores unless they are going to be read by a downstream module. Substreams stores are a way to aggregate data, but they are not meant to be a storage layer.
- Do not save all transfers of a chain in a
storemodule, rather, output them in a
mapand have a downstream system store them for querying.
There are limitations impose on store usage. Specifically, each key/value entry must be smaller than 10MiB while a store cannot exceed 1GiB total. Keys being string, each character in the key account for 1 byte of storage space.
The two important store properties are
valueTypeproperty instructs the Substreams runtime of the data to be saved in the
updatePolicyproperty determines what methods are available in the runtime.
updatePolicyalso defines the merging strategy for identical keys found in two contiguous stores produced through parallel processing.
Tip: All update policies provide the
The merge strategy is applied during parallel processing.
- A module has built two partial stores containing keys for segment A, blocks 0-1000, and segment B, blocks 1000-2000, and is prepared to merge them into a complete store.
- The complete store is represented acting as if the processing was done in a linear fashion, starting at block 0 and proceeding up to block 2000.
Important_: _ To preserve the parallelization capabilities of the system, Substreams is not permitted to read what it has written or read from a
storeactively being written.
A downstream module is created to read from a store by using one of its inputs to point to the output of the
Ordinals allow a key-value store to have multiple versions of a key within a single block. The
storeAPIs contain different methods of
For example, the price for a token can change after transaction B and transaction D, and a downstream module might want to know the value of a key before transaction B and between B and D.
Important: Ordinals must be set every time a key is set and you can only set keys in increasing ordinal order, or by using an ordinal equal to the previous.
In situations where a single key for a block is required and ordering in the store is not important, the ordinal uses a value of zero.
You can consume data in one of two modes when declaring a
storeas an input to a module.
get modefunction provides the module with a key-value store that is guaranteed to be synchronized up to the block being processed. It's possible to query stores by using the
Tip: Lookups are local, in-memory, and extremely high-speed.
The definition of
storemethod behavior is:
get_lastmethod is the fastest because it queries the store directly.
get_firstmethod first goes through the current block's deltas in reverse order, before querying the store, in case the key being queried was mutated in the block.
get_atmethod unwinds deltas up to a specific ordinal, ensuring values for keys set midway through a block are still reachable.
deltasmode provides the module with all the changes occurring in the source
storemodule. Updates, creates, and deletes of the keys mutated during the block processing become available.
Note: When a
storeis set as an input to the module, it is read-only and you cannot modify, update or mutate them.