Change log
All notable changes to this project will be documented in this file.
- "undo" messages incorrectly contained too many module outputs (all modules, with some duplicates)
- Fixed status bar message cutoff bug
- Added command
substreams gui
, providing a terminal-based GUI to inspect the streamed data. Also adds--replay
support, to save a stream toreplay.log
and load it back in the UI later. You can use it as you wouldsubstreams run
. Feedback welcome. - Modified command
substreams protogen
, defaulting to generating themod.rs
file alongside the rust bindings. Also added--generate-mod-rs
flag to togglemod.rs
generation. - Added support for module parameterization. Defined in the manifest as:
module:
name: my_module
inputs:
params: string
...
params:
my_module: "0x123123"
"imported:module": override value from imported module
and on the command-line as:
substreams run -p module=value -p "module2=other value" ...
Servers need to be updated for packages to be able to be consumed this way.
This change keeps backwards compatibility. Old Substreams Packages will still work the same, with no changes to module hashes.
- Added support for
{version}
template in--output-file
flag value onsubstreams pack
. - Added fuel limit to wasm execution as a server-side option, preventing wasm process from running forever.
- Added 'Network' and 'Sink{Type, Module, Config}' fields in the manifest and protobuf definition for future bundling of substreams sink definitions within a substreams package.
- Improved execution speed and module loading speed by bumping to WASM Time to version 4.0.
- Improved developer experience on the CLI by making the
<manifest>
argument optional.The CLI when<manifest>
argument is not provided will now look in the current directory for asubstreams.yaml
file and is going to use it if present. So if you are in your Substreams project and your file is namedsubstreams.yaml
, you can simply dosubstreams pack
,substreams protogen
, etc.Moreover, we added to possibility to pass a directory containing asubstreams.yaml
directly sosubstreams pack path/to/project
would work as long aspath/to/project
contains a file namedsubstreams.yaml
. - Fixed a bug that was preventing production mode to complete properly when using a bounded block range.
- Improved overall stability of the Substreams engine.
- Breaking Config values
substreams-stores-save-interval
andsubstreams-output-cache-save-interval
have been merged together intosubstreams-cache-save-interval
in thefirehose-<chain>
repositories. Refer to chain specificfirehose-<chain>
repository for further details.
- The
<manifest>
can point to a directory that contains asubstreams.yaml
file instead of having to point to the file directly. - The
<manifest>
parameter is now optional in all commands requiring it.
- Fixed valuetype mismatch for stores
- Fixed production mode not completing when block range was specified
- Fixed tier1 crashing due to missing context canceled check.
- Fixed some code paths where locking could have happened due to incorrect checking of context cancellation.
- Request validation for blockchain's input type is now made only against the requested module it's transitive dependencies.
- Updated WASM Time library to 4.0.0 leading to improved execution speed.
- Remove distinction between
output-save-interval
andstore-save-interval
. substreams init
has been moved undersubstreams alpha init
as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands.substreams codegen
has been moved undersubstreams alpha codegen
as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands.
This upcoming release is going to bring significant changes on how Substreams are developed, consumed and speed of execution. Note that there is no breaking changes related to your Substreams' Rust code, only breaking changes will be about how Substreams are run and available features/flags.
Here the highlights of elements that will change in next release:
In this rest of this post, we are going to go through each of them in greater details and the implications they have for you. Full changelog is available after.
Warning Operators, refer to Operators Notes section for specific instructions of deploying this new version.
We introduce an execution mode when running Substreams, either
production
mode or development
mode. The execution mode impacts how the Substreams get executed, specifically:- The time to first byte
- The module logs and outputs sent back to the client
- How parallel execution is applied through the requested range
The difference between the modes are:
- In
development
mode, the client will receive all the logs of the executedmodules
. Inproduction
mode, logs are not available at all. - In
development
mode, module's are always re-executed from request's start block meaning now that logs will always be visible to the user. Inproduction
mode, if a module's output is found in cache, module execution is skipped completely and data is returned directly. - In
development
mode, only backward parallel execution can be effective. Inproduction
mode, both backward parallel execution and forward parallel execution can be effective. See Enhanced parallel execution section for further details about parallel execution. - In
development
mode, every module's output is returned back in the response but only root module is displayed by default insubstreams
CLI (configurable via a flag). Inproduction
mode, only root module's output is returned. - In
development
mode, you may request specificstore
snapshot that are in the execution tree via thesubstreams
CLI--debug-modules-initial-snapshots
flag. Inproduction
mode, this feature is not available.
The execution mode is specified at that gRPC request level and is the default mode is
development
. The substreams
CLI tool being a development tool foremost, we do not expect people to activate production mode (-p
) when using it outside for maybe testing purposes.If today's you have
sink
code making the gRPC request yourself and are using that for production consumption, ensure that field production_mode
in your Substreams request is set to true
. StreamingFast provided sink
like substreams-sink-postgres, substreams-sink-files and others have already been updated to use production_mode
by default.Final note, we recommend to run the production mode against a compiled
.spkg
file that should ideally be released and versioned. This is to ensure stable modules' hashes and leverage cached output properly.We now only support 1 output module when running a Substreams, while prior this release, it was possible to have multiple ones.
- Only a single module can now be requested, previous version allowed to request N modules.
- Only
map
module can now be requested, previous version allowedmap
andstore
to be requested. InitialSnapshots
is now forbidden inproduction
mode and still allowed indevelopment
mode.- In
development
mode, the server sends back output for all executed modules (by default the CLI displays only requested module's output).
Note We addedoutput_module
to the Substreams request and keptoutput_modules
to remain backwards compatible for a while. If anoutput_module
is specified we will honor that module. If not we will checkoutput_modules
to ensure there is only 1 output module. In a future release, we are going to removeoutput_modules
altogether.
With the introduction of
development
vs production
mode, we added a change in behavior to reduce frictions this changes has on debugging. Indeed, in development
mode, all executed modules's output will be sent be to the user. This includes the requested output module as well as all its dependencies. The substreams
CLI has been adjusted to show only the output of the requested output module by default. The new substreams
CLI flag -debug-modules-output
can be used to control which modules' output is actually displayed by the CLI.Migration Path If you are currently requesting more than one module, refactor your Substreams code so that a singlemap
module aggregates all the required information from your different dependencies in one output.
It is now forbidden to request a
store
module as the output module of the Substreams request, the requested output module must now be of kind map
. Different factors have motivated this change:- Recently we have seen incorrect usage of
store
module. Astore
module was not intended to be used as a persistent long term storage,store
modules were conceived as a place to aggregate data for later steps in computation. Using it as a persistent storage make the store unmanageable. - We had always expected users to consume a
map
module which would return data formatted according to a finalsink
spec which will then permanently store the extracted data. We never envisionedstore
to act as long term storage. - Forward parallel execution does not support a
store
as its last step.
Migration Path If you are currently using astore
module as your output store. You will need to create amap
module that will have as input thedeltas
of saidstore
module, and return the deltas.
Let's assume a Substreams with these dependencies:
[block] --> [map_pools] --> [store_pools] --> [map_transfers]
- Running
substreams run substreams.yaml map_transfers
will only print the outputs and logs from themap_transfers
module. - Running
substreams run substreams.yaml map_transfers --debug-modules-output=map_pools,map_transfers,store_pools
will print the outputs of those 3 modules.
Now that a
store
cannot be requested as the output module, the InitialSnapshots
did not make sense anymore to be available. Moreover, we have seen people using it to retrieve the initial state and then continue syncing. While it's a fair use case, we always wanted people to perform the synchronization using the streaming primitive and not by using store
as long term storage.However, the
InitialSnapshots
is a useful tool for debugging what a store contains at a given block. So we decided to keep it in development
mode only where you can request the snapshot of a store
module when doing your request. In the Substreams' request/response, initial_store_snapshot_for_modules
has been renamed to debug_initial_store_snapshot_for_modules
, snapshot_data
to debug_snapshot_data
and snapshot_complete
to debug_snapshot_complete
.Migration Path If you were relying onInitialSnapshots
feature in production. You will need to create amap
module that will have as input thedeltas
of saidstore
module, and then synchronize the full state on the consuming side.
Let's assume a Substreams with these dependencies:
[block] --> [map_pools] --> [store_pools] --> [map_transfers]
- Running
substreams run substreams.yaml map_transfers -s 1000 -t +5 --debug-modules-initial-snapshot=store_pools
will print all the entries in store_pools at block 999, then continue with outputs and logs frommap_transfers
in blocks 1000 to 1004.
There are 2 ways parallel execution can happen either backward or forward.
Backward parallel execution consists of executing in parallel block ranges from the module's start block up to the start block of the request. If the start block of the request matches module's start block, there is no backward parallel execution to perform. Also, this is happening only for dependencies of type
store
which means that if you depends only on other map
modules, no backward parallel execution happens.Forward parallel execution consists of executing in parallel block ranges from the start block of the request up to last known final block (a.k.a the irreversible block) or the stop block of the request, depending on which is smaller. Forward parallel execution significantly improves the performance of the Substreams as we execute your module in advanced through the chain history in parallel. What we stream you back is the cached output of your module's execution which means essentially that we stream back to you data written in flat files. This gives a major performance boost because in almost all cases, the data will be already for you to consume.
Forward parallel execution happens only in
production
mode is always disabled when in development
mode. Moreover, since we read back data from cache, it means that logs of your modules will never be accessible as we do not store them.Backward parallel execution still occurs in
development
and production
mode. The diagram below gives details about when parallel execution happen.
parallel processing
You can see that in
production
mode, parallel execution happens before the Substreams request range as well as within the requested range. While in development
mode, we can see that parallel execution happens only before the Substreams request range, so between module's start block and start block of requested range (backward parallel execution only).The state output format for
map
and store
modules has changed internally to be more compact in Protobuf format. When deploying this new version, previous existing state files should be deleted or deployment updated to point to a new store location. The state output store is defined by the flag --substreams-state-store-url
flag parameter on chain specific binary (i.e. fireeth
).- Added
production_mode
to Substreams Request - Added
output_module
to Substreams Request
- Fixed
Ctrl-C
not working directly when in TUI mode. - Added
Trace ID
printing once available. - Added command
substreams tools analytics store-stats
to get statistic for a given store. - Added
--debug-modules-output
(comma-separated module names) (unavailable inproduction
mode). - Breaking Renamed flag
--initial-snapshots
to--debug-modules-initial-snapshots
(comma-separated module names) (unavailable inproduction
mode).
- Moved Rust modules to
github.com/streamingfast/substreams-rs
- Gained significant execution time improvement when saving and loading stores, during the squashing process by leveraging vtprotobuf
- Added XDS support for tier 2s
- Added intrinsic support for type
bigdecimal
, will deprecatebigfloat
- Significant improvements in code-coverage and full integration tests.
- Added
substreams tools proxy <package>
subcommand to allow calling substreams with a pre-defined package easily from a web browser using bufbuild/connect-web - Lowered GRPC client keep alive frequency, to prevent "Too Many Pings" disconnection issue.
- Added a fast failure when attempting to connect to an unreachable substreams endpoint.
- CLI is now able to read
.spkg
fromgs://
,s3://
andaz://
URLs, the URL format must be supported by our dstore library). - Command
substreams pack
is now restricted to local manifest file. - Added command
substreams tools module
to introspect a store state in storage. - Made changes to allow for
substreams
CLI to run on Windows OS (thanks @robinbernon). - Added flag
--output-file <template>
tosubstreams pack
command to control where the.skpg
is written,{manifestDir}
and{spkgDefaultName}
can be used in thetemplate
value where{manifestDir}
resolves to manifest's directory and{spkgDefaultName}
is the pre-computed default name in the form<name>-<version>
where<name>
is the manifest's "package.name" value (_
values in the name are replaced by-
) and<version>
ispackage.version
value. - Fixed relative path not resolved correctly against manifest's location in
protobuf.files
list. - Fixed relative path not resolved correctly against manifest's location in
binaries
list. substreams protogen <package> --output-path <path>
flag is now relative to<package>
if<package>
is a local manifest file ending with.yaml
.- Endpoint's port is now validated otherwise when unspecified, it creates an infinite 'Connecting...' message that will never resolves.
- Fixed error when importing
http/https
.spkg
files inimports
section.
New updatePolicy
append
, allows one to build a store that concatenates values and supports parallelism. This affects the server, the manifest format (additive only), the substreams crate and the generated code therein.- Store APIs methods now accept
key
of typeAsRef<str>
which means for example that bothString
an&str
are accepted as inputs in:StoreSet::set
StoreSet::set_many
StoreSet::set_if_not_exists
StoreSet::set_if_not_exists_many
StoreAddInt64::add
StoreAddInt64::add_many
StoreAddFloat64::add
StoreAddFloat64::add_many
StoreAddBigFloat::add
StoreAddBigFloat::add_many
StoreAddBigInt::add
StoreAddBigInt::add_many
StoreMaxInt64::max
StoreMaxFloat64::max
StoreMaxBigInt::max
StoreMaxBigFloat::max
StoreMinInt64::min
StoreMinFloat64::min
StoreMinBigInt::min
StoreMinBigFloat::min
StoreAppend::append
StoreAppend::append_bytes
StoreGet::get_at
StoreGet::get_last
StoreGet::get_first
- Low-level state methods now accept
key
of typeAsRef<str>
which means for example that bothString
an&str
are accepted as inputs in:state::get_at
state::get_last
state::get_first
state::set
state::set_if_not_exists
state::append
state::delete_prefix
state::add_bigint
state::add_int64
state::add_float64
state::add_bigfloat
state::set_min_int64
state::set_min_bigint
state::set_min_float64
state::set_min_bigfloat
state::set_max_int64
state::set_max_bigint
state::set_max_float64
state::set_max_bigfloat
- Bumped
prost
(and related dependencies) to^0.11.0
- Environment variables are now accepted in manifest's
imports
list. - Environment variables are now accepted in manifest's
protobuf.importPaths
list. - Fixed relative path not resolved correctly against manifest's location in
imports
list. - Changed the output modes:
module-*
modes are gone and become the format forjsonl
andjson
. This means all printed outputs are wrapped to provide the module name, and other metadata. - Added
--initial-snapshots
(or-i
) to therun
command, which will dump the stores specified as output modules. - Added color for
ui
output mode under a tty. - Added some request validation on both client and server (validate that output modules are present in the modules graph)
- Added support to serve the initial snapshot
- Changed
substreams manifest info
->substreams info
- Changed
substreams manifest graph
->substreams graph
- Updated usage
- Multiple fixes to boundaries
- Various bug fixes around store and parallel execution.
- Fix null pointer exception at the end of CLI run in some cases.
- Do log last error when the CLI exit with an error has the error is already printed to the user and it creates a weird behavior.
- Ensure arguments can be passed to Docker built image.
- Various bug fixes around store and parallel execution.
- Fixed logs being repeated on module with inputs that was receiving nothing.
- Added
substreams::hex
wrapper around hex_literal::hex macro
- Added
substreams run -o ui|json|jsonl|module-json|module-jsonl
.
- Fixed a whole bunch of issues, in parallel processing. More stable caching. See chain-specific releases.
- Fixed
substreams
crate usage from tagged version published on crates.io.
code:
is now defined in thebinaries
section of the manifest, instead of in each module. A module can select which binary with thebinary:
field on the Module definition.- Added
substreams inspect ./substreams.yaml
orinspect some.spkg
to see what's inside. Requiresprotoc
to be installed (which you should have anyway). - Added command
substreams protogen
that writes a temporarybuf.gen.yaml
and generates Rust structs based on the contents of the provided manifest or package. - Added
substreams::handlers
macros to reduce boilerplate when create substream modules.substreams::handlers::map
is used for the handlers corresponding to modules of typemap
. Modules of typemap
should return aResult
where the error is of typeError
/// Map module example#[substreams::handlers::map]pub fn map_module_func(blk: eth::Block) -> Result<erc721::Transfers, Error> {...}substreams::handlers::store
is used for the handlers corresponding to modules of typestore
. Modules of typestore
should have no return value./// Map module example#[substreams::handlers::store]pub fn store_module(transfers: erc721::Transfers, s: store::StoreAddInt64, pairs: store::StoreGet, tokens: store::StoreGet) {...}
- Added
substreams::Hex
wrapper type to more easily deal with printing and encoding bytes to hexadecimal string. - Added
substreams::log::info!(...)
andsubstreams::log::debug!(...)
supporting formatting arguments (acts likeprintln!()
macro). - Added new field
logs_truncated
that can be used to determined if logs were truncated. - Augmented logs truncation limit to 128 KiB per module per block.
- Updated
substreams run
to properly report module progress error. - When a module WASM execution error out, progress with failure logs is now returned before closing the substreams connection.
- The API token is not passed anymore if the connection is using plain text option
--plaintext
. - The
-c
(or--compact-output
) can be used to print JSON as a single compact line. - The
--stop-block
flag onsubstream run
can be defined as+1000
to stream from start block + 1000.
- Added Dockerfile support.
- Improved defaults for
--proto-path
and--proto
, using globs. - WASM file paths in substreams.yaml manifests now resolve relative to the location of the yaml file.
- Added
substreams manifest package
to create .pb packages to simplify querying using other languages. See the python example. - Added
substreams manifest graph
to show the Mermaid graph alone. - Improved mermaid graph layout.
- Removed native Go code support for now.
- Always writes store snapshots, each 10,000 blocks.
- A few tools to manage partial snapshots under
substreams tools
First chain-agnostic release. THIS IS BETA SOFTWARE. USE AT YOUR OWN RISK. WE PROVIDE NO BACKWARDS COMPATIBILITY GUARANTEES FOR THIS RELEASE.
See https://github.com/streamingfast/substreams for usage docs..
- Removed
local
command. See README.md for instructions on how to run locally now. Buildsfeth
from source for now. - Changed the
remote
command torun
. - Changed
run
command's--substreams-api-key-envvar
flag to ``--substreams-api-token-envvar, and its default value is changed from
SUBSTREAMS_API_KEYto
SUBSTREAMS_API_TOKEN`. See README.md to learn how to obtain such tokens.
Last modified 1d ago