Manifests
StreamingFast Substreams manifests reference
Tip: When writing and checking your
substreams.yaml
file, it may help to check your manifest against our JSON schema to ensure there are no problems. JSON schemas can be used in Jetbrains and VSCode. Our manifest schema can be seen here.The manifest is the high-level outline for a Substreams module. The manifest file is used for defining properties specific to the current Substreams module and identifying the dependencies between the
inputs
and outputs
of modules.This reference documentation provides a guide for all fields and values used in a Substreams manifest.
Excerpt pulled from the example Substreams manifest.
manifest excerpt
specVersion: v0.1.0
Use
v0.1.0
for the specVersion
field.Excerpt pulled from the example Substreams manifest.
manifest excerpt
package:
name: module_name_for_project
version: v0.5.0
doc: |
Documentation heading for the package.
More detailed documentation for the package.
The
package.name
field is used to identify the package.The
package.name
field infers the filename when the pack
command is run by using substreams.yaml
as a flag for the Substreams package.The content of the
name
field must match the regular expression: ^([a-zA-Z][a-zA-Z0-9_]{0,63})$
The regular expression ruleset translates to the following:
- 64 characters maximum
- Separate words by using
_
- Starts by using
a-z
orA-Z
and can contain numbers thereafter
The
package.version
field identifies the package for the Substreams module.The
package.url
field identifies and helps users discover the source of the Substreams package.The
package.doc
field is the documentation string of the package. The first line is a short description and longer documentation proceeds a blank line.The
imports
section references WASM code, compiled protobuf definitions, and module definitions.Tip: Imported modules can be referred to later in the
modules
section of the manifest through the use of a key.Example:
imports:
ethereum: substreams-ethereum-v1.0.0.spkg
tokens: ../eth-token/substreams.yaml
prices: ../eth-token/substreams.yaml
The filename can be absolute or relative or a remote path prefixed by
http://
or https://
.Imports differ across different blockchains. For example, Ethereum-based Substreams modules reference the matching
spkg
file created for the Ethereum blockchain. Solana, and other blockchains, reference a different spkg
or resources specific to the chosen chain.The
protobuf
section points to the Google Protocol Buffer (protobuf) definitions used by the Rust modules in the Substreams module.protobuf:
files:
- google/protobuf/timestamp.proto
- pcs/v1/pcs.proto
- pcs/v1/database.proto
importPaths:
- ./proto
- ../../external-proto
The Substreams packager loads files in any of the listed
importPaths
.Note: The
imports
section of the manifest also affects which .proto
files are used in the final Substreams package.Protobufs and modules are packaged together to help Substreams clients decode the incoming streams. Protobufs are not sent to the Substreams server in network requests.
The
binaries
field specifies the WASM binary code to use when executing modules.The
modules[].binary
field uses a default value of default
.binaries:
default:
type: wasm/rust-v1
file: ./target/wasm32-unknown-unknown/release/my_package.wasm
other:
type: wasm/rust-v1
file: ./snapshot_of_my_package.wasm
Important: Defining the
default
binary is required when creating a Substreams manifest.The type of code and implied virtual machine for execution. There is only one virtual machine available that uses a value of:
wasm/rust-v1
.The
binaries[name].file
field references a locally compiled WASM module. Paths for the binaries[name].file
field are absolute or relative to the manifest's directory. The standard location of the compiled WASM module is the root directory of the Substreams module.Tip: The WASM file referenced by the
binary
field is picked up and packaged into an .spkg
when invoking the pack
and run
commands through the substreams
CLI.This example shows one map module, named
events_extractor
and one store module, named totals
:substreams.yaml
- name: events_extractor
kind: map
initialBlock: 5000000
binary: default # Implicit
inputs:
- source: sf.ethereum.type.v2.Block
- store: myimport:prices
output:
type: proto:my.types.v1.Events
- name: totals
kind: store
updatePolicy: add
valueType: int64
inputs:
- source: sf.ethereum.type.v2.Block
- map: events_extractor
The identifier for the module, prefixed by a letter, followed by a maximum of 64 characters of
[a-zA-Z0-9_]
. The same rules applied to the package.name
field applies to the module name
.The module
name
also corresponds to the name of the Rust function invoked on the compiled WASM code upon execution. The module name
is the same #[substreams::handlers::map]
as defined in the Rust code. Maps and stores both work in the same fashion.Important: When importing another package, all module names are prefixed by the package's name and a colon. Prefixing ensures there are no name clashes across multiple imported packages and almost any name can be safely used for a module
name
.The initial block for the module is where Substreams begins processing data for a module. The runtime never processes blocks prior to the one for any given module.
If all the inputs have the same
initialBlock
, the field can be omitted and its value is inferred by its dependent inputs
.initialBlock
becomes mandatory when inputs have different values.There are two module types for
modules[].kind
:map
store
Specifies the merge strategy for two contiguous partial stores produced by parallelized operations.
The values for
modules[].updatePolicy
are defined using specific rules stating:set
, the last key wins the merge strategyset_if_not_exists
, the first key wins the merge strategyappend
, concatenates two keys' valuesadd
, sum the two keys' valuesmin
, min between two keys' valuesmax
, max between two keys' values
Tip: The module
updatePolicy
field is only available for modules of kind: store
.Specifies the data type of all keys in the
store
, and determines what WASM imports are available to the module and are able to write to the store
.The values for
modules[].valueTypes
can use various types including:bigfloat
bigint
int64
bytes
string
proto:path.to.custom.protobuf.Model
Tip: The module
valueType
field is only available for modules of kind: store
.The
modules[].binary
field overrides which binary is used from the binaries
declaration section. This means multiple WASM files can be bundled in the Package.modules:
name: hello
binary: other
...
The default value for
binary
is default
. Therefore, a default
binary must be defined under binaries
.substreams.yaml
inputs:
- params: string
- source: sf.ethereum.type.v2.Block
- store: my_store
mode: deltas
- store: my_store # defaults to mode: get
- map: my_map
The
inputs
field is a list of input structures. One of three keys is required for every object.The key types for
inputs
include:source
store,
used to definemode
keysmap
params
substreams.yaml
output:
type: proto:eth.erc721.v1.Transfers
The value for
type
is always prefixed using proto:
followed by a definition specified in the protobuf definitions, and referenced in the protobuf
section of the Substreams manifest.Tip: The module
output
field is only available for modules of kind: map
.The
params
mapping changes the default values for modules' parameterizable inputs.modules:
...
params:
module_name: "default value"
"imported:module": "overridden value"
You can override those values with the
-p
parameter of substreams run
.When rolling out your consuming code -- in this example, Python -- you can use something like:
my_mod = [mod for mod in pkg.modules.modules if mod.name == "store_pools"][0]
my_mod.inputs[0].params.value = "myvalue"
which would be inserted just before starting the stream.
Last modified 2d ago