Search
⌃K

Quickstart

Get off the ground by using Substreams by StreamingFast

Run your first Substreams

You will first need to get a StreamingFast API key from https://app.streamingfast.io. Using this API key, retrieve an API token by using:
export STREAMINGFAST_KEY=server_123123 # Use your own API key
export SUBSTREAMS_API_TOKEN=$(curl https://auth.streamingfast.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token)
After you have authenticated, you're ready to run your first Substreams.
Tip: The substreams CLI must be installed to continue.
substreams run
$ substreams run -e mainnet.eth.streamingfast.io:443 https://github.com/streamingfast/substreams-ethereum-quickstart/releases/download/1.0.0/substreams-ethereum-quickstart-v1.0.0.spkg map_block --start-block 12292922 --stop-block +1
The run command starts a consumer by using the --endpoint serving a given blockchain, for the spkg package. Processing starts at the given block, then stops after processing one block. The output of the map_block module is streamed to the requesting client.
Note: While Substreams technology is chain-agnostic, you must write your Substreams for a specific chain. In this quickstart, we are using Ethereum as our specific chain, general concepts given in the quick start applied to every Substreams supported chain.

Build a Substreams module

In this section we are going to:
  • Create your first Substreams module
  • Use the substreams CLI to run the module
Note: Before continuing, ensure that your system meets the basic requirements for Substreams development.

Create Substreams manifest

To create a "Substreams module", you must first create the manifest file. This example manifest includes the minimal required fields to demonstrate the core values that you must provide.
To use the example manifest, copy and paste it into a new file named substreams.yaml. Save this file in the root directory of your Substreams module. You can find the example manifest in the official GitHub repository for substreams-ethereum-quickstart.
specVersion: v0.1.0
package:
name: 'substreams_ethereum_quickstart'
version: v1.0.0
protobuf:
files:
- block_meta.proto
importPaths:
- ./proto
binaries:
default:
type: wasm/rust-v1
file: ./target/wasm32-unknown-unknown/release/substreams.wasm
modules:
- name: map_block
kind: map
inputs:
- source: sf.ethereum.type.v2.Block
output:
type: proto:sf.ethereum.block_meta.v1.BlockMeta

Create Rust manifest file

To complete your new Substreams module, you must also create a Rust manifest file.
To use the example Rust manifest file, copy and paste its content into a new file named Cargo.toml. Save this file in the root directory of your Substreams module. It's important to provide a unique and useful value for the "name" field and to make sure that crate-type = ["cdylib"] is defined so the WASM is generated.
Include any dependencies on Substreams crates for helpers and prost for protobuf encoding and decoding. Finally, use the values provided in the example for the profile.release section to build an optimized .WASM file for your module.
[package]
name = "substreams-ethereum-quickstart"
version = "1.0.0"
edition = "2021"
[lib]
name = "substreams"
crate-type = ["cdylib"]
[dependencies]
substreams = "0.5.0"
substreams-ethereum = "0.9.0"
prost = "0.11"
[profile.release]
lto = true
opt-level = 'z'
strip = "debuginfo"
Tip: The [profile.release] section can dramatically reduce the size of your wasm code and its loading time in substreams engine. lto = true is always beneficial to performance. opt-level = 'z' optimizes for size, which results in a reduced "Time To First Byte", in some cases by several seconds, which is usually beneficial, but it may not always be the best choice for performance. See more info. We do, however, recommend it. The strip = "debuginfo" will remove useful information from stack traces, but the faster load time still make it a good choice when iterating in development.

Create protobufs

Substreams modules are required to output protobuf encoded messages. The example protobuf definition from the substreams-ethereum-quickstart defines a simple BlockMeta message that contains that block's hash, number, parent hash and timestamp all in human readable form.
Copy and paste the content for the example protobuf definition into a new file named block_meta.proto and save it to a proto directory in the root directory of your Substreams module.
syntax = "proto3";
package sf.ethereum.block_meta.v1;
message BlockMeta {
string hash = 1;
uint64 number = 2;
string parent_hash = 3;
string timestamp = 4;
}
Use the substreams protogen command to generate the Rust code to communicate with the protobuf.
substreams protogen substreams.yaml"
Note: You might have noticed that the substreams protogen substreams.yaml generate some files that are not referenced, add flag --exclude-paths="sf/substreams,google to avoid generating those files which are already provided implicitly for you..
The protobufs generate model must be referenced by a Rust module, to do so, create a file named mod.rs within the src/pb directory with the following content:
#[path = "sf.ethereum.block_meta.v1.rs"]
#[allow(dead_code)]
pub mod block_meta;

Create Substreams module handlers

Your Substreams module must contain a Rust library that houses the module handlers, the code that is invoked to perform your customized logic. These handlers are responsible for handling blockchain data injected into the module at runtime, see Substreams Modules for further details about module and module handlers.
To include this example module handler in your module, copy it into a new Rust source code file named lib.rs within the src directory.
mod pb;
use pb::block_meta::BlockMeta;
use substreams::Hex;
use substreams_ethereum::pb::eth;
#[substreams::handlers::map]
fn map_block(block: eth::v2::Block) -> Result<BlockMeta, substreams::errors::Error> {
let header = block.header.as_ref().unwrap();
Ok(BlockMeta {
number: block.number,
hash: Hex(&block.hash).to_string(),
parent_hash: Hex(&header.parent_hash).to_string(),
timestamp: header.timestamp.as_ref().unwrap().to_string(),
})
}
Compile your Substreams module.
cargo build --release --target wasm32-unknown-unknown
Note: If you have a lots of weird compilation errors like cannot find macro 'assert' in this scope, cannot find tuple struct or tuple variant 'Some' in this scope, you probably don't have the target wasm32-unknown-unknown installed in your Rust environment, see Rust installation for instructions how to install it.

Execute

To execute, or run, the example use the substreams run command:
substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_block --start-block 10000001 --stop-block +1
To ensure that the run command is executed correctly, you need to pass the proper endpoint, manifest name, and module handler name. The --start-block and --stop-block flags are optional, but they can help limit the results that are returned to the client.
You have successfully created your first Substreams module that extracts data from the Ethereum blockchain.

Next steps