StreamingFast Substreams fundamental knowledge
This documentation aims to outline information to further help developers working with Substreams. Specifically, how the multitude of different pieces fit together including the manifest, protobufs, Rust modules, module handlers, WASM, and Substreams CLI.
Substreams in Action
- Identify smart contract addresses of interest including wallets, decentralized exchanges (DEXs), etc.
- Identify data, and define and create protobufs.
- Write Rust Substreams event handler functions.
- Update substreams manifest, point to protobufs and handlers.
- Issue command to Substreams CLI passing manifest.
The Substreams engine basically is the CPU, or brain, of the Substreams system. The engine handles requests, and communication and orchestrates the transformation of blockchain data.
Developers send commands, flags, and a reference to the manifest configuration file through the Substreams CLI to the Substreams engine.
Developers create the data transformation strategies in Substreams “module handlers” defined using the Rust programming language. The module handlers act on protobuf-based data models referenced from within the Substreams manifest. Learn more about the protobufs for the different blockchains in the chains and endpoints section of the Substreams documentation.
The Substreams engine runs the code defined by developers in the Rust-based module handlers.
The flow of data is defined in the Substreams manifest through the “inputs” and “outputs” fields of the configuration file. These fields generally reference the protobuf definitions for the targeted blockchain data. The flow of data can also be defined using the “inputs” field to send data directly from one module to another.
The Substreams manifest references the modules, the handlers defined within them, and lays out the intention of how each is used by the Substreams engine.
Directed acyclic graphs contain nodes, in this case, modules, that communicate in only one direction, passing from one node, or module, to another.
The Substreams engine creates the “compute graph”, or “dependency graph” at runtime through commands sent to the CLI using code in modules referenced by the manifest.
Substreams module handlers linked to protobuf
View the protobuf file in the repo by visiting the following link.
View the Rust module handlers in the lib.rs file in the repo by visiting the following link.
Custom smart contracts targeted by developers, such as UniSwap, will have protobuf definitions that have already been created for them by others. The custom data models are referenced in the Substreams manifest and made available to module handler functions.
In object-oriented programming terminology, the protobufs are the objects or object models. In front-end web development terms, protobufs are similar to the REST, or other data access API.
Firehose and Substreams treat the data as the API.
Protobufs essentially provide the API to the targeted data, usually associated with a smart contract address.
Writing Rust Modules for Substreams
Designing an overall strategy for how to manage and transform data is the first thing developers will do when creating a Substreams implementation. Substreams modules are processed by the engine with the relationships between them defined in the manifest.
The design and complexity of the modules and the way they work together will be based on the smart contracts and data being targeted by the developer.
Two types of module handlers are defined within the Rust modules; maps and stores. The two module types work in conjunction to sort, sift, temporarily store and transform blockchain data from smart contracts for use in data sinks, such as databases or subgraphs.
Continue to the modules documentation to learn more about detailed aspects of their use and purpose.