Search…
⌃K

Modules Overview

StreamingFast Substreams modules overview

What are Modules?

Modules are a crucial aspect of working with Substreams. Modules provide hooks into the execution of the Substreams compute engine. Developers will craft their own data manipulation and transformation strategies within modules.
In further detail, modules are small pieces of Rust code running in a WebAssembly (WASM) virtual machine. Modules coexist within the stream of blocks sent by the Substreams compute engine arriving from a blockchain node.
Tip: Modules can also process network history from flat files backed by StreamingFast Firehose. See Firehose documentation for additional information.
Modules have one or more inputs. The inputs can be in the form of a map or store, or a Block or Clock received from the blockchain's data source.
Note: Multiple inputs are made possible because blockchains are clocked.
Blockchains allow synchronization between multiple execution streams opening up great performance improvements over comparable traditional streaming engines.

Single Output

Note: Modules have a single typed output. Modules are typed to inform consumers of the types of data to expect and also how to interpret the bytes being sent.
Modules are typically formed into a directed acyclic graph (DAG). Information flows from one module to another in a forward progression for DAG data structures.
Tip: Data that is output from one module is used as the input for subsequent modules basically forming a daisy chain of data flow from module to module.

Modules Example

In the diagram shown below the transfer_map module extracts all transfers in each Block, and the transfer_counter store module tracks the number of transfers that have occurred.
Substreams modules data interaction diagram
Modules can also take in multiple inputs as seen in the counters store example diagram below. Two modules feed into a store, effectively tracking multiple counters.
Modules with multiple inputs diagram
All of the modules are executed as a directed acyclic graph (DAG) each time a new Block is processed.
Note: The top-level data source is always a protocol's Block protobuf model, and is deterministic in its execution.