v1.9.5

Substreams

• Fix a panic when a module times out on tier2 while being executed from cached outputs
• Add environment variables to control retry behavior, "SUBSTREAMS_WORKER_MAX_RETRIES" (default 10) and "SUBSTREAMS_WORKER_MAX_TIMEOUT_RETRIES" (default 2), changing from previous defaults (720 and 3)
  The worker_max_timeout_retries is the number of retries specifically applied to block execution timing out (ex: because of external calls)
• The mechanism to slow down processing segments "ahead of blocks being sent to user" has been disabled on "noop-mode" requests, since these requests are used to pre-cache data and should not be slowed down.
• The "number of segments ahead" in this mechanism has been increased from >number of parallel workers> to <number of parallel workers> * 1.5
• Tier2 now returns GRPC error codes for DeadlineExceeded when it times out, and ResourceExhausted when a request is rejected due to overload
• Tier1 now correctly reports tier2 job outcomes in the substreams request stats
• Added jitter in "retry" logic to prevent all workers from retrying at the same time when tier2 are overloaded

v1.9.4

Substreams (v1.14.5)

• Bugfix for panics on some requests

v1.9.3

🚫 DO NOT USE - Panics on some requests

Substreams (v1.14.4)

• Bugfix: Properly reject requests with a stop-block below the "resolved" StartBlock (caused by module initialBlocks or a chain's firstStreamableBlock)
• Bugfix: Added the resolved-start-block to the substreams request stats log

v1.9.2

Substreams (v1.14.3)

Bugfixes

• Fixed runtime error: slice bounds out of range error on heavy memory usage with wasmtime engin

• Added a validation on a module for the existence of 'triggering' inputs: the server will now fail with a clear error message
  when the only available inputs are stores used with mode 'get' (not 'deltas'), instead of silenlty skipping the module on every block.

Performance

• Added a mechanism for 'production-mode' requests where the tier1 will not schedule tier2 jobs over { max_parallel_subrequests } segments above the current block being streamed to the user.
  This will ensure that a user slowly reading blocks 1, 2, 3... will not trigger a flood of tier2 jobs for higher blocks, let's say 300_000_000, that might never get read.

Service lifecycle

• Improved connection draining on shutdown: Now waits for the end of the 'shutdown-delay' before draining and refusing new connections, then waits for 'quicksaves' and successful signaling of clients, up to a max of 30 sec.

Logging / errors

• Added information about the number of blocks that need to be processed for a given request in the sf.substreams.rpc.v2.SessionInit message
• Added an optional field limit_processed_blocks to the sf.substreams.rpc.v2.Request. When set to a non-zero value, the server will reject a request that would process more blocks than the given value with the FailedPrecondition GRPC error code.
• Improved error messages when a module execution is timing out on a block (ex: due to a slow external call) and now return a DeadlineExceeded Connect/GRPC error code instead of a Internal. Removed 'panic' from wording.
• In 'substreams request stats' log, add fields: remote_jobs_completed, remote_blocks_processed and total_uncompressed_read_bytes

v1.9.1

Substreams

• Fix another cannot resolve 'old cursor' from files in passthrough mode -- not implemented bug when receiving a request in production-mode with a cursor that is below the "linear handoff" block

v1.9.0

Substreams

• Rust modules will now be executed with wasmtime by default instead of wazero.

  • Prevents the whole server from stalling in certain memory-intensive operations in wazero.
  • Speed improvement: cuts the execution time in half in some circumstances.
  • Wazero is still used for modules with wbindgen and modules compiled with tinygo.
  • Set env var SUBSTREAMS_WASM_RUNTIME=wazero to revert to previous behavior.

• Implement "QuickSave" feature to save the state of "live running" substreams stores when shutting down, and then resume processing from that point if the cursor matches.

  • Added flag substreams-tier1-quicksave-store to enable quicksave when non-empty
    (requires --common-system-shutdown-signal-delay to be set to a long enough value to save the in-flight stores)

Misc

• The firecore tools print one-block is now able to print from a file directly.

v1.8.0

Substreams

Capacity Management

• Integrated the GlobalRequestPool service in the Tier1App to manage global requests pooling.

• Integrated the GlobalWorkerPool service in the Tier1App to manage global worker pooling.

• Added flag substreams-tier1-global-worker-pool-address, the address of the global worker pool to use for the substreams tier1. (disabled if empty)

• Added flag substreams-tier1-global-worker-pool-keep-alive-delay delay between two keep alive call to the global worker pool. Default is 25s")

• Added flag substreams-tier1-global-request-pool-keep-alive-delay delay between two keep alive call to the global worker pool for request. Default is 25s

• Added flag substreams-tier1-default-max-request-per-user default max request per user, this will be use of the global worker pool is not reachable. Default is 5

• Added flag substreams-tier1-default-minimal-request-life-time-second default minimal request life time, this will be use of the global worker pool is not reachable. . Default is 180

• Limit parallel execution of a stage's layer: Previously, the engine was executing modules in a stage's layer all in parallel.
  We now change that behavior, development mode will from now on execute every sequentially and when in production mode will
  limit parallelism to 2 (hard-coded) for now.
  The auth plugin can control that value dynamically by providing a trusted header X-Sf-Substreams-Stage-Layer-Parallel-Executor-Max-Count.

Performance

• Fixed a regression since "v1.7.3" where the SkipEmptyOutput instruction was ignored in substreams mappers
• Add shared cache for tier1 execution near HEAD, to prevent multiple tier1 instances from reprocessing the same module on the same block when it comes in (ex: foundational modules)
• Improved fetching of state caches on tier1 requests to speed up "time to first data"

Tools

• make 'compare-blocks' command support one-blocks stores as well as merged-blocks

v1.7.3

• Bump substreams lib to v1.12.2
  • fix panic when using an index that allows skip_empty_output

v1.7.2

• Fixed substreams-tier2 not setting itself ready correctly on startup since v1.7.0.

• Added support for --output=bytes mode which prints the chain's specific Protobuf block as bytes, the encoding for the bytes string printed is determined by --bytes-encoding, uses hex by default.

• Added back -o as shortand for --output in firecore tools ... sub-commands.

v1.7.1

• Add back grpc.health.v1.Health service to firehose and substreams-tier1 services (regression in 1.7.0)
• Give precedence to the tracing header X-Cloud-Trace-Context over Traceparent to prevent user systems' trace IDs from leaking passed a GCP load-balancer