v1.10.2

Substreams (v1.6.2)

• Added mechanism to immediately cancel pending requests that are doing an 'external call' (ex: eth_call) on a given block when it gets forked out (UNDO because of a reorg).
• Fixed handling of invalid module kind: prevent heavy logging from recovered panic
• Error considered deterministic which will cache the error forever are now suffixed with <original message> (deterministic error).

v1.10.1

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.10.0

DO NOT USE in production

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.9.12

• This release is a hotfix for a thread leak leading to a slow memory leak.

v1.9.11

[!WARNING] This version contains a thread leak, leading to eventual higher memory usage, bump to v1.9.12

Substreams improvements v1.15.8

Rework the execout File read/write to improve memory efficiency:

• This reduces the RAM usage necessary to read and stream data to the user on tier1,
  as well as to read the existing execouts on tier2 jobs (in multi-stage scenario)

• The cached execouts need to be rewritten to take advantage of this, since their data is currently not ordered:
  the system will automatically load and rewrite existing execout when they are used.

• Code changes include:

  • new FileReader / FileWriter that "read as you go" or "write as you go"
  • No more 'KV' map attached to the File
  • Split the IndexWriter away from its dependencies on execoutMappers.
  • Clock distributor now also reads "as you go", using a small "one-block-cache"

• Removed SUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT env var (since this is not a RAM issue anymore)

• Add uncompressed_egress_bytes field to substreams request stats log message

Various

• (dstore) Add storageClass query parameter for s3:// urls on stores (@fschoell)
• Update the firehose-beacon proto to include the new Electra spec in the 'well-known' protobuf definitions (@fschoell)
• Use The Graph's Network Registry to recognize chains by genesis blocks and fill the 'advertise' server on substreams/firehose

v1.9.10

Substreams improvements v1.15.7

• Tier2 jobs now write mapper outputs "as they progress", preventing memory usage spikes when saving them to disk.
• Tier2 jobs now limit writing and loading mapper output files to a maximum size of 8GiB by default.
• Tier2 jobs now release existingExecOuts memory as blocks progress
• Speed up DeleteByPrefix operations on all tiers (5x perf improvement on some heavy substreams)
• AddedSUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT (int) environment variable to control this new limit.
• Added SUBSTREAMS_STORE_SIZE_LIMIT (uint64) env var to allow overwriting the default 1GiB value
• Added SUBSTREAMS_PRINT_STACK (bool) env var to enable printing full stack traces when caught panic occurs
• Added SUBSTREAMS_DEBUG_API_ADDR (string) environment variable to expose a "debug API" HTTP interface that allows blocking connections, running GC, listing or canceling active requests.
• Prevent a deterministic failure on a module definition (mode, valueType, updatePolicy) from persisting when the issue is fixed in the substreams.yaml streamingfast/substreams#621
• Metering events on tier2 now bundled at the end of the job (prevents sending metering events for failing jobs)
• Added metering for: "processed_blocks" (block * number of stages where execution happened) and "egress_bytes"

v1.9.9

Substreams performance improvements v1.15.4

• (RAM+CPU) dedupe execution of modules with same hash but different name when computing dependency graph. (#619)
• (RAM) prevent memory usage burst on tier2 when writing mapper by streaming protobuf items to writer
• Tier1 requests will no longer error out with "service currently overloaded" because tier2 servers are ramping up

New 'firehose' reader

• Add reader-node-firehose which creates one-blocks by consuming blocks from an already existing Firehose endpoint. This can be used to set up an indexer stack without having to run an instrumented blockchain node, or getting redundancy from another firehose provider.

Other

• Bumped grpc-go lib to 1.72.0
• Now building amd64 and arm64 Docker images on push & release.

v1.9.8

• Flag --reader-node-arguments now accepts to expand {first-streamable-block} with the value defined by config flag --common-first-streamable-block.

• Flag --reader-node-arguments will now expand environment variables if present within the string.

v1.9.7

• Bump substreams to v1.15.2
• fix the 'quicksave' feature on substreams (incorrect block hash on quicksave)

v1.9.6

Substreams (v1.15.1)

• Save deterministic failures in WASM in the module cache (under a file named errors.0123456789.zst at the failed block number), so further requests depending on this module at the same block can return the error immediately without re-executing the module.