Skip to content

Lodestar Weekly Standups 2025

Jump to bottom
Phil Ngo edited this page Sep 8, 2025 · 40 revisions

The Lodestar team hosts planning/standup meetings weekly on Tuesdays at 3:00pm Universal Standard Time. These meetings allow the team to conduct release planning, prioritise tasks, sync on current issues, implementations and provide status updates on the Lodestar roadmap.

Note that these notes are transcribed and summarized by AI language models and may not accurately reflect the context discussed.

Lodestar Standup 2025-31 (September 2, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8236

Recording/Transcript: https://drive.google.com/drive/folders/1IM4R4UhmaiMMlGfW2NxsLf8eUmTrXoPL?usp=sharing

Planning/Discussions

DevNet 3 and 5 Status Updates

The team discussed the current state of Fusaka DevNet 3 and upcoming timeline for DevNet 5. DevNet 5 is scheduled for early next week, though dates remain fluid. The team has been working on fixing issues in the Fusaka DevNet 3 branch after several squash merges created confusion about what changes were missing or present.

DevNet 3 Improvements:

  • A significant bug fix was implemented related to request tracking and stream handling
  • Tuyen identified an issue where the finally block in request handling code wasn't being reached when server responses had errors
  • This caused problems with the self-rate limiter tracking, leading to requests being considered stuck for over 30 seconds
  • The fix involved adding timeout logic to treat requests as complete after 30 seconds if they haven't been properly closed at the application level
  • This was a workaround for improper request tracking at the application level, where libp2p handles stream closing but the application layer wasn't tracking it correctly

Performance Metrics:

  • Lodestar performed notably well on DevNet 3 over the weekend, syncing fine while other clients experienced issues - however the root cause of why Lodestar remained in sync while others didn't has not concluded
  • Team deployed unstable branch to DevNet 3 and confirmed it's syncing and running properly
  • Metrics analysis is ongoing to provide Barnabas with detailed information about what changes contributed to improved performance

Fusaka Timeline Discussions

  • Team expressed cautious optimism about Fusaka timeline following recent improvements
  • Matthew noted higher confidence in current implementation but expects timeline delays based on past devnet experiences
  • Early December mainnet target discussed as potential realistic timeline rather than November 5th
  • Team acknowledged that aggressive timelines often shift based on devnet performance across all clients

Block Input Refactoring Progress

Cayman provided updates on the major block input refactoring effort that's been a significant focus for the team:

Current Status:

  • The refactoring is now capable of syncing mainnet, Holesky, and DevNet 3 successfully
  • Most unit testing is complete and the code is building properly
  • The team is close to being ready to merge the refactoring into the unstable branch
  • There are still some outstanding items like proper peer downscoring that need to be ported over from the current implementation

Technical Scope:

  • This represents a major architectural change to how block inputs are handled
  • Matthew has been working on ensuring feature parity by comparing diffs between major files and identifying what functionality might have been left behind
  • The refactoring aims to improve conceptual clarity and make the codebase easier to work with

getBlobs v1 Specification Compliance

The team identified an important specification compliance issue with their current getBlobsV1 implementation:

Current vs. Spec Behavior:

  • Current implementation: Wait some amount of time after receiving a block, then trigger getBlobsV1 only if not all blobs have been received
  • Specification requirement: Immediately trigger getBlobsV1 upon receiving a block (since blob sidecars don't contain commitments that are in blocks)

Implementation Plan:

  • Katya will work on correcting this behavior since she's already working on getBlobsV2 metrics
  • The change should be relatively small (approximately 5 lines of code) involving moving the call from the unknown block handler to the gossip handler
  • This change will provide valuable metrics about how getBlobsV2 might perform on mainnet
  • The corrected implementation should also handle IWANT publishing for blobs received via getBlobsV1 and regossip them appropriately

Data Collection Value:

  • This change is considered worthwhile despite the small time window before fork because it will provide real-world data about blob availability patterns
  • The metrics will help inform decisions about getBlobsV2 implementation effectiveness

Syncing Strategy Discussion

The team briefly touched on syncing strategy improvements, though detailed work is being postponed:

Current Approach:

  • Cayman emphasized focusing on completing the block input refactoring before tackling syncing improvements
  • The block input refactoring represents a substantial diff that needs to be stabilized
  • As Tuyen fixes issues, they're being merged back into the branch, requiring constant integration work
  • The team needs to get this major refactoring merged before taking on additional large architectural changes

Future Planning:

  • Once the block input refactoring is complete and merged, the team can focus on syncing strategy improvements
  • The current approach prioritizes completing one major refactoring before starting another to avoid compounding complexity

Data Column Sidecar Implementation

  • Tuyen mentioned that the team currently doesn't serve data column sidecar requests by range, which Nazar is investigating.
  • This represents additional work needed for full Fusaka compliance.

Updates

Matt

  • Worked on fixing the Fusaka DevNet 3 branch after squash merge issues created confusion
  • Deployed unstable branch to DevNet 3 which is now syncing and running properly
  • Continuing work on block input refactoring, focusing on ensuring feature parity
  • Going through diffs in major files to identify any functionality that wasn't ported over
  • Most unit testing is complete and the code is building successfully
  • Working on final items needed to make the refactoring ready for merge

Tuyen

  • Identified and fixed critical bug in request tracking where finally blocks weren't being reached on server response errors
  • Implemented timeout-based workaround for self-rate limiter tracking issues
  • Added logic to consider requests completed after 30 seconds if not properly closed
  • Working on BLST-Z branch which is ready for review and potential release
  • Fixed buffer-from-hex performance issues that were causing getBlobsV2 performance problems
  • Identified that browser implementation of buffer-from-hex is significantly faster than the table implementation

Cayman

  • Made significant progress on block input refactoring, achieving successful sync on mainnet, Holesky, and DevNet 3
  • Working on SSZ library integration with Bun, developing benchmarks to compare JavaScript vs. Zig implementation performance
  • Experimenting with boundaries between Zig and JavaScript to optimize performance
  • Initial experiments suggest using Zig for beacon state while keeping other objects (attestations, beacon blocks) in JavaScript
  • Plans to push benchmarking results to the lodestar-bun repository for team review
  • Emphasized need to complete current refactoring before starting syncing improvements

Nazar

  • Working on Bun runtime compatibility issues, focusing on deep dependency conflicts
  • Resolved Bun types package conflicts that were causing global type modifications incompatible with NodeJS
  • Implementing import aliases feature to handle runtime-specific dependencies
  • Working on GC profiler compatibility issues between Bun and NodeJS runtimes
  • Developing polyfill patterns for dependencies not compatible with Bun runtime
  • Fighting with TypeScript compatibility issues when using import aliases with different runtimes

Nico

  • Fixed critical issue with block production where blocks with zero blobs couldn't be produced
  • Conducted KZG verification performance benchmarking to investigate performance issues
  • Noted that current data column gossip metrics may be skewed because the implementation waits for all data columns
  • Investigated event loop blocking issues in block production that caused HTTP requests to builders to appear sent but not actually be transmitted
  • Found edge case affecting 0.01% of blocks from Lido nodes where event loop blocking prevented proper builder communication
  • Working on sync committee aggregate handling bugs that affect small devnets where validators can appear multiple times in sync committees
  • Planning to bring sync from unfinalized branch to unstable to help with devnet testing

Katya

  • Completed FOCIL metrics refactoring, moving fork choice metrics into the fork choice package
  • Working on slashing implementation and state logic improvements
  • Interested in helping with end-to-end test refactoring that Tuyen has initiated
  • Will take on the getBlobsV1 specification compliance fix as part of her metrics work

Nina (EPF)

  • Continued work on migration from Kurtosis endpoints to current Lodestar endpoints
  • Processing service discovery and API construction adaptations based on feedback from Nazar
  • Estimates approximately two more weeks to complete the migration work

Rahul (EPF)

  • Completed era file reading implementation with round-trip testing (read era file, write to new era file, verify reading works)
  • Following EthereumJS implementation since the ultralight portal client implementation hasn't been ported yet
  • Used Nimbus code as reference since EthereumJS doesn't have writing implemented
  • Planning to push the completed PR for review soon

Vedant (EPF)

  • Had personal commitments that limited development time this week
  • Made fixes based on Nico's suggestions and wrote test cases that are passing
  • Awaiting review from Matthew on current PR
  • Seeking guidance on how to proceed with next steps in backfill implementation

Bun Updates

Cayman - SSZ Library Integration

  • Experimenting with exposing SSZ library to Bun and benchmarking performance
  • Comparing JavaScript deserialization and hashing versus Zig implementation with hash tree root
  • Initial findings suggest using Zig for beacon state while keeping other objects in JavaScript
  • Working on understanding optimal boundaries between Zig and JavaScript for performance
  • Planning to push experimental results to lodestar-bun repository for team evaluation

Nazar - Runtime Compatibility

  • Successfully resolved Bun types package conflicts with NodeJS global types
  • Implementing import aliases pattern to handle runtime-specific dependencies
  • Working on GC profiler polyfill since Bun doesn't support the same GC information as NodeJS
  • Developing patterns for handling deep dependencies that conflict between runtimes
  • Fighting TypeScript compatibility issues with import aliases feature

Lodestar-Zig Updates

Bing

  • Continuing work on state transition implementation and architectural improvements
  • Collaborating with Cayman on code style and architecture decisions
  • Planning to present observations and recommendations at Thursday's Zig roadmapping session
  • Working on balance between maintaining familiarity with current TypeScript implementation and optimizing for Zig best practices
  • Preparing discussion points about type imports and overall code organization

Tuyen

  • BLST-Z branch is ready for review and release
  • Waiting for Cayman's review before finalizing release
  • Limited time available for Zig work due to performance issue investigations

Lodestar Standup 2025-30 (August 26, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8236

Recording/Transcript: https://drive.google.com/drive/folders/1Sim5_GD672keTOJy_ronO80Tugb_Wbhz?usp=sharing

Planning/Discussions

DevNet 3 Status and Performance Issues

The team discussed the critical state of DevNet 3, which is experiencing network-wide instability. While appearing to be a general network issue, the team identified several client-specific problems that need addressing:

Performance Issues Identified:

Sync-Related Problems:

Network Timeline Constraints:

  • DevNet 5 is scheduled for September 3rd and will likely still use the unstable branch unless otherwise specified
  • Optimistic timeline shows DevNet 5 completion by September 10th, Holesky and Sepolia releases for September 15th, and Holesky fork on September 22nd
  • Team expressed concerns about this aggressive timeline given current stability issues
  • The team acknowledged that Teku is the only client currently maintaining stable sync on the network, indicating other implementations need significant work.
  • Though timelines are tight, Lodestar functions on a stable Fusaka devnet with potential loss of validator effectiveness.

Sync Architecture Refactoring Discussion

The team engaged in extensive discussion about the approach to fixing the fundamental sync issues: https://github.com/ChainSafe/lodestar/issues/8225

Philosophy on Implementation:

  • Cayman emphasized avoiding rushed implementations that create technical debt, advocating for "doing it right" rather than fast
  • The team agreed not to commit to specific timelines for mainnet readiness, instead focusing on proper architecture
  • Decision to approach sync problems from first principles: understanding what sync means, what data needs tracking, and what algorithms are most appropriate

Collaboration Strategy:

  • Matthew reached out to the Teku team to understand their successful sync implementation
  • Plan to have conversations with Teku developers to learn about their algorithms and approach
  • Goal is not to copy their code verbatim but understand their problem-solving methodology

Development Approach:

  • Extensive refactoring of sync code is ongoing, with Matthew working on unit testing and getting CI to pass
  • The refactored code is building and has been deployed to a feature branch for initial testing
  • Testing on DevNet 3 will be limited due to network instability, so additional Kurtosis testing may be necessary

Performance Optimization Research

Tuyen identified several potential optimization areas for the block processing bottleneck:

Memory Allocation Issues:

  • Hypothesis that inefficient Unit8Array allocation is contributing to the 4-second block processing time
  • Proposed solution: allocate large buffers upfront and deserialize into sub-fields rather than multiple small allocations
  • Discussion about memory management across worker boundaries, particularly for KZG library calls
  • Plan to implement array buffer reuse strategies for processing multiple slots

Threading Concerns:

  • Clarification that getBlocksV2 runs on the main thread, making it a potential bottleneck
  • Discussion about the cost implications of garbage collection pressure on the main thread
  • Investigation into whether cell processing can be optimized to use shared memory buffers

Timeline and Fork Discussion

The team reviewed the aggressive Fusaka timeline and expressed reservations:

September Timeline Concerns:

  • DevNet 4 launching September 3rd using current unstable branch
  • DevNet 5 completion targeted for September 10th
  • Holesky fork scheduled for September 22nd (less than a month away)
  • Team consensus that timeline may be too optimistic given current stability issues

Readiness Assessment:

  • Cayman noted that while sync issues are concerning, mainnet performance might be acceptable since it won't include the stress testing scenarios that break DevNets
  • Matthew agreed that mainnet deployment might work despite DevNet issues, as Teku demonstrates client capability
  • Team acknowledged they might not be the only client struggling with DevNet conditions

Lodestar-Bun Architecture

  • Cayman opened new lodestar-bun repository as proof of concept for centralized Bun wrapper approach
  • Architectural shift from individual Zig library wrappers to consolidated Bun wrapper library
  • Rationale: separating Zig module concerns from Bun-specific binding concerns improves code quality
  • Allows Zig modules to focus on best practices without Bun-specific constraints
  • Currently exposing hashtree and persistent Merkle tree with testing framework
  • Planning to add more SSZ types and experiment with access patterns and interfaces
  • Considering Lodestar-specific optimizations like pre-allocated memory buffers for gossip data
  • Team discussed configurability vs. Lodestar-specific optimizations, decided to experiment with approach

Peer Count Increase Testing

  • Tested increased peer count limits starting from Lighthouse's approach
  • Results show minimal performance regression even at 300 peers
  • Bandwidth impact measured at only 3 MB/s increase (from 16 to 19 MB/s)
  • Findings align with Lighthouse's reported results
  • Recommendation to proceed with peer count increase for Holesky deployment
  • May simplify peer selection logic as additional benefit

End-to-End Test Improvements

  • Tuyen implemented end-to-end tests supporting current fork (Fulu) and next fork (Electra) transitions
  • Found significant debugging value in fork transition testing
  • Discovered network finalization issues related to missing data column setup
  • Recommends migrating sim tests to start from current fork rather than Phase 0 for efficiency
  • Emphasizes importance of maintaining updated test infrastructure before each fork deployment
  • Plans to make fork transition testing mandatory for future development cycles

Updates

Tuyen

  • Identified performance bottlenecks with block processing taking nearly 4 seconds
  • Discovered issues with getBlocksV2 causing excessive GC pressure
  • Found network processor inefficiencies affecting overall performance
  • Plans to investigate memory allocation optimization strategies tomorrow
  • Identified attestation wait times of 1.5 seconds as problematic
  • Limited time available for Zig work due to performance issue investigations
  • Coordinating with Bing on branch management strategy

Matthew

  • Made significant progress on sync architecture refactoring
  • Currently working on unit testing for the new sync implementation
  • Deployed refactored code to feature branch, reports it's building successfully
  • Focused on getting check types working and CI passing for existing unit tests
  • Reached out to Teku team for collaboration on sync strategies
  • Working on final files needed for complete type checking

Cayman

  • Deployed the new sync code to a feature group for testing
  • Emphasized importance of proper architectural approach over rushed fixes
  • Advocating for first-principles thinking about sync implementation
  • Supports reaching out to other teams for collaborative problem-solving

Nina (EPF - Kurtosis)

  • Continued work on migration from Kurtosis endpoints to current Lodestar endpoints
  • Processing service discovery and API construction adaptations
  • Estimates approximately two more weeks to complete the migration work

Rahul (EPF - Era Files)

  • Completed the era file reading PR and awaiting review
  • Working on the era file writing implementation
  • Added tests using era files from Nimbus mainnet for validation
  • Considering whether to include writing functionality in the same PR or separate it

Kai (EPF - zig-libp2p)

  • Opened PR for QUIC transport certificate signature verification and remote peer ID extraction
  • Encountered issues with TLS certificate access during handshake callbacks
  • Implemented workaround using thread-local variables instead of C++ libp2p patches
  • Working on multi-threading tests to ensure stability
  • Opened draft PR for gossip sub stream management and peer membership
  • Created PR for disc v5 library replacing SECP256K1 algorithm to match other clients
  • Fixed minor bugs in the disc v5 repository

Nazar

  • Resolved discussion about peer response handling when data column sidecars are unavailable
  • Decision made to skip missing data rather than respond with resource unavailable
  • Experimenting with running Lodestar on Bun in clean virtual machine without NodeJS
  • Discovering new edge cases and errors specific to Bun runtime environment
  • Resolved CPU features package compatibility issues
  • Currently addressing data store layer issues that rely on Node-specific APIs
  • Working toward cross-runtime compatibility solutions

Bing

  • Continuing API updates and first-principles review of TypeScript port to Zig
  • Collaborating with Cayman on architectural decisions for proper Zig module design
  • Working on opaque type implementations as discussed in architectural reviews
  • Coordinating with Tuyen on unified branch strategy for easier collaboration and review
  • Planning mega review approach once branch consolidation is complete

Lodestar Standup 2025-29 (August 19, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8214

Recording/Transcript: https://drive.google.com/drive/folders/1cs6WjvBi-N5pzF7kVocDM7OWiSMbyTxn?usp=sharing

Planning/Discussions

Block Input Refactoring Progress

  • Matthew provided a comprehensive update on the ongoing block input refactoring initiative, emphasizing the primary goal of merging the existing block input data types while maintaining current behavior.
  • The refactoring has revealed several optimization opportunities that the team has consciously chosen to defer to maintain project scope:

Range Sync Improvements Identified

  • Current implementation discards entire batches of data when any error occurs (including simple network terminations)
    • For range sync, a single error (e.g., socket termination due to network instability) on any batch caused the whole batch to be discarded, and a full re-polling effort would ensue.
  • With updated tools, it’s possible to salvage the valid parts of batch downloads and merge them with subsequent sync attempts, reducing wasted work and network churn.
  • Despite the technical allure, this change is intentionally not part of the present refactor to prevent β€œfeature creep”—the drive is to stabilize data types and maintain functional parity.

Unknown Block (byRoot) Sync Enhancements

  • Similar opportunities exist for improving the byRoot heads sync mechanism
  • Team decided to back out some exploratory work that was expanding beyond the intended scope
  • Matthew noted the importance of avoiding "pulling on threads" that lead to feature creep.
  • The minimum-diff approach is preferred β€” merge core changes, identify what breaks, and defer broader fixes.

Testing and Validation Strategy

  • Only three files remain before build completeness
  • The primary focus for this week involves extensive unit testing and debugging to ensure no behavioral regressions are introduced.
  • The team lacks comprehensive unit tests and integration tests for the current implementation, making this migration particularly challenging from a validation perspective.
    • Immediate priority is deep unit testing and robust error-driven debugging to unearth regressions.
    • The lack of historic unit/integration tests makes validation a top risk, especially as behavior changes may be subtle.
  • Tuyen noted missing end-to-end tests, particularly for post-Deneb features like unknown blocking with blobs.
    • Plans to develop new scenarios and improve end-to-end coverage to catch functional issues early

Next Steps and Timelines

  • Cayman has worked in tandem with Matthew to rapidly address migration and testing blockers, contributing deep architectural feedback and hands-on validation.
  • The rest of the team (including Nazar and Nico) offered support in typing, data column refactoring, and reviewing as soon as the core work is build-ready.
  • Matthew clarified the refactor is isolated to a handful of files, making drastic rewrites unlikely, but significant architectural decisions around data structures and sync modes remain.
  • Changing how columns are handled transforms the overall sync paradigm β€” a challenge Lighthouse is also grappling with.
  • The refactor is scoped to take β€œa few weeks at most,” aiming for a pragmatic, minimum-viable sync for Fulu and further improvements incrementally, especially with tight timelines for the next release cycle.
  • He expects a testable branch before Fulu mainnet, with integration and parallel branching for range and forward root sync.
  • Cayman agreed that a robust sync implementation is a short-term priority and not a great long-term solution.
  • Revisiting sync now is crucial, even if it could have been part of earlier DA restructuring.
  • Nico asked if reviews could start; Matthew confirmed only three files remain.
    • Cayman indicated that while gossip handlers, process pipelines, and APIs are ready to review, core sync files (especially source sync) should be left for later.
    • Range sync components should be avoided during review until completion
    • Event handling has been shifted from network- to chain-level boundaries for clarity, and is also ready for review.
    • Nazar offered to assist with TypeScript/data column types
  • Network events have been relocated from network boundary to chain events, reflecting their actual scope

Syncing Strategy

https://github.com/ChainSafe/lodestar/issues/8225

Current Modes & Deficiencies

Cayman outlined three modes in Lodestar’s sync pipeline:

  • Finalized Range Sync: Works reliably; no major issues.
  • Unfinalized Range Sync: Shares logic with finalized but is fundamentally flawed for unfinalized states.
    • The problem is a race condition: after fetching a peer’s status (head/root/slot), the peer’s head can quickly advance before a range-request is issued, leading to inconsistent or mixed chain data, especially if grouping peers for parallel fetches.
    • This unpredictability makes the sync error-prone in dynamic head contexts, and this class of problems is detailed in Issue #8225.
  • Unknown Block Sync: Originally designed to β€œwalk back” unknown blocks to attach to a known chain (using parent-by-parent traversal), but expanded over time to also handle blob and column fetching for missing data.
    • This coupling causes memory pressure (OOM risk) if syncing long unfinalized branches, as all data must be cached in memory before reaching a known checkpoint.
    • This makes the approach unscalable for large backward syncs or during periods of high chain β€œunfinality”.

Proposed Solution & Architectural Improvements

  • Decouple Data Download from Backward Sync: Matthew and Cayman discussed a new strategy: Use only block headersβ€”rather than full block data, blobs, columns, etc. β€” when traversing backwards to connect unknown blocks to the canonical chain. Once the header path is established (the β€œscaffold”), only then begin downloading bodies, blobs, and columns. This reduces memory pressure, as headers are lightweight and allow tracking β€œinfinity” unfinalized depth without OOM risks.
  • Forward Sync by Root: Once the header chain is determined, forward sync leverages β€œby root” instead of β€œby slot” or β€œby range”, so all data is directly linked and integrity-checked via hashes. This minimizes ambiguity and peering complexity; you precisely ask for what you need from each peer, rather than hoping their view matches yours when running bulk range syncs.
  • Unknown Block Sync Only Handles Headers: By refocusing unknown block sync on just headers, it sidesteps the complexity of also tracking unavailable blobs/columns, and avoids complicated peer-balancing schemes required when data types have heterogeneous availability. Peer balancing complexities are relegated to forward (by-root) syncs, which are more deterministic.

Dynamic Scenarios & Future Opportunities

  • Healthy Network Behavior: In normal operation, most missing data (blobs/columns) can be handled via single-slot or β€œmicro” forward syncs once the relevant block header/root is seen, minimizing the need for complex backward traversal.

  • Corner Cases: If a block or slot is missed entirely, unknown block sync activates to walk back headers one-by-one, then triggers the normal forward (by-root) sync sequence.

  • Integrity & Recursion Handling Simplified: Decoupling headers and bodies allows synchronous verification by hash/root at each step, greatly reducing the risk of out-of-sync state and recursive download/processing complications.

  • Documentation & Maintainability: Matthew stressed that drawing clear flow diagrams will help internalize the sync process and prevent the β€œbolt-on blobs” anti-pattern seen in past ad-hoc expansions. This architectural clarity will also inform planned rewrites in Zig and porting elsewhere.

  • Nazar plans to take on the task of creating a flowchart/wire diagram of how we flow through the syncing process to make sure we have something helpful for future maintainers.

Data Availability Window Handling for Backfill

SSZ Optional Types Discussion The team discussed handling data outside the data availability window, particularly around epoch boundaries:

  • Data availability window: 4,096 epochs for data columns
  • Block availability: 8,192 blocks (approximately 18,000 epochs)
  • Nico suggested using SSZ optional types to handle null values for unavailable data
  • Matthew requested Nico's review of the implementation approach for proper null handling

Data Encoding Issues

  • The team is grappling with how to represent state for epochs that may be missing data (e.g., blobs, columns) in a way compatible with SSZ types and the database:
    • Vedant pointed out the need to encode absence properlyβ€”sometimes columns or blobs for a given epoch are β€œnull” (not present or not applicable), while other times they’re present but β€œfalse” (known to be missing).
    • Challenge: SSZ types and some database systems do not natively support null values for labels or data types.
    • Vedant merged feedback and is exploring tweaks; unit tests cover this logic.

Boolean, Nullable, and Array State Approaches

  • Nico questioned whether the pipeline should bother supporting backfill for pre-Deneb or transitional periods, suggesting focus on columns alone, since pre-Fudu production readiness isn’t expected.
  • The team debated nullable vs. boolean logic:
    • Matthew: Wants to distinguish between true/false states and β€œlack of value” (null/undefined) for blob/column presence, especially for epochs outside the active availability window (e.g., 4,096 epochs for columns, ~18,000 for blocks). β€œNullish” for columns indicates that they’re not relevant for the epoch, while true/false indicates actively tracked/synced status.
    • Outside the data window, epochs may simply lack any entry, and null is validβ€”the system doesn’t need to create or backfill for them.
    • Nico: Suggested using SSZ optional types to encode nullable values for this scenario and offered help.

Database & Custody Details

  • For columns, the backfill state is stored as an array of indices (indicating which columns are present for the epoch) versus a boolean for blobs (all or nothing).
  • Custody Change Handling: Cayman asked whether this handles changing custody correctly; Matthew confirmed that columns as arrays allow parsing based on the current custody indices at each epoch, while blobs are binary (present or absent).

Lodestar Zig Updates

SSZ and API Coverage:

  • The immediate goal is to fill gaps in missing APIs within the Zig SSZ implementation, fix failing spec tests, and release an initial version this month.
  • Cayman noted that pending PRs for these APIs are not currently blocking development, as developers can use local commits or path dependencies for testing before the final merge.

BLST-Zig Integration:

  • The team also needs to fix and merge the BLST-Z branch, particularly targeting Tuyen’s fixed state transition work.
  • Once compiled and stable, a 0.1 release version will be published to support further development and integration.

State Transition Unit Tests:

  • Bing reported that the Zig unit tests for process block/state transition are mostly implementedβ€”sanity checks have validated compilation and structural correctness.
  • The focus now shifts to making the codebase reviewable; merged fixes across multiple branches have left the code fragmented, so refactoring and aggregation are needed before thorough team review.

Spec Test Compliance:

  • After merging and unit testing, the next milestone is to ensure the Zig implementation passes all relevant consensus spec tests.

Updates

Matthew

  • Made significant progress on block input refactoring with most components near completion
  • Successfully moved network events to chain events boundary for better architectural alignment
  • Focused on maintaining behavioral consistency while consolidating data types
  • Collaborating closely with Cayman on debugging and implementation challenges
  • Targeting completion of buildable state within the next day or two

Cayman

  • Providing substantial support on the block input refactor through collaborative debugging
  • Identified critical race condition in unfalized range sync architecture
  • Helping with process pipeline and API component reviews
  • Contributing to architectural decisions around sync strategy improvements

Nico

  • Available to provide guidance on SSZ optional types implementation
  • Offering technical review of data availability window handling approach
  • Ready to contribute insights on null value management for epoch boundary cases

Nazar

  • Available to assist with TypeScript type definitions and data column input types
  • Standing by to provide technical support for complex typing challenges in the refactoring effort
  • Request for review: Optimize the leaf node creation for persistent Merkle trees, improving performance during packing of values.
    • Merging and releasing this PR will produce a new version of the persistent Merkle tree package that includes required dependency updates for Bun. This is necessary for Lodestar’s migration from Node.js constructs to Bun.

Vedant (EPF - Backfill)

  • Vedant described completing the backfill pipeline for Prysm, including posting a diagram for review and iterating on feedback regarding storing epoch backfill state in the codebase.
  • The backfill pipeline logic tracks and stores historical data availability for blobs and columns, critical for reconstructing state when historical chain segments are missing or incomplete.
  • Vedant implemented unit tests for the backfill state logic and contributed proposals for this feature in the EPF repository.

Nena (EPF - Kurtosis)

  • Nena is working on switching the current simulation runner over to use Kurtosis, with a focus on integrating Kurtosis into the main simulation and test files.
  • The goal is to have assertions (for node state, merge states, etc.) use data from Kurtosis-based runs instead of the legacy runner, ensuring that inputs and verification logic remain consistent.
  • The transition involves reverse engineering how node pairing and helper functions work under Kurtosis, and methodically testing each change in a stepwise, incremental fashion.
  • Because the codebase’s simulation and testing setup is intricate, Nena is taking care to validate each step and solution as the migration progresses.
  • Nazar has prepared a dedicated implementation branch for this migration, and Nena will begin pushing code and opening PRs there for team review once ready.
  • This steady, validated migration is expected to eventually enable more robust and flexible simulation and test assertions powered by Kurtosis.

Kai/Grapebaba (EPF - zig-libp2p)

  • Kai opened three pull requests for review:
    1. Gossip Sub Message Cache Implementation: This introduces a caching layer for gossip sub messages, likely aimed at optimizing message deduplication and reducing redundant network traffic.
    2. Net Address Migration: Replaces standard net address fields with multi-address format, which is more flexible and compatible with modern libp2p networking stacks.
    3. Gossip Sub Message Generation for Buff Code: Adds generation logic for gossip sub messages within the buffer/code context, possibly automating or improving message serialization/encoding for network transmission.
  • This week, Kai focused on gossip peer stream management and peer membership, which involves refining how peer connections and group memberships are tracked and managed at the P2P layer, improving overall network robustness and handling of peer lists for message propagation and sync integrity.

Lodestar Standup 2025-28 (August 12, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8132

Recording/Transcript: https://drive.google.com/drive/folders/1GqdfNmwvFtdtu6UHmccvoVPUsPe6KjhK?usp=sharing

Planning/Discussions

Fulu Milestone Achievement and Continuous Development

  • Team successfully completed major milestone by getting Fulu on unstable
  • This achievement will unblock and facilitate continuous development moving forward
  • Philip emphasized the importance of this stability for enabling further protocol implementations

Current Top Priority Items Review

Philip outlined the immediate critical priorities requiring team focus:

  • Block input refactor - Currently in progress with active PR work ongoing
  • Stop subscribing to all column subnets - PR available and ready to merge to unblock Sunnyside Labs partnership work
  • Comprehensive syncing review and improvements - Collaborative effort across multiple team members to address current syncing state holistically

Data Column Storage Architecture Discussion

Detailed technical discussion around storing each data column sidecar separately:

  • Tuyen initially considered taking on this work but identified it as non-urgent priority
  • Team consensus that syncing stabilization must take precedence before this refactoring work
  • Cayman emphasized this is lower priority focused on safety and correctness rather than immediate functionality
  • Important consideration: should complete before mainnet to avoid introducing data layout that would require breaking changes later
  • Nazar expressed interest in taking this on later for better understanding of data column implementation details

Network Connectivity and Peer Management Issues

Comprehensive analysis of ongoing peer disconnection patterns:

  • Agent reset and status message sequences - Nazar identified distinct patterns between inbound vs outbound connections:
    • Inbound connections typically show agent version issues
    • Outbound connections show different behavior patterns, possibly related to stored peer IPs or stream opening attempts
  • Metadata and status message timing - Discussion of typical sequence: identify, campaign, request, then status
  • Agent reset events happen separately from the typical ping/metadata/status batch, representing different operational patterns

Custody Group Count Validation Requirements

Technical discussion on disconnecting peers that do not announce custody group count:

  • Matthew noted this issue may already be resolved but requires verification
  • Tuyen emphasized need for stricter validation now that network has matured beyond early testing phase
  • Current implementation defaults to minimum custody requirement when metadata deserialization fails
  • Team agreed need to be more stringent with validation requirements as network stabilizes
  • Cayman suggested integration with existing ENR filtering mechanisms for implementation

Data Availability Window and Backfill Management

Discussion on **updating earliest available slot for backfill priorities:

  • Matthew distinguished between two different update needs:
    • Updating current backfill position tracking
    • Moving data availability window back end as it closes
  • Team consensus this is lower priority since requests outside data availability window won't be responded to anyway
  • Nico raised concern about block request usage spanning longer periods (5+ months)
  • Decision that having older available slot timestamps causes no harm, with spec naturally gating inappropriate requests

Network Protocol and Sidecar Validation Performance

  • Data column sidecar validation timing improvements - Resolved performance issues by switching from synchronous C-KZG to async implementation
  • Sunnyside Labs collaboration - Active external team engagement with focus on providing better error handling rather than design consultation

Spec Test Coverage Gaps

Review of missing networking spec test runners:

  • Nazar volunteered to investigate current gaps in test coverage
  • Cayman noted recent updates may have already addressed portions of this work
  • Nico confirmed additions to PeerDAS branch for networking test runners
  • Discussion around whether networking tests truly require Fulu-specific handling

Updates

Tuyen:

  • Long range sync stability: Extensively tested over 10 iterations, confirmed stable operation and recommended removing from active concern list
  • Data column validation performance: Successfully resolved timing issues by migrating from synchronous CKG to async implementation approach
  • Priority focus areas: Will continue prioritizing syncing stabilization before moving to data column sidecar storage architectural improvements
  • Cache validation issue identification: Discovered higher priority issue involving adding columns to cache before validation step, requiring investigation
  • Timer implementation: Identified need to switch to timer-based approach for certain operations, will investigate this issue

Matthew:

  • Block input refactor progress: Confirmed that current cache validation issues will be comprehensively addressed as part of broader block input refactor work
  • Custody group count validation: Will review previous implementation details and ensure proper defaulting behavior for peers without proper metadata
  • Backfill vs. data availability distinction: Clarified technical differences between updating current backfill position versus managing data availability window boundaries
  • Architecture review: Providing guidance on distinguishing between different types of slot availability updates

Nazar:

  • Sunnyside Labs partnership: Actively collaborating with external team on subnet subscription issues, focusing on improved error messaging rather than design input
  • Peer disconnection pattern investigation: Analyzing distinct patterns in agent reset versus status message sequences, particularly differences between inbound and outbound connection handling
  • Spec test coverage: Will take ownership of investigating missing networking spec test runners and addressing gaps
  • IP storage and connection behavior: Researching how stored peer IPs relate to connection patterns and disconnection events

Cayman:

  • ENR filtering enhancement: Identified integration opportunities for stricter custody group validation within existing peer discovery filtering mechanisms
  • Data availability prioritization: Reinforced importance of completing data layout architectural changes before mainnet deployment to avoid breaking changes
  • Network protocol alignment: Confirmed recent sync testing demonstrates improved stability across the network

Nico:

  • Networking spec test updates: Confirmed recent additions to pas branch specifically for networking test runner capabilities
  • Block request validation scope: Clarified that earliest available slot validation applies to both blocks and data columns, not exclusively blob/sidecar data
  • Testing infrastructure: Providing context on recent improvements to test coverage and validation frameworks

Katya:

  • Custody group validation: Volunteered to investigate custody group announcement issues through log analysis to determine current status and resolution needs
  • Issue verification: Will provide analysis of whether certain GitHub issues have already been resolved through recent changes

NC:

  • ACD call decided to use basis points for slot timing: https://github.com/ChainSafe/lodestar/issues/8013
  • Just need to address some CI failures.
  • This will help unblock us in future Glamsterdam implementations
  • Rebasing of FOCIL will happen on top of ePBS, then will take a look at how to implement them together.
  • New ePBS branch planned since it has changed drastically and want to retain Gajinder's work for reference.

Lodestar Standup 2025-27 (August 5, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8101

Recording/Transcript: https://drive.google.com/drive/folders/1-FFQksnh7WJvRsk5Xntpqp0YZC05BTVo?usp=sharing

Planning/Discussions

Lodestar v1.33.0

  • v1.33 has been in beta for 4–5 days with no observed instabilities or regressions, including after recent networking upgrades.
  • The team is prompted to review and flag any final concerns before release.
  • Some minor gossipsub issues regarding tracking more peers than required. Documented here: https://github.com/ChainSafe/lodestar/issues/8115

Fusaka Development Priority and Timeline Assessment

  • The team discussed the urgency of Fusaka preparation, with Phil emphasizing that client releases are scheduled for approximately one month from the meeting date (early September 2025).
  • The discussion centered around the critical need to ensure stability before Fusaka deployment, particularly as the Ethereum ecosystem moves toward non-finality devnets.

PeerDAS Implementation Challenges and Merge Strategy

Matthew provided a comprehensive overview of the major PeerDAS (Peer Data Availability Sampling) implementation issues currently blocking progress:

Range Sync Instability Issues

  • Multiple batches requested from the same peer in sync chain causing "more than two open requests at a time" errors
  • Rate limiting error in snappy frame that provides wait time instructions but is being ignored by Lodestar
  • Issues stem from PeerDAS complexity requiring repeated requests to the same peers for column data
  • Issues with setting the agent version with an unexpected end of input stemming likely from the libp2p upgrade: https://github.com/ChainSafe/lodestar/issues/7916
  • Column matching bug is when we're getting an empty request back because we're requesting the wrong columns from the peer. We may not be handling empty responses correctly.
  • Backfilling is a long ways out and will take some time.
    • Other teams may not have backfill done in time either.
  • Tuyen will help with the peer limiting and ByRoot syncing issues

Rate limiting in PeerDAS

  • Matt explains that Cayman is correct that the primary issue isn't traditional rate limiting (requests per unit time) but rather a concurrent request limit per peer:
    • The Ethereum P2P specification now limits peers to maximum 2 concurrent requests for protocols like "blobs by range" to any single peer
    • Lodestar wasn't implementing this spec requirement, while other clients had already adopted it
    • When Lodestar exceeds this limit, it violates the protocol specification
  • The second type involves actual temporal rate limiting:
    • Peers can respond with snappy frame messages telling requesters to wait a specific duration (e.g., "do not call me back for 4.3 seconds" or "2.1 seconds")
    • Lodestar was ignoring these rate limiting signals from peers
    • This creates a feedback loop where ignored rate limits lead to further restrictions
  • Root Cause: Pure DAS Implementation
    • Both issues are exacerbated by the PeerDAS feature:
    • PeerDAS requires requesting specific data columns from peers who custody them
    • This leads to repeated targeting of the same peers because they have the required data
    • The result is effectively "hammering" the same subset of peers with multiple requests
    • As Nico notes, this creates a similar situation to when syncing nodes get "banned" by peers due to excessive requests
  • Matt characterizes the current state as "fragile" - working but unstable:
    • These rate limiting violations cause peer disconnections
    • The peer pool shrinks as peers disconnect due to protocol violations
    • Remaining connected peers face increased load, creating a cascading effect
    • While not exclusively a Fulu upgrade issue, it's significantly "exacerbated on Fulu" due to the data column sampling patterns

Database Storage Architecture Decision

  • Following Teku and Prysm implementations, the team adopted a design where blob storage uses concatenated bytes sliced up rather than individual objects.
  • This decision was made to address database object growth issues that other clients experienced, though it adds implementation complexity.
  • Tuyen requests data columns for a single route stored together to have it keyed by root and index.
  • Another idea was brought up to break the database schema in Lodestar v2 necause that's how we're storing blobs - as a concatenated set of blobs by root.

Critical Merge Conflict Resolution Strategy Nico raised concerns about the inability to work on PeerDAS issues until Matthew's branch merges to unstable, as targeting unstable creates millions of merge conflicts. The team agreed on a multi-phase merge strategy:

  1. Complete remaining block input refactor work this week
  2. Perform final merge from unstable to peerDAS branch
  3. Merge peerDAS branch to unstable in one comprehensive operation
  4. Team members should prioritize working on unstable for final 3 weeks before Fusaka

Cayman emphasized this approach prevents repeating workflow mistakes from the Dencun upgrade that caused long-term technical debt.

Action Plan

  • We should deploy PeerDAS branch to feature groups if required, otherwise the nextfork group is automatically updating its build to see any regressions on Pectra.

Updates

Matthew:

  • Range sync remains fragile but functional - "all nodes are green again, but it's a fragile green"
  • Completed recast PR preparation for merging, including comprehensive unit tests and moved slicing functions
  • Block input pipeline completely rewritten on the feature branch
  • Outstanding issues not related to the data availability pipeline include:
    • Stream reset issues introduced by libp2p 2.0
    • Rate limiting on outgoing requests (assigned to Tuyen for Tuesday work)

Nina:

  • Met with Nazar to define maintenance scope and removal of deprecated components
  • Successfully spun up Kurtosis enclave and exposed beacon URL from the new environment
  • Validated responsiveness with health checks and genesis timing retrieval
  • Next phase: implement new Kurtosis runner class and integrate with simulation components
  • Focus area: multifork validation testing

Nico

  • Confirms stability in local beta testing, provides input on branching/merge headaches.
  • Expresses concerns about the maintainability and readiness of large feature branches not yet thoroughly reviewed or aligned with unstable.
  • Advocates committing to merge deadlines and addressing diffs early, favoring mainline integration over continued parallel development.

Nazar:

  • Focuses on ongoing work for networking upgrades and fixes, especially issues triggered by libp2p 2.0 and multi-select handling.
  • Supports review and maintenance of simulation/test frameworks in coordination with Nina.

Bing:

  • Working on the main state transition function PR with a structured checklist approach
  • Currently progressing through the block processing components
  • Implemented intersect values functionality in the SSZ repo as a dependency for the state transition work
  • Created additional smaller PRs in the SSZ repo to support the main implementation
  • Reviewed Nico's PR on "get through bit indices" since it's required for parts of the state transition conflict resolution
  • Coordinating with Nico on related SSZ functionality needed for the state transition implementation

Tuyen:

Katya:

  • Completed FOCIL metrics work
  • Will take issue relating to PeerDAS logs and custody group count
  • Coordinating reviews with team members to ensure quality of the metrics implementations

Nazar:

NC: (async)

  • Zig stuff: Implemented getTrueBitIndexes(PR) and getSingleTrueBit (PR draft) Casually browsing state-transition-z
  • FOCIL: Jihoon is doing a writeup on implementation difficulties of combining ePBS and focil. Gave some input over there.
  • Timing refactor: Gave some general opinion in https://github.com/ethereum/consensus-specs/pull/4476 . Generally in favour of precise cutoff times (in milliseconds) then in basis points (percentage of slot duration). Happy to be convinced otherwise

Lodestar Standup 2025-26 (July 29, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8081

Recording/Transcript: https://drive.google.com/drive/folders/1rmMOalE_iS73jB-_roHvrEOSNOIsNfeR?usp=sharing

Planning/Discussions

Fusaka Updates

  • DevNet 3 Launch Status: DevNet 3 has successfully launched with generally minimal problems reported
  • However, significant syncing issues have emerged as the primary concern, particularly affecting the team's super nodes
  • Issue #8093 - Genesis Validator Handling: The team discussed a technical issue brought up by Barnabas regarding how Genesis validators are considered as finalized state. The core problem centers around the custody group count (CGC) that only updates on finalized blocks. Nico proposed an alternative approach where CGC updates could trigger on every validator registration rather than waiting for finalized state, sparking technical debate about the proper implementation according to the Ethereum consensus specifications.
  • Syncing Infrastructure Problems: The most critical issue identified involves three super nodes and Erigon-2 experiencing synchronization problems. Matthew reported that the super nodes are not properly connecting to super node peers, with the root cause still under investigation. This represents a significant operational challenge that requires immediate attention.

Bun Technical Architecture Discussions

  • Benchmarks posted here seem to indicate much better performance with bun-ffi: https://github.com/ChainSafe/lodestar/issues/7280#issuecomment-3113254649
  • Getting Bun to work with beacon-node is a priority and we're keeping track of external blockers in this issue: https://github.com/ChainSafe/lodestar/issues/7280
  • LevelDB NAPI Binding is erroring: If you require the binding from the main thread and then you require the same binding from a worker thread, there is an error instead of resolving the module properly.
    • The issue described in the Lodestar standup refers to a well-documented Node.js problem where native modules with NAPI bindings fail to load properly when required in worker threads. This is a common issue that affects many native Node.js modules, including the classic-level (LevelDB) database module that Lodestar uses.
  • Classic-level Database Module Issues: A complex technical problem was identified with the classic-level database module when used in worker thread contexts:
    • The classic-level module uses native C++ bindings through NAPI to interface with the LevelDB database. When a native module is loaded in the main thread and then subsequently loaded in a worker thread, Node.js can encounter several problems:
    • "Module did not self-register" Error: This is the most common error message that appears when native modules fail to properly initialize in worker thread contexts.
    • Context Awareness Problem: Native modules need to be "context-aware" to work properly across multiple V8 contexts (main thread vs worker threads). Many older native modules were not designed with this requirement in mind.
    • Binding Singleton Issues: As mentioned in the transcript, there are "two pointers that need to get shared" - the native database pointer and the bindings singleton. These don't properly transfer between main and worker processes.
    • Cayman discovered that while the module functions properly in the main process, it fails when required within worker threads. The issue appears to stem from module loading and caching mechanisms not properly handling the transition between main and worker processes.
    • Issue posted here: https://github.com/oven-sh/bun/issues/21432
  • The technical discussion revealed that the problem involves two critical pointers that need proper sharing: the native database pointer and the bindings singleton. Multiple team members (Cayman, Matthew, Nazar) are collaborating to understand why the same bindings aren't being loaded consistently across main and child processes
  • Currently a blocker for using level. We will consider using SQLite or something else that's more stable.
  • Issue with being able to use bun types as it relies on react types as a peer dependency.
    • Seems like they use it for experimental types
    • We are in favor of creating an issue and reference it into the exception list we have
  • Hashtree now has a generic C fallback: https://github.com/OffchainLabs/hashtree/pull/43/files
    • We will clone from a commit hash
  • Migration to pnpm starting
  • Once dependency issues are complete, we will move onto runtime issues for Bun

Lodestar Zig Updates

  • Tuyen continuing with testing process_epoch function. Having sanity tests there continue to help catch bugs.
  • Bing continuing on the main state transition function and breaking out PR into smaller PRs.
  • Additional work was completed on QUIC bindings with Grapebaba (Kai) and connecting it with the larger zig-libp2p implementation
  • Grapebaba focused on writing the QUIC transport tests, Had issues troubleshooting this week potentially due to differences between Zig/C/C++ compilers, but upstreamed some issues with lsquic. Temporarily bypassed by forking repo.

Backfill

Database and Repository Refactor:

  • Drafted and submitted a PR for changes to the database repositories.
  • Waiting for Matthew to review this initial PR.
  • Focused on understanding and implementing updates to the SSZ types, specifically around encoding and decoding, to improve familiarity and map out next steps.

Backfill Mechanism Research:

  • Investigated how Prysm's codebase handles backfill:
    • Noted support for batch-based backfillβ€”fetching backfill data in batches of one epoch (32 slots per batch).
    • Observed that Prysm provides configuration flags for backfill behavior, such as batch size and the ability to parallelize backfilling using multiple worker threads.
    • Discovered that backfill in Prysm is still in a developmental stage and is disabled by default.

Comparisons with Other Clients:

  • Began reviewing Lighthouse and Teku implementation strategies:
    • Noted that TEKU seemingly fetches blocks iteratively (one-by-one), but indicated uncertainty and plans to confirm details.

Next Steps and Workflow:

  • Working on a main backfill class redesign and plans to draft a PR with this new design.
  • Intends to create a walkthrough video to explain this work when ready.
  • Mentioned this week's progress was slower due to spending time learning aspects of Golang and other client codebases, but expects to realign and catch up next week.

Tuyen's Work on SingleAttestation

  • Optimized Pool Scanning: Tuyen discussed improvements in handling attestations, focusing on reducing redundant scans. After a new attestation is processed, there is typically no need to rescan already-seen attestations in the pool. This leads to a much smaller and more efficient attestation pool.
  • Mainnet Performance: When running a node that subscribes to subnets on mainnet, Tuyen observed effective consolidation of attestations, with 98–99% of cases reaching the expected "package" (i.e., group) size for attestations. The impact on block profitability was minimal or only slightly positive, indicating the optimization is safe and practical for production use.
  • Future Improvements: Tuyen mentioned that, in the future, once the proposer assignments for the next epoch are known, the node can preemptively subscribe for those proposers. This would maintain the same efficiency, further improving the attestation handling pipeline.
  • Extensive Unit Testing: Significant effort was devoted to building comprehensive unit tests for the new code. The last step before full completion is to confirm that a regular (non-validator) node can successfully run the implementation, produce a block, and see it included on-chain. Tuyen hopes to achieve this milestone in the near-term (possibly by slot 123).
  • Proposal Lookahead Enhancements: Tuyen made improvements in "proposal lookahead" by introducing a final process for epoch handling, addressing differences between epoch zero ("fu") and later epochs to make the logic more straightforward.
  • Benchmarking NAPI vs. FFI: Tuyen ran benchmarks comparing native code integrations (FFI vs. NAPI bindings, e.g., for hash functions). Results indicated that NAPI performance was actually worse than not using it, particularly for hash operations. Since FFI lacks async support, this highlighted a need to revisit and re-benchmark using the current module (possibly BSD).

Key Takeaways

  • Attestation pool management is more efficient, reducing CPU/scan overhead.
  • Mainnet testing confirms practical effectiveness and negligible downside risk.
  • Additional architectural work on epoch transitions and performance benchmarking is ongoing.
  • Extensive unit tests and practical block production verification are the remaining steps before full rollout of this optimization.

Libp2p/Nimbus Request Issue and Cooloff Period

libp2p Issue (Related to Nimbus Interop)

  • Context: Matthew brought up a networking problem initially highlighted by Nazar and previously discussed with Cayman. The issue centers on Lodestar's interaction with the libp2p networking stackβ€”specifically with the Nimbus client.
  • Symptoms: The error appears during "recrest" (request) methods and manifests primarily in communication with Nimbus, rather than with other Ethereum consensus clients.
  • Nature of the Problem: The behavior is "tricky" and affects timing-sensitive methods. The root cause remains unclear, and Cayman and Matthew have both been investigating. Matthew acknowledged it's showing up across different request pathways, suggesting it's a protocol/interop bugβ€”not isolated to a single API or message type.
  • Status: The issue is under active investigation, with Cayman also providing metrics/monitoring improvements to help diagnose the problem. The team intends to add further metrics to better isolate and understand the problem from the main thread's perspective.

Cooloff Period PR

  • Progress: Matthew is also working on updating a pull request regarding the "cooloff period" logic, incorporating a second (or third) round of review comments from Tian.
  • Next Steps: He plans to push these changes shortly, with the expectation there may be another round of comments. He is committed to closing out this review cycle promptly.

Updates

Matthew

  • Actively investigating super node connectivity issues in DevNet 3
  • Debugging why super nodes aren't connecting to designated peers
  • Contributing technical expertise to the classic-level worker thread problem analysis
  • Taking ownership of complex infrastructure debugging tasks

Nico

  • Conducted research into issue #8093 regarding Genesis validator CGC handling
  • Proposed alternative technical approach for CGC updates based on validator registration events
  • Challenged existing implementation assumptions about finalized state requirements
  • Bringing fresh perspective to established technical patterns

Cayman

  • Identified and documented specific technical issue with classic-level module in worker threads
  • Confirmed reproducible failure pattern: works in main process, fails in worker context
  • Leading investigation into module loading and caching problems
  • Providing detailed technical analysis of Node.js worker thread behaviors

Nazar

  • Contributing technical analysis on worker thread binding issues
  • Questioning fundamental assumptions about binding consistency across processes
  • Helping debug native database pointer sharing problems
  • Bringing systems-level expertise to complex threading issues

NC

  • Completed read-through of EPBS spec and corrected inconsistencies with PRs
  • Attestation Deadline Timing Refactor: Noted an outstanding issue previously opened regarding the migration from the use of "intervals per slot" to "milliseconds" for attestation deadline calculations and sync committees within Lodestar. This change is significant because both the EPBS specification and ongoing "shorter time slot" network research depend on precise time-based deadlines instead of slot-based intervals.
  • Now sees this as an optimal moment to implement the change in Lodestar, with work already in progress and anticipated to be completed within a few days (noting the change itself is relatively small in scope).

Lodestar Standup 2025-25 (July 22, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8068

Recording/Transcript: https://drive.google.com/drive/folders/1ICiSIUrCw4rVDaz_h5O6Mq_EKEWgo-BI?usp=sharing

Planning/Discussions

PeerDAS and Column Overlap Issues

  • The team identified critical issues with networking performance related to PeerDAS implementation.

  • The primary issue involves column overlap causing excessive requests to the same peers, resulting in rate limiting and disconnections - Matt explained that the sync struggles stem from how range sync requests (reqresps) are happening, leading to "hammering the same peers for lots of batches" because of column overlap.

  • This issue causes nodes to get downscored and disconnected from peers, requiring a cooldown period before they can reconnect and resume syncing.

  • The same fundamental problem affects genesis sync, where nodes encounter rate limiting after approximately 1,000 slots and must wait again.

  • Tuyen is actively working on implementing rate limiting solutions to prevent constantly hitting the same peers and getting banned.

  • The fix involves reducing the download requirement from 128 columns to 64 columns for super node configurations, though the issue affects both super nodes and regular nodes.

DevNet Status and Performance

  • fusaka-devnet-2 has been stabilized and all systems are running smoothly.
  • The network is now fully operational after resolving the peer connectivity and rate limiting issues [2][3].
  • fusaka-devnet-3 preparation is progressing well with only an update required for https://github.com/ethereum/builder-specs/pull/123.
  • Matthew noted that DevNet 3 should be smooth since it's essentially a spec freeze running the same code to ensure it passes all BPO forks without problems.
  • The team expects DevNet 3 to launch soon, with three PRs remaining to be merged before launch

Proposer Reorg

Testing and Simulation Framework Discussion

A significant portion of the discussion centered around Antithesis-style deterministic simulation testing. Cayman shared insights from a conversation about Tiger Beetle's development approach, which started with building a simulation framework first, then creating a database that could hook into their simulation.

Key points about deterministic simulation testing:

  • Tiger Beetle was created by first building a simulation framework for transactions, then building a database that could integrate with their simulation
  • This approach enables deterministic testing where you can "change time" and replay scenarios exactly
  • Antithesis provides deterministic testing capabilities that have been successfully used for blockchain testing, including extensive use during the Ethereum Merge
  • The framework acts like a "time machine" where you can recreate any problem down to the second

The team expressed strong interest in exploring this approach for Lodestar, with Cayman noting it's "definitely worth looking into" as an "interesting paradigm of programming" for their new initiatives.

Integration Strategy with Kurtosis

Nena is coordinating with Matthew on integrating Kurtosis testing with Lodestar's existing simulation tests.

The plan involves:

  • Understanding the differences between what Kurtosis currently covers versus Lodestar's assertion coverage
  • Determining how to bring both frameworks together effectively
  • Figuring out session management and data retrieval mechanisms (HTTP RPC vs hooks)

Updates

  • Successfully stabilized all networks with comprehensive fixes for the sync and peer management issues
  • DevNet 2 is now fully operational with all nodes running smoothly
  • DevNet 3 is ready for deployment with the same stable codebase
  • Blacklist PR has gone through another round of review comments and has been addressed, with only timing considerations remaining
  • Completed work on the rec rest PR that was pending, moving changes from the "puras branch to unstable"
  • Making progress on the by-root sync implementation with updated unit tests planned for completion this week
  • Matthew confirmed that getting everything from the PeerDAS branch onto unstable is "the heart of the issue".
  • Cayman provided excellent review comments on the by-root sync implementation
  • Matthew has started addressing these with additional unit tests planned.

Lodestar Standup 2025-24 (July 15, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/8050

Recording/Transcript: https://drive.google.com/drive/folders/1N953MUSYSTDbjBH5WheVfua58vxjdN8V?usp=sharing

Planning/Discussions

Lodestar 1.32 Release Status

  • The team discussed the readiness of Lodestar 1.32, which has been in beta for six days.
  • Phil confirmed the team was ready to release later that day, noting improved performance metrics from recent updates.
  • The release includes important optimizations for block production, parallelization of block building steps, and builder/engine race improvements[see release notes].

Fusaka Development Progress

  • DevNet 2 Status: The team reported that Fusaka DevNet 2 is running fine with some debugging work ongoing.

  • Matt mentioned debugging a memory issue brought up by Barnabas and completing blacklisting work

  • Peering issues with Nimbus client peering were noted, requiring further investigation with their team[1].

  • DevNet 3 Preparation: Two key PRs remain for DevNet 3:

    • Nico's ENR PR (needs fork digest function updates) and the submit blinded block v2 PR (awaiting spec discussions on status codes).
    • The team is preparing for DevNet 3 launch expected later in July.

Glamsterdam Hardfork Headliner Discussion

The team discussed the upcoming Glamsterdam hardfork headliner selection, with the decision scheduled for July 24, 2025. Key topics included:

  • EPBS (Enshrined Proposer-Builder Separation): Strong consensus in the Fusaka breakout room supporting EPBS as the headliner, with FOCIL as complementary
  • Slot Timing: Discussion of whether to pursue 6-second slots or maintain 12-second slots with EPBS implementation
  • PureETH Advocacy: Cayman suggested pushing for PureETH as the execution layer headliner to complement EPBS and Fusaka
  • Timeline considerations were discussed, with the team noting that shipping Fusaka before DevConnect seems impossible, with mainnet potentially targeting March 2026.

Graphite's Diamond AI Code Review Bot Trial

  • The team discussed their trial with an AI code review bot.
  • Initial feedback indicated that 80% of suggestions were documentation-related, which was considered noise.
  • The team decided to configure the bot to ignore documentation issues and focus on logic errors and bugs, extending the trial for another month.
  • The team decided to also include the bot for some of the Zig libraries being worked on.

Updates

Matt

  • Memory Issues: Debugged memory issues and completed memory profiling work
  • Blacklisting Implementation: Finished blacklisting work using peerscore delay/decay approach
  • Backfill Research: Spent time with Katya reviewing backfill implementations, examining Teku codebase and preparing for Lighthouse research
  • DevNet 3 Preparation: Noted minimal PeerDAS changes needed for DevNet 3
  • Nimbus Issues: Planning to investigate libp2p protocol negotiation issues specific to Nimbus client

Katya

  • Dashboard Updates: Updated PeerDAS dashboard with reconstruction metrics for DevNet 3 comparison
  • Backfill Research: Collaborated with Matthew on backfill strategy, examining Teku and Lighthouse codebases
  • Nimbus Engine Issues: Identified missing responses from Nimbus on engine_getBlobsV2, requiring investigation
  • Metrics Work: Preparing FOCIL metrics implementation for the Fusaka group's evaluation needs

NC

  • DevNet 3 PRs: Two key PRs remain - ENR PR needing fork digest updates and submit blinded block v2 PR
  • Fusaka Support: Confirmed EPBS + Fusaka combination is popular in breakout rooms and technically feasible
  • FOCIL Development: Working on FOCIL branch rebasing and preparing for spec updates
  • SSZ Implementation: Implementing JSON functions in Zig with bug fixes already merged

Nico

  • ENR PR: Needs to update PR due to fork digest function renaming
  • Block Production: Working on block packing profitability testing and single attestation implementation
  • Fusaka DevNet 2: Debugging issues resolved, major differences now on unstable branch
  • Performance Testing: Conducting proposal boost profitability analysis

Tuyen

  • State Transition in Zig: Achieved 90-95% port completion of TypeScript state transition to Zig
  • PeerDAS Work: Implemented data column subnet piece availability to unblock subscription optimizations
  • Single Attestation: Planning to work on block production flow integration
  • Testing Status: Working to stabilize Zig implementation for broader team collaboration[1]

Cayman

  • QUIC Integration: Spent several days working on QUIC with Lodestar, encountering rate limiting, garbled responses, and uninitialized data issues
  • Zig Fork Choice: Continuing work on Zig fork choice implementation
  • LibP2P Upgrade: Discussed necessity of upgrading from year-old LibP2P version due to security and maintenance concerns

Bing

  • Zig Libraries: Reviewing state transition implementation and smaller library issues
  • Beam Chain: Recently discovered beam chain post-quantum devnet requirements involving QUIC with Zig LibP2P
  • Issue Resolution: Opening issues and fixing configuration problems in SSZ and other libraries

EPF Fellows Updates

  • Grapebaba: Working on Zig LibP2P proposal for GossipSub and QUIC, validating C API calls and preparing specification draft
  • Nena: Researching Lodestar's sim test framework for Kurtosis integration, preparing requirements documentation
  • Vedant: Exploring backfill implementations across clients, benchmarking performance, developing algorithm from scratch in coordination with Matthew and Katya

Lodestar 2.0 Development

  • Bun Runtime: Cayman reported challenges with LibP2P compatibility in Bun, requiring monkey patching and significant debugging
  • State Transition: Tuyen's Zig implementation showing strong progress despite current instability
  • Timeline: Estimated 3-month timeline for Lodestar 2.0 completion, potentially targeting DevConnect for major announcements

Lodestar Standup 2025-23 (July 8, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7926

Recording/Transcript: https://drive.google.com/drive/folders/1N953MUSYSTDbjBH5WheVfua58vxjdN8V?usp=sharing

Planning/Discussions

Infrastructure Discussion on Metrics Implementation

  • Chiemerie raised concerns about implementing log-based metrics for attestation monitoring. The infrastructure team prefers using built-in metrics for alerting rather than parsing logs, as log-based queries can be expensive and potentially slow down the Loki instance.
  • Log-based queries can be expensive because they require scanning through large volumes of log data. This is particularly problematic in blockchain environments where:
    • Validators generate continuous streams of attestation-related logs
    • Log volumes can be substantial with hundreds of thousands of active validators
    • Complex pattern matching requires computational resources that can "slow down the Loki instance"
  • Suggested integration of a collection point in the metrics codebase and we can alert on the metric as opposed to current strategy of parsing through collected logs
  • The team agreed to follow up with Nico and Katya to determine if baking these metrics directly into the codebase would be more efficient than the current log-based approach[1].

SSH Authentication Security Proposal - OpkSSH

ChainSafe infrastructure presented a proposal to implement OpkSSH (similar to OIDC) for server authentication, which would provide temporary keys with periodic re-authentication (configurable intervals from 1 day to 1 week). The main benefits would be improved security posture and simplified management when team members leave. However, the proposal met significant resistance from the development team:

Matt and Cayman strongly opposed the change, arguing that:

  • The team is small (7-8 people) with low churn (only 1-2 people leaving per year)
  • Current workflow efficiency would be compromised
  • Security risk on development servers is minimal (no economic value)
  • Production servers already have separate, tighter security patterns
  • The added friction outweighs the benefits for our use case

The infrastructure team ultimately accepted the feedback and withdrew the proposal.

Lodestar V2 Hybrid Zig/JavaScript Direction

Phil highlighted the significant interest generated by Lodestar's announcement of going hybrid ZIG/JavaScript, evidenced by new contributors joining the Discord. The main catalyst goal is running Lodestar on Bun runtime, with a comprehensive checklist created to supersede previous V2 targets.

Cayman presented a visual diagram showing the ZIG transformation architecture, illustrating:

  • ZIG libraries that will be native
  • ZIG libraries exposed to Lodestar via FFI
  • JavaScript layer components
  • Two parallel work streams: component integration and rearchitecting (with rearchitecting dependent on integration progress)[1]

Bun-FFI vs NAPI Technical Decision

A significant discussion centered on the choice between Bun FFI and NAPI for the binding layer:

Arguments for Bun FFI:

  • Performance benefits (2-6x faster than NAPI in benchmarks)
  • Lower overhead for frequent operations
  • Direct pointer manipulation capabilities[1]

Arguments for NAPI:

  • Nazar shared concerning experiences with Bun, including:
    • Fundamental issues with module loading (top-level await problems)
    • 4,600+ open issues on Bun's repository
    • Difficulty getting critical issues resolved
    • Unstable runtime behavior[1]
  • Phil noted warnings from other developers (Fucory of TEVM project) about Bun's long resolution times for issues
  • Runtime compatibility concerns with maintaining both Node.js and Bun support

Team Consensus: The team agreed to proceed with NAPI for the binding layer to maintain compatibility with both Node.js and Bun runtimes, with the option to add specific Bun FFI bindings for performance-critical operations if needed.

ZIG Ecosystem Updates

Cayman provided updates on ZIG development:

  • ZIG 0.15 targeted for August 1st release
  • Async Await revival discussed in Andrew Kelly's recent live stream, following an allocator-style pattern where IO objects are passed into functions and then you allocate with it. Likewise, if you want to do IO, you're going to pass in an IO object into your function and then do IO with it.
  • This will address a missing piece for building large products in ZIG

Constantine BLS Library Evaluation

Cayman reported on research into the Constantine BLS library by ex-Nimbus developer Mamy:

  • Features comprehensive thread pool implementation with detailed documentation
  • Pre-1.0 status due to pending security audit
  • Active development with audit in progress
  • Team will monitor for post-audit availability
  • Cannot be used until security audit is completed

Proposer Boost Testing Results

NC provided an update on the proposer boost feature testing:

  • Deployed to feat4 validator for approximately 6 days on mainnet
  • Caught 6-7 weak/late blocks and successfully suppressed FCU calls
  • Challenge: All captured blocks were too late (13+ seconds, extending into the next slot), making it impossible to test the actual reorg functionality
  • Some "unknown" reasons for FCU suppression logged, requiring further investigation
  • Phil removed the feature from the 1.32 milestone due to testing difficulties

Release Planning - Lodestar 1.32

The team discussed the 1.32 release status:

  • 36 commits ready for release
  • Key features include:
    • KZG switch to Rust implementation
    • Default gas limit increase to 45 million
  • Proposer boost feature removed from milestone due to testing challenges
  • Tuyen emphasized prioritizing PeerDAS refactor PRs merging to unstable after release
  • Matthew Keil noted several refactor PRs ready for review

Fusaka Devnet 2 and Future Work

Matthew mentioned merging work from Fusaka DevNet 2 into unstable branch, including:

  • BPO fork functionality
  • Fork digest updates
  • Consider what else needs to land on unstable and move it over to unstable

ERA Format for Blobs

History Pruning Implementation

  • The team discussed Ethereum execution clients' implementation of history pruning for pre-merge data on mainnet, with external history providers now becoming necessary for archive node users and application developers.
  • History providers can be found here: https://eth-clients.github.io/history-endpoints/

Updates

Bun Runtime Progress

Nazar provided updates on Bun-related development:

Devnet Performance

Phil celebrated Lodestar being the first client team to achieve all green status on all viable test combinations in fusaka-devnet-2

Lodestar Standup 2025-22 (June 3, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7885

Recording/Transcript: https://drive.google.com/drive/folders/1ydcYmPXusmjj8PAQyqd2V-nyPqcwfEfG?usp=sharing

Planning/Discussions

Standup Schedule and Travel Plans

The team discussed scheduling for the upcoming weeks due to ZuBerlin attendance.

The next standup was tentatively scheduled for July 8th, with the possibility of async updates during the interim period.

Release 1.31 Planning

  • The team finalized preparations for release 1.31, with Nico confirming the final PR had been merged. The release strategy involves cutting from unstable, creating an RC1 for sanity testing, then shipping.
  • The release primarily contains UX improvements with minimal functional changes that would affect core operations.
  • Post-release, the team plans to deploy to Lido to gather additional metrics and block data.

MEV Relay Configuration Updates

  • Based on conversations with Toni, the team discussed removing min-bid requirements on their Lido validators and connecting to the top four relays to reduce latency.
  • They evaluated different relay options, noting that Titan offers both filtering and non-filtering relays, with preference for non-censoring variants.
  • For Bloxroute, they identified avoiding the regulated relay in favor of their max profit option.

Berlin Conference Session Assignments

Assignment to sessions were noted and members were assigned accordingly to ensure we have coverage on all important breakout sessions. Please see internal chat for scheduling.

Cayman emphasized the importance of preparation, noting that attendees should read associated documentation before sessions to contribute meaningfully rather than passively absorbing information[1].

Ethereum Foundation Reorganization Impact

  • The team discussed significant changes at the Ethereum Foundation, including the elimination of several teams and reorganization into three initiatives: Scaling L1, Scaling L2/Blobs, and UX.
  • Notable eliminations included EthereumJS (despite millions of monthly downloads) and the entire Portal Network team.
  • The team expressed surprise at these decisions, particularly given EthereumJS's usage and their previous assumptions about Portal Network integration with history expiry.
  • This reorganization appears to shift EF focus more toward research rather than implementation and tooling.

DevNet Readiness and Technical Preparation

  • Nico identified several items needed for DevNet preparation, including merging unstable into PeerDAS branch to include SSZ updates, digest mismatch fixes, and blob schedule improvements to maintain spec compliance.
  • Matthew committed to handling the merge and addressing identified conflicts.
  • The team also discussed the need to merge proposer look-ahead functionality into unstable and subsequently port it to PeerDAS for fusaka-devnet-1.

Validator Custody and Backfill Discussion

Matthew provided extensive analysis of validator custody implementation challenges.

  • While the team has working validator custody functionality, they face significant challenges with the 18-day expiry window for DevNet participation.
  • The primary blocker is the lack of robust backfill capabilities, which Matthew described as a "fairly substantial body of work" that is "going to be pretty bugprone".
  • The current approach will involve enabling validator custody (available as a simple flag) but not announcing custody changes to avoid penalties during the initial period when historical data isn't fully available.
  • Matthew noted that the existing backfill code needs substantial refactoring, recommending a clean-slate approach using current range sync methodologies rather than the complex mix of range sync and root sync currently implemented.

Gas Limit Discussion

  • Phil raised the topic of the proposed 60 million gas limit increase, noting discussions from the ACDT call about EF signaling support.
  • The team confirmed this falls within CL gossip limits (which cap around 65-70 million gas before hitting 10MB block limits).
  • They plan to monitor existing HoleΕ‘ky or Sepolia nodes for performance data before implementing any default changes.

Future Roadmap Planning

  • The team scheduled a dedicated roadmap discussion for Monday evening (17:00-18:30) during the Berlin interop period.
  • This session will focus on technical direction decisions including the transition to Bun runtime and Zig-based library development, incorporating findings from Cayman, Tuyen, and Nazar's research into Lodestar's future architecture.

Updates

Matthew:

  • Matthew reported significant progress on by-range functionality debugging, including identification and resolution of multiple bugs through unit testing.
  • He encountered a challenging out-of-memory issue that consumed two days of debugging time, ultimately traced to fixture building processes.
  • Despite the debugging challenges, unit testing is proceeding smoothly and the codebase appears stable.
  • Matthew confirmed completion of the merge from unstable and readiness to proceed with additional small PRs.

Tuyen

  • Tuyen reported progress on data column site guide implementation and upcoming work on subnet subscription optimization.
  • He's working on preventing nodes from subscribing to their own subnets, with an existing PR available but blocked by current data column work.
  • The optimization focuses on topic-based subnet maintenance to improve network efficiency.

Nico

  • Nico provided updates on specification compliance gaps, identifying several areas needing attention beyond backfill capabilities. = He highlighted missing attestations as potentially more critical than backfill issues.
  • Additionally, he identified a timing issue in block building where header requests are delayed after block body construction, which could be interpreted as timing games and violates builder specifications.
  • Nico emphasized that while backfill is the major missing component, these other specification gaps also require attention.

NC (Async)

Lodestar Standup 2025-21 (May 27, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7851

Recording/Transcript: https://drive.google.com/drive/folders/17k_ReeYL9JjGBZ84t7eRimaMHaT9Y3ht?usp=sharing

Planning/Discussions

Ethereum Foundation Collaboration and Administrative Items

Phil initiated discussions about coordinating an Ethereum guild meeting with Toni from the Ethereum Foundation to discuss Lodestar's Lido validators and their impact on the network.

  • Meeting is tentatively scheduled for June 3rd
  • The team also discussed the Ethereum Season of Internships program, with Phil clarifying that they can host interns and EPF fellows as long as they work on different projects, contingent on team bandwidth and establishing minimum qualifications for engineering proficiency.
  • The team evaluated implementing a CI workflow for nightly spec test vectors, which would track unstable test vectors that may frequently fail. -Despite concerns about notification management and frequent failures, the consensus was to proceed with the implementation if the team finds it useful for development purposes.

Mainnet Attestation Performance Investigation

  • A significant discussion focused on sporadic attestation performance issues identified by Sam at ETH Panda Ops following the Pectra upgrade.
  • Analysis of validator indexes revealed that Lodestar nodes are experiencing inconsistent attestation inclusions, with some blocks showing only 40% participation in attestation zero while sync aggregates maintain 99% participation.
  • This represents a critical mainnet performance issue that appears sporadic rather than systematic.
  • Nico confirmed that individual validator performance remains excellent, with his validators achieving top 1% effectiveness ratings.
  • The problem specifically relates to attestation packing during block production, where attestations from the previous slot are being omitted inconsistently.
  • The team identified the need to implement comprehensive metrics dashboards to analyze:
    • block production timing
    • parent block processing
    • potential correlation with network conditions
    • reorgs

PeerDAS Performance Optimization

  • Matthew and Tuyen provided detailed analysis of performance bottlenecks identified through Sunnyside Labs testing.
  • Key optimization opportunities include:
    • batching KZG commitments, which could save approximately 160 milliseconds per operation
    • implementing signature verification caching to avoid redundant BLS verifications for identical block headers and signatures.
  • The team discussed CPU usage concerns, particularly related to subscribing to all 128 subnets instead of the standard 8 for normal nodes. - Tuyen identified message ID inconsistencies in gossip sub implementation that may be contributing to low peer scores and reduced mesh participation.
  • The team acknowledged that while fusaka-devnet-0 is functioning, performance optimizations are critical for mainnet readiness.

Berlin Interop Preparation

  • Work allocation for Berlin interop was discussed, with three main focus areas identified:
    • performance tuning and networking issue resolution,
    • PeerDAS implementation completion and testing
    • comprehensive unit testing of the new block input system.
  • The team emphasized the need for better preparation compared to previous Electra work

SSZ Library AVX Crash Resolution Strategy

  • A critical discussion addressed the AVX instruction crash affecting users without AVX support in their processors.

  • Matthew presented two potential solutions:

    • a short-term fix using CPU detection in the application layer with the existing nan library (which breaks Bun and Deno compatibility),
    • a longer-term solution implementing CPU detection in the Rust binding layer using napi-rs.

  • Cayman advocated for the Rust-based solution to maintain cross-runtime compatibility, while Matthew emphasized the urgency of addressing user crashes before the Berlin interop.

  • The team reached consensus to implement Matthew's temporary fix for the upcoming release candidate hotfix, with plans to develop the proper Rust-based solution post-interop.

  • This decision prioritizes immediate user experience while acknowledging technical debt.

Release Planning and Timeline Coordination

  • With team members traveling to Berlin and potential participation in subsequent events, the window for release management becomes constrained.
  • The consensus was to cut a hotfix 1.30.1 release candidate addressing the AVX crash issue, as this represents a critical user-facing problem that has caused users to switch to alternative clients.
  • Phil emphasized that without specific planning, the next release opportunity may not occur until after the Berlin interop period, making the current release candidate crucial for addressing known issues before the team's availability becomes limited.

Updates

Matthew

  • Matthew reported significant progress on cache PR implementation, incorporating feedback from Cayman and Tuyen with comprehensive unit testing additions.
  • He completed unit testing for the block input system, discovering and fixing minor bugs in the process.
  • Range code unit testing is substantially complete with only minor remaining work.
  • His investigation into the AVX crash issue in SSZ resulted in a working CPU detection solution, though it introduces compatibility issues with Bun and Deno runtimes.
  • He researched native C library alternatives and identified suitable options for long-term implementation.
  • Matthew emphasized the need to balance immediate user needs against technical debt considerations for the upcoming release.

Tuyen

  • Tuyen focused extensively on networking investigation for PeerDAS, discovering message ID inconsistencies in gossip sub implementation when connecting to different client types.
  • His analysis revealed that the publish function in gossip sub was not updating the message tracer, for which he submitted a PR.
  • He also identified duplicate message sending to peers and opened a PR to address sync contribution and proof metrics.
  • His attempt to merge v1.30.0 into the Bun branch encountered significant issues with high CPU usage and frequent disconnections, despite previous successful testing.
  • Given the focus required for PeerDAS optimization before interop, he decided to deprioritize Bun development in favor of mainnet performance issues.

Nico

  • Nico completed work on blob schedule functionality and addressed debugging issues raised by Barnabas, determining that reported problems were primarily on the external side rather than Lodestar implementation.
  • He identified an orphan block issue on the PeerDAS branch where block imports were occurring 11-12 seconds into the next slot, suggesting potential problems with the unknown block handling or peer discovery mechanisms.
  • His priorities for the current week include investigating the attestation packing issue and potentially implementing his minimal approach for parallel block processing refactoring.
  • However, he emphasized that bug fixes must take precedence over major refactoring efforts until critical issues are resolved.

NC

  • NC completed the proposal boost rearchitecture PR and addressed review feedback from Nico.
  • He opened an issue with the Attestant go-eth2 client regarding blob schedule format support in the beacon API, which affects the nightly Kurtosis CI testing.
  • This external dependency issue may impact other teams using similar testing infrastructure.
  • With available capacity for the current week, NC offered to contribute to either PeerDAS development or attestation-related issue investigation.

Cayman

  • Cayman achieved an additional 40% performance improvement in SSZ tree navigation
  • Collaborated with Matthew on block input unit testing, specifically focusing on get block by route functionality.
  • He worked with external teams on SSZ EIP development, planning collaboration opportunities for the Berlin interop period that may result in stable container EIP modifications.
  • His continued focus remains on block input system development and providing support where needed for the team's interop preparation efforts.

Katya

  • Katya opened a PR implementing getBlobv2 metrics to enable comparison between gossip and direct blob retrieval performance.
  • She continued development of validator activity metrics with significant refactoring collaboration with Nico.
  • Her work on the Grafana dashboard script addressed testing issues that were subsequently merged.
  • Volunteered to assist with performance investigation despite limited experience with specific metrics and monitoring, demonstrating willingness to learn debugging processes through shadowing experienced team members.

Gajinder

  • Gajinder worked on optimizations for data column sidecar requests but encountered resistance from other client teams who prefer batching blocks from different DAS sources via gossip rather than maintaining column consistency.
  • The optimization PR may be dropped pending broader client team consensus on the approach.

Lodestar Standup 2025-20 (May 20, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7833

Recording/Transcript: https://drive.google.com/drive/folders/11K-YGskKUk-V_wzX0HT-jst6eII5eaLz?usp=sharing

Planning/Discussions

Fusaka Progress & Berlin Interop Readiness

  • The team is focused on completing Fusaka-related goals ahead of the Berlin interop event.
  • Key goals include:
    • Completing the refactoring of the DA (Data Availability) module.
    • Merging the peerDAS branch into unstable. Progress is ongoing; most PR comments have been addressed, with unit testing and final reviews underway. Some unit tests are being added to the byRoot and byRange routes. Cayman is assisting with reviews.
    • Performance degradation has been observed in the peerDAS branch, but this will be addressed after functional code is merged and stabilized.
    • There is discussion on whether merging to unstable before interop is necessary. The main concern is spec test downloads, but recent improvements mean download speed is no longer a bottleneck. The consensus is to ensure the code is in a merge-ready state, even if not merged, to minimize risk during Berlin.
    • NC will reuse the peerDAS branch for Fusaka-Devnet-0, and we will cherry-pick commits such as BPO fork as necessary.

DevNet and Gossip Score Issues

  • Tuyen noted low gossip scores on DevNet, particularly for p7 where there are a lot of broken IWANT promises due to slow responsiveness and not related to main thread.
  • There are ongoing efforts to update to the latest consensus spec tests, though some new test runners and processing logic require more sophisticated updates to pass.

EF Security "Trillion Dollar Initiative" Feedback

  • The EF Security team is soliciting feedback from client teams on improving Ethereum security, including user experience (UX) concerns.
  • Team members highlighted persistent wallet UX issues:
    • Lack of transparency in what users are signing, particularly with complex contract interactions and account abstraction.
    • Even advanced hardware wallets often resort to blind signing for contracts, which is a security risk.
    • The idea of a "debug transaction" RPC endpoint was discussed, allowing users or wallets to simulate transactions and see their effects before signing. Existing tools and UIs (like MetaMask) already simulate transactions, but trust in the UI remains a concern.
  • The team acknowledged that while simulation tools exist, reliance on UI can still be a vector for attacks (e.g., Safe wallet incident), and more robust, protocol-level solutions may be needed.

Lido Min-Bid and APR Discussion

  • The team revisited the use of the min-bid parameter in Lido node operations, noting that changing network conditions (e.g., more local blocks, lower MEV) have made the current value potentially outdated.
  • It was observed that the team's APR is about 0.3% lower than top operators, despite similar effectiveness, likely due to the min-bid setting.
  • There is interest in collecting more data (e.g., from rated.network and Lido's reporting) to inform whether the min-bid should be adjusted for better network alignment and operator competitiveness.
  • The difference between builder and local block values has increased, raising questions about the current selection logic and whether it is still optimal.

P2P Config/Fork Handling and BPO Forks

  • Ongoing discussions about how to handle configuration mismatches and fork awareness in the P2P protocol, especially as BPO (blob per block parameterization) and future forks proliferate.
  • Gajinder proposed an EIP to canonicalize and Merkle-ize the active configuration, allowing handshake-based config verification between peers, reducing the risk of accidental peering across incompatible forks.
  • There is some skepticism about feasibility for the Fulu fork, but consensus that long-term, protocol-level config verification is needed for network health and security.

Testnet Gas Limit Increases

  • Phil checked with the team about increasing gas limits on the Hoodi testnet to 60M, noting no issues so far and alignment with Sepolia's current settings.

Sync Aggregate Issue

  • Nico raised a concern about mainnet blocks with zero sync aggregates, an unexpected situation possibly introduced by recent changes (e.g., Pectra). The team agreed to open a thread for further investigation.

Updates

Cayman

  • Completed major work on the SSZ-zig library's persistent Merkle tree, achieving 5–15x performance improvements in tree navigation.
  • Ready to merge a large refactor PR; will shift focus to block input refactor to help accelerate progress.
  • Released bun-ffi utility library v1.0, enabling faster and more efficient use of native modules with Bun.

Tuyen

  • Worked on Bun bindings with Cayman; only upgrade Bun FFI to the correct release version remains before merging and releasing.
  • Preparing to integrate into the peerDAS branch; merged unstable into peerDAS, with the PR ready and green.
  • Fixed an issue with PeerDAS long job wait time due to isProcessingCurrentSlotBlock. Context can be found here: https://github.com/ChainSafe/lodestar/issues/7847#issuecomment-2893179748
  • Gossip score in PeerDAS is low, that's why we're not acccepting messages from some peers, should be resolved before the interop.
  • We don't run the merkle spec test in peerDAS. Discovered when trying to merge unstable in to peerDAS.
  • When we get the merkle proof, we fail it - might be an issue in SSZ but not sure why because it's been stable for a while.
  • Missing new networking spec tests in peerDAS: https://github.com/ChainSafe/lodestar/issues/7839

Matthew

  • Progressed on Fusaka PRs, with unit tests and reviews ongoing.
  • Accidentally squash-merged a branch intended for regular merge; will revert and re-merge as required.
  • Continues to target merging into unstable before Berlin.

Nico

  • Reviewed BarnabΓ‘s's blob schedule work; branch is ready pending review and interop considerations with other clients.
  • Plans to resume work on block parallelization after DevNet zero is ready.
  • Discussed the need for improved fork/config handling in P2P.

NC

  • Worked on BPO PR, tested with BarnabΓ‘s's config, and followed BPO P2P discussions.
  • Updated Deneb attestation code and error messages, merged related PRs.
  • Fixed rewards API pre-state calculation.
  • Kurtosis CI test PR merged and running smoothly.
  • Reorg proposal PR will now include both refactor and fix, as little refactoring was needed.

Gajinder

  • Opened a PR for consensus spec optimization, specifically around data sidecar byRoot request/response.
  • Provided feedback on Justin's KZG crypto crate.
  • Participated in BPO fork discussions, advocating for handshake-based config verification.
  • May draft an EIP to formalize Merkle-ized config for peer handshakes with Nico.

Katya

  • Completed validator monitor refactor; follow-up PRs are open and approved.
  • Set up Ansible and resolved Python version/server storage issues.
  • Cleaned up server images and restored validator operations.
  • Working on pure dust metrics and a Kurtosis dashboard; validator activity metrics PR is ready for review.

Nazar

  • Investigated Bun vs. NodeJS runtime discrepancies (e.g., MessageChannel differences) impacting tooling.
  • Developing polyfills to bridge compatibility gaps.
  • JSR publishing PR for SSZ packages pending Cayman's review; differential states PR reviewed by Matthew, awaiting further feedback.

Lodestar Standup 2025-19 (May 13, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7807

Recording/Transcript: https://drive.google.com/drive/folders/1YF_GCA4aFp037AIr_aYUax4Ljtc0LoMD?usp=sharing

Planning/Discussions

PeerDAS Integration

  • Matthew reported that PeerDAS DevNet7 is running smoothly with all monitored nodes performing well along with moving keys over to our local validators
  • Cayman's PR for PeerDAS has been merged, with Matthew working on a follow-up PR that will be pushed shortly
  • Team expects PeerDAS integration into the unstable branch within approximately two weeks
  • Integration work is being done incrementally with most components being merged directly to unstable as the refactoring progresses

Version 1.30.0 Release Review

  • The team performed an initial review of version 1.30 with no major issues detected
  • Nico identified one broken VM host panel issue, but this is unrelated to core functionality and will be fixed separately
  • Team noted that post-Pectra performance metrics show Lodestar performing better than before, while the network average has actually declined
  • Performance improvement appears to be related to optimized attestation handling, placing Lodestar in a favorable position compared to other clients

Blockprint Infrastructure

  • Chiemerie is setting up Blockprint for both the Execution Layer (EL) and Consensus Layer (CL)
  • The team discussed using the Blockprint's Blockdreamer for triggering block production, which Nico confirmed had been recently updated (within 10 hours) to add Electra support
  • Plan is to extract and trigger the build process to send blocks to the SigP Blockgauge instance rather than setting up a separate block print instance
  • Team decided to implement a prudent strategy regarding Electric Eel (Eleel) integration: start without it and implement only if block production degradation is observed
  • Concerns were raised about potential performance with blocks every 12 seconds, which may require Eleel faster block creation capabilities

Block Production Parallelization

  • Nazar and Nico presented different approaches to parallelizing block production steps
  • Two competing strategies are being considered: a minimal approach versus more extensive refactoring
    • Nazar believes refactoring is needed due to the code's complexity
    • Nico is concerned about introducing regressions.
    • Matthew also agreed that refactoring is a good idea but echoed concerns about the complexity and suggested a long testing period for any changes
  • Team acknowledged that the current block production code path is complex with up to 6-8 levels of indentation, making it difficult to read, understand, and debug
  • Matthew suggested that any refactoring should be thoroughly tested over an extended period (up to a month) to ensure no regressions occur under various network conditions
  • Cayman proposed using Blockdreamer to test block production every slot for comprehensive validation
  • Discussion will continue asynchronously in a dedicated Discord thread to determine the optimal approach

Proposer Boost Implementation

  • Team discussed how the proposed booster should work according to specification requirements
  • Technical conversation centered around ForkchoiceUpdate (FCU) call timing and suppression strategies
  • NC and Nico agreed on when to issue and suppress FCU calls, particularly regarding previous slot and T0 behaviors
  • Team considered Lighthouse's approach of issuing another FCU call 500 milliseconds before the slot
  • The discussion continued around ensuring the EL (Execution Layer) does not reorg due to multiple FCU calls.
  • Tuyen suggested issuing two FCU calls with different payload IDs, but NC believed the EL would ignore the second call
  • Nico clarified that the EL can be told to prepare a payload for the parent and later for the weak block without causing a reorg

CI/CD Infrastructure Improvements

  • NC opened a draft PR implementing Kurtosis CI nightly tests, which currently run in approximately 18 minutes
  • Additional test suites from external sources are available but not yet integrated, including Pectra, maxEB, and consolidation tests
  • Team decided to schedule the automated CI action at midnight UTC to minimize disruption to development workflows
  • Infrastructure will be adjusted if timing becomes problematic for team members across different time zones

Grafana Dashboard Access

  • Chiemerie acknowledged Cayman's request about sharing dashboards publicly
  • Work is underway to provide public access to Grafana dashboards before the week ends
  • Team discussed the feature availability on both self-hosted and cloud instance

Updates

Matthew

  • Successfully maintaining PeerDAS DevNet7 with all monitoring showing positive results
  • Rebased and fixed issues in a PR that will be pushed within hours
  • Preparing additional PR with components already broken out from previous work

Nico

  • Investigated approaches for block refactoring parallelization, focusing on understanding constraints before proceeding with implementation
  • Reviewed the proposer boost topic specification and implementation details
  • Noticed that Lodestar's performance improved post-Pectra while the overall network performance declined
  • Observed issues with Geth during reorgs when trying to incorporate multiple FCU calls
  • Working on various PR reviews

Nazar

  • Working on integrating Bun-related unit tests into the Lodestar repository to track working and broken components
  • Developing a PR related to state diffing that will be opened soon
  • Advocating for refactoring complex block production code paths to reduce technical debt
  • Created simplified code snippets to help team members evaluate different parallelization approaches

NC

  • Working on proposer boost reorg architecture implementation details
  • Set up Kurtosis CI integration with basic tests completing in approximately 18 minutes
  • Planning to add additional test suites for Pectra, max EB, and consolidation from available resources
  • Reached consensus with Nico on FCU call timing strategies for proposed booster implementation
  • Preparing to configure the CI workflow to run at midnight UTC

Tuyen

  • Completed work on building for State Transition util with Bun integration
  • Developed a tool to handle publishing Bun libraries for different platforms
  • Working on implementing top-level await functionality based on Cayman's feedback
  • Migrated BLST-Z version to Zig 0.14
  • Fixed merge issue for PeerDAS
  • Identified performance issue with validating messages in PeerDAS taking approximately 30 seconds to execute

Katya

  • Set up mainnet beacon node for testing and monitoring
  • Facing configuration challenges with Grafana when attempting to run both Hoodi and mainnet nodes simultaneously
  • Completed validator monitor metrics reorg
  • Discussing options with Nico for either adding network labels to Prometheus config or running separate Prometheus/Grafana instances
  • Planning to use Ansible for validator metrics configuration

Cayman

  • Successfully merged core data structures for the data availability and block input refactors with Matthew's help
  • Creating tooling to support Zig projects with platform-specific library builds similar to the NAPI-RS approach
  • Starting work on persistent Merkle tree implementation for SSZ-zig
  • Developing proof infrastructure to differentiate Lodestar from other clients
  • Supporting Tuyen with Bun integration work
  • Highlighted Bun's advantages including built-in S3 client capabilities and better tooling

Chiemerie

  • Setting up both Execution Layer and Consensus Layer for Blockprint
  • Researching options for Electric Eel implementation
  • Working on providing public access to shared Grafana dashboards

Mercy (External Contributor)

  • Working on optimizing node traversal by combining get-all-nodes functionality into a single traversal method
  • Seeking guidance on implementation approach based on team feedback

The meeting concluded with a final note to have Tuyen and Cayman review metrics for the upcoming release, and Tuyen highlighting a significant performance issue with message validation in PeerDAS that requires attention.

Action Items

  • Matthew to complete and push PeerDAS-related PRs
  • Team to review 13.0 release metrics before final approval
  • Chiemerie to implement public access to Grafana dashboards
  • Further discussion needed on block production parallelization approach
  • Matthew to investigate performance issue with PeerDAS message validation
  • Cayman to provide reviews for Matthew's upcoming PRs when ready

Lodestar Standup 2025-18 (May 6, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7778

Recording/Transcript: https://drive.google.com/drive/folders/1dW0nBmlAO_Wt_gBemSnQbajgtvhhUrjQ?usp=sharing

Planning/Discussions

Pectra Fork Final Preparations

  • Team confirmed all Pectra-related preparations are complete
  • Pectra fork is scheduled to occur in approximately 20 hours
  • Phil acknowledged all mainnet preparations have been handled by the infrastructure team

Release Planning

  • Team agreed to target a release before merging PeerDAS changes
  • Aiming to cut RC on May 9th (Friday) with a potential v1.30 release on May 13th
  • Cayman emphasized the importance of maintaining the regular release cadence
  • Proposer boost feature was deprioritized over concerns about its implementation:
    • Nico noted it's a significant change despite small PR size as it "triggers a lot of code"
    • The implementation might not be correct based on current reviews
  • Discussion on potentially including async aggregate and snappy changes:
    • Nico shared metrics from Gnosis nodes showing promising improvements
    • Metrics show improved event loop lag performance, especially important for 5-second slot times
    • Team will review additional metrics before deciding

Testing Strategy

  • Discussed integration of Kurtosis tests alongside current sim tests
  • Team agreed to maintain current sim tests as lightweight sanity checks for PR validation
  • Will implement more comprehensive Kurtosis tests to run nightly rather than on every PR
  • Focus on basic functionality rather than extensive assertions to prevent test instability
  • NC shared examples from Reth team's approach of running tests every 12 hours
  • How much do we want to assert for every PR and nightly?
    • A simple sanity check: Can the chain progress? It should at least progress and finalize.
    • Maybe consider the top 3 assertions and use those?
    • One request that was made from NC was for sim test assertions with execution requests (calling, sending and checking deposit transactions)
  • There is some value in the current sim tests though, like how we found out Lighthouse deprecated mplex.
  • Sim tests: Should be interop related tests. Can we peer with Lighthouse? Can we work with mixed clients? Can we interop with Geth/Nethermind?
  • Kurtosis tests: Nightly cron job run with more assertions.
  • We may have self-hosted runners we should look into specifically for purposes like this.
  • Nazar mentioned fixing e2e leaky tests (PRs #7762 and #7780)

Simple-serialize.com Extraction

  • Team confirmed agreement to extract simple-serialize.com from the main repository
  • Nazar received confirmation to preserve git history during the extraction process
  • Plan is to add an extra remote to the new repo and push the master branch
  • Team acknowledged the website needs improvement, but extraction will increase exposure to front-end developers

Unit Tests Preset Configuration

  • Discussion about whether to use minimal or mainnet preset for unit tests
  • Nico investigated and found minimal performance difference between presets for unit tests
  • Team agreed to use mainnet preset for unit tests as it's easier to reason about (32 slots vs. 8 slots)
  • Will continue using minimal preset for sim/e2e tests where the performance improvement is significant
  • Some tests that are more end-to-end in nature will be refactored and moved accordingly

Women in Ethereum Protocol (WiEP) Feedback Discussion

  • Katya requested feedback on WiEP contributors
  • Nico provided feedback that contributors are enthusiastic but face technical knowledge barriers
  • Suggestions for improvement:
    • Prepare easier "good first issues" that are additive rather than refactoring
    • Create fresh issues rather than using old ones
    • Potentially create onboarding materials specific to Lodestar structure
    • Consider recordings explaining the repository structure

Updates

Matthew PeerDAS & DevNet Updates

  • Made progress on block input refactoring PRs
  • Blob code changes are being merged into unstable branch
  • Work on range and gossip syncing mechanisms progressing
  • Performance testing planned for next week once changes are merged
  • DevNet 7 being planned for this week after Pectra fork
  • Sunnyside network crashed and stopped syncing, team working on identifying the issue
  • Bug likely related to validator custody changes and column retrieval from database

Tuyen State Transition in Zig

  • Made progress on state transition utilities in Zig
  • Identified performance issues with SHA256 implementation:
    • Shuffling is 20% slower compared to Rust implementation
    • Committee indices computation is 40% slower
  • Fixed build issues in the hash tree implementation
  • Added unit tests and benchmarks
  • Expects to release the beta version within the week
  • Plans to start integration with Bun branch next week

Derek PeerDAS Updates

  • Available to help with DevNet 7 branch
  • Temporarily reassigned to work on general testing of the network
  • Will focus on improving stability and Kurtosis tooling
  • Working on identifying syncing issues across clients
  • May help with ProbeLabs testing to verify clients are properly custodying advertised data

Katya Validator & Beacon Node Metrics

  • Updated beacon node for Petra using quickstart
  • Working on moving validator monitor metrics
  • Planning to implement Tuyen's validator metrics after current PRs are merged
  • Received guidance on what to monitor during Petra fork (blocks, attestations, justification, finalization)

Gajinder getBlobsv2 Metrics

  • Shared metrics discussion from recent call
  • Reported 60-70% hit rate for getBlobs v2, validating the team's observations
  • Analysis showed that in 30% of failed cases, typically one blob was missing
  • Block blob propagation metrics show improvement:
    • Previous: 700ms for six blobs at 95th percentile
    • Current: approximately 200ms with aggressive pull strategies
  • Working on spec PR to improve get data column sidecar API
  • Engaged in discussions about edge cases for dynamic validator custody updates

NC Proposal Boost Rework

  • Reviewing proposal boost rework implementation
  • Nico noted importance of first determining theoretical FCU call patterns before implementation
  • Attended FOCIL breakout session which presented a solution for delay execution coexisting with FOCIL
  • This could potentially allow both EIPs to go into Glamsterdam

Chiemierie

  • Chiemerie confirmed he'll look into Kurtosis setup

Glenn

  • Glenn introduced himself as taking over communications
  • Mentioned upcoming community update next week and plans for internal newsletters

Cayman

  • Working on data availability refactoring and Zig build tools

Nazar

  • Merged two PRs related to binary improvements, with two more coming this week

Nico

  • Nico: Merged various PRs and added tests for "staring" functionality

Lodestar Standup 2025-17 (April 29, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7739

Recording/Transcript: https://drive.google.com/drive/folders/1YYXA-M-7aeaZeFxLvmju2EGTsRZ_axjN?usp=sharing

Planning/Discussions

PeerDAS Development Status and DevNet Readiness

  • Lodestar team is "in the middle of the pack" with PeerDAS implementation and ready for the next devnet-7.
  • All necessary PRs have been merged, with a few more open from Derek and Hugh.
  • The team has been conducting extensive testing with Kurtosis to ensure everything works smoothly.
  • Standby for another list of remaining items from Derek relating to PeerDAS work. Will be available on Discord as needed but ramping down contributions.

A significant technical challenge discussed was column syncing, which Derek described as deceptively complex despite being just "a sentence in the spec." The team identified several implementation challenges:

  • Managing peer prioritization when syncing columns
  • Handling scenarios where specific columns aren't available from peers
  • Decision points around whether to block syncing or allow blocks to run ahead without data columns

The team also discussed issues observed in DevNet-6:

  • Validator custody operations ran "wild" with fluctuating custody group counts.
  • When nodes fell behind, they couldn't catch up because "nobody knew who had what columns anymore".

Refactoring Goals for Berlin Interop

The team discussed two major goals to accomplish before the Berlin interop in June:

  1. Complete the refactoring of DA (Data Availability) modeling to make it more conceptually clear and easier to work with
  2. Get the PeerDAS branch merged into unstable
  • Matthew reported progress on the first piece of the refactor code structure, having gone through three different versions before settling on the right approach. He's currently working on block synchronization components including "by range," "by route," and "unknown block input sync".
  • Phil emphasized the importance of improving visibility into their own codebase to better identify problems during the interop. The team agreed that having PeerDAS merged into unstable before the interop would significantly improve their ability to iterate quickly without having to switch between branches.
  • We should also target unstable for further Fulu related implementations as soon as we can get PeerDAS branch merged into unstable (after mainnet Pectra fork and stability). Cayman noted that while the timeline is tight, both goals are achievable and would greatly benefit their workflow during the interop.

Gnosis Node Testing Strategy

  • The team discussed setting up Gnosis nodes to gain performance insights, particularly around handling shorter slot times.
  • Nico emphasized that running Gnosis nodes in their test infrastructure would provide valuable data on how different slot timing affects Lodestar implementation

The team discussed potential approaches:

  • Running a basic node without subscribe all subnets
  • Potentially running validators (requiring GNO tokens at approximately 130 USD per validator)
  • Reaching out to the Gnosis Foundation to potentially delegate validators without requiring token ownership
  • We will run a gnosis group of servers that will collect data on our performance on the Gnosis network with and without subscribe all subnets.

Test Infrastructure Improvements

Nazar reported investigating flaky end-to-end tests that have been causing issues. The team debated whether to fix, delete, or modify these tests:

  1. Nazar noted that while flaky, the tests cover important edge cases that verify specific error conditions crucial to network request flow
  2. Nico observed that the tests appear to fail due to timing/race conditions, particularly when CI runners are under load

Several potential solutions were discussed:

  • Moving expensive/flaky tests to nightly cron jobs
  • Using beefier CI runners
  • Implementing self-hosted runners to eliminate GitHub runner-specific issues
  • Identifying specific configuration mismatches causing connection refusal errors

Matt suggested the possibility that some issues might be related to how GitHub builds runners, potentially causing cross-VM interference.

Nazar will continue investigating and we can conduct experiments with different hardware if we feel it is a cause of the issue.

Validator Genesis State Root Verification

Mercy (external contributor) brought up work on validating Genesis state roots across different networks. The team discussed the value of embedding state roots in the client:

  1. Nico explained that currently we "yolo download the state from GitHub" without validation, and having Genesis state routes baked into the client would improve verification
  2. The team debated extending this approach to weak subjectivity checkpoints, with g11tech suggesting that regular releases could include updated checkpoints as a value-add
  3. Concerns were raised about baking in checkpoints potentially undermining the community consensus aspect of weak subjectivity, with Nico noting "the point of weak subjectivity is that it should be rough community consensus around the right chain, not controlled by five teams"

Updates

Cayman

  • Completed work on the Zig SSZ implementation, passing all spec tests with promising benchmark results
  • Ready to merge the refactored branch to the main branch
  • Next step is creating codegen to expose the library to Bun via a NAPI-compatible interface
  • Made progress on libP2P-quic implementation but encountered segfaults requiring further investigation

Nazar

  • Worked on parallelizing block processing and production to improve performance
  • Implemented processing steps for generating common block body and builder/engine block body in parallel
  • Received and addressed feedback from Tuyen on the implementation
  • Currently investigating flaky tests focusing on connection issues in the libp2p layer

Nico

  • Completed cleanup work after the Electra release
  • Removed produce block v2 APIs to simplify the codebase
  • Investigated performance issues reported by Luca on Gnosis, identified high CPU usage in "aggregate with randomness" operations
  • Reviewing and merging various PRs

Katya

  • State transition implementation merged with a follow-up issue identified for later work
  • Currently working on validator monitor metrics implementation
  • Approved for attendance at protocol.berlin

Derek & Hugh (Base Contributors)

  • Continuing testing of PeerDAS implementation this week
  • Will create comprehensive documentation of remaining work needed for PeerDAS
  • Transitioning off the Lodestar project but will remain available via Discord

Lodestar Standup 2025-16 (April 22, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7706

Recording/Transcript: https://drive.google.com/drive/folders/1ASUYXlWvKEzJ1VcfmYS7UOvhSTtlQors?usp=sharing

Planning/Discussions

PeerDAS DevNet 6 Stability Issues

DevNet 6 experienced significant stability issues resulting in network failure. Multiple factors contributed to the problems:

  • Bad blocks causing forking across all clients
  • No client was able to stay synced even with itself, suggesting bugs across all implementations
  • Root cause appears to be related to incomplete validator custody implementations
  • Matthew explained that teams had to merge half-finished validator custody code when focus shifted to "getBlobsV2" and "getPayloadV5" implementation
  • Derek mentioned the initial trigger may have been EL out-of-memory issues with Reth and Geth that cascaded to Nethermind

Validator Custody Implementation Concerns

Gajinder raised important concerns about peer custody updates:

  • Current Lodestar implementation uses static custody settings rather than dynamic ones
  • When peer custody changes dynamically, there's no established protocol for notifying other peers
  • Peers need to be informed to update their metadata when custody changes
  • The team discussed whether metadata is updated with each ping or if additional mechanisms are needed
  • This issue wasn't previously considered in the peerDAS planning discussions

getPayloadV5 Implementation Status

The team discussed the relationship between getPayloadV5 implementation and DevNet 6 issues:

  • The PR for getPayloadV5 is ready for final approval from Gajinder
  • Gajinder clarified that in the current version, proofs from EL are discarded and replaced with self-computed proofs
  • In getPayloadV5, the proofs sent from EL will be used directly
  • This change relates to how cell proofs vs. blob proofs are handled

Testing Configuration Standards

Nico initiated a discussion about test configuration standardization:

  • Question about which presets (mainnet vs. minimal) to use in unit and end-to-end tests
  • Nazar explained that only beacon node and state transition packages are still using mainnet preset for unit tests due to hardcoded fixtures
  • Team consensus is to prefer minimal preset for faster tests where possible
  • Noted concern that some unit tests take over a minute to run, which is excessive for developer feedback cycles
  • Nazar will open an issue to address removing mainnet preset from unit tests

Ethereum Protocol Fellowship Opportunities

Phil noted that EPF applications are open:

  • Encouraged directing promising contributors to apply
  • Team can mentor EPF participants on Lodestar projects
  • The beacon harness has been tagged as a potential EPF project
  • Called for team members interested in mentoring to sign up and to tag any other projects of potential for EPF candidates

Pectra Fork Preparation

The upcoming Pectra fork is scheduled for May 7th:

  • Will be early morning in North America, daytime in Europe
  • Configuration for mainnet has been triple-checked
  • Team is hoping for a smooth transition unlike some previous forks that required extensive debugging

SSZ in Beacon API

Nico reported increased momentum for SSZ adoption in the Beacon API:

  • Prysm is now pushing for full SSZ Beacon API implementation
  • Discussion about making it mandatory in Fusaka
  • Performance concerns with hex-encoded blobs driving the need for SSZ
  • Lighthouse doesn't see JSON serialization as a significant issue

Updates

Nico:

  • Critical vulnerability fix in consensus has been merged and is now public
  • Made improvements based on NC's ideas using a map
  • Fixed boot node issues identified by Nethermind on Chiado testnet
  • Reviewed Electra-related PRs
  • Participated in SSZ Beacon API discussions with other client teams

NC:

  • Completed review of all state PRs, awaiting Nazar's feedback implementation
  • Opened Blob parameter only fork work PR, discovered common unhandled corner cases with Derek's parallel work
  • Updated weak subjectivity calculation for Electra
  • Reported on FOCIL breakout meeting where Prysm and Lodestar are successfully interacting, while Teku continues having issues with incorrect root calculations

Katya:

  • Completed validator monitor implementation (now merged)
  • Working on validator monitor metrics and checks
  • Investigating issues with ParaD monitor running locally
  • Implemented validator activity metrics for active exits, consolidations, and withdrawals
  • Applied script for Grafana dashboard updates but facing version compatibility issues
  • Working on state transition metrics
  • Plans to collaborate with Derek on pure DAS metrics[1]

Tuyen:

  • Tested multi-functions implementation - all working except randomness due to WASM callback issue
  • Continued work on pubkey index map implementation, using implementation from existing Zig code
  • Fixed issue with handling zero block returns
  • Fixed metric issue by tracking group index instead of group column index
  • Working on additional metrics for range syncing

Nazar:

  • Working on parallelizing block processing
  • Current implementation processes block body attributes sequentially
  • Opened PR with necessary refactoring to split implementation into smaller parallelizable functions
  • Will address feedback on state diff PR based on NC's review

Derek:

  • Planning to focus on testing this week
  • Will combine writing tests with running DevNets to identify any unusual behaviors

Gajinder

  • Reviewing PeerDAS PRs as requested by team members
  • Investigating new exit mechanism proposed by Mike Neuter

Other Updates

  • Phil noted the peerDAS branch is now running on Hoodi testnet and visible in Grafana under "nextfork" group for metric collection

Lodestar Standup 2025-15 (April 15, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7699

Recording/Transcript: https://drive.google.com/drive/folders/1uvbbHIZlPYsFqRuz3N6K60A4Vt0Zj4Wi?usp=sharing

Planning/Discussions

DevNet 6 Status and Implementation

DevNet 6 deployment is progressing well with the team successfully getting nodes stood up and participating in the network. Matthew reported that the three major features required for DevNet 6 are either merged and deployed or ready to merge. However, there are some remaining implementation tasks:

  • Hugh and Derek are working on updating validator custody implementation components, specifically:
    • Updating peer registry to cache custody information from peers during status exchanges
    • Updating network caches for optimized data fetching from appropriate peers
    • Implementing ENR (Ethereum Node Record) updates when metadata changes

Tuyen noted peer stability issues in DevNet 6:

  • Reporting low peer counts and connection failures: which is why we have hiccup issues sometimes getting out of sync and then getting back to normal.
  • He observed that about 90% of disconnections were due to errors, and many peers were being ignored due to bad scoring.

Barnabas has reported that everything is going smoothly from the network perspective, despite these local peer issues[1].

PR Review Process and Workflow

The team engaged in a thorough discussion about improving the PR review workflow. Matthew proposed formalizing the review request process when PRs don't get sufficient attention:

  • Developers should request specific reviewers through GitHub UI when PRs languish
  • This would serve as a signal that a PR needs immediate attention without requiring centralized management
  • Team members can decline reviews if they don't have the expertise or bandwidth

Nico expressed concerns about formalized assignment creating pressure to prioritize others' work over current tasks. Nazar suggested it's a human factor that helps ensure teamwork and prevents PRs from hanging.

The team reached consensus that:

  • The current workflow works well for most situations
  • Formal reviewer requests should be used sparingly when normal processes fail
  • Monthly PR grooming will continue to address stale PRs
  • The goal is to balance individual priorities with team needs

Pectra Fork Readiness

The team discussed preparations for version 1.29 release and Pectra fork readiness:

  • Tuyen identified P2P issues as a potential blocker for 1.29
  • Nico mentioned work on a PR for 1.29 that needs to be merged
  • Phil assigned issue #7680 (producing block for build common block body while waiting for the builder) to Nazar
  • The team noted timing may not be critical as mainnet releases would likely happen after the current development cycle

Nazar is working with NC on state diffing PRs, with a focus on breaking down the implementation into smaller, reviewable pieces.

PeerDAS CI Issues

Tuyen raised concerns about failing tests and linting issues in PeerDAS.

  • Requesting team assistance to fix CI problems:
  • Need more people to help with getting CI green because it's annoying to merge PRs.
  • It doesn't require deep knowledge on PeerDAS to make this work.

The team acknowledged the importance of cleaning up CI before merging to unstable, with Phil encouraging everyone to contribute to this effort to make the process smoother.

"War Mode" Testnet Invitation

Matthew shared an invitation from Soispoke for the team to participate in a security-focused "war mode" testnet:

  • The initiative aims to simulate various attack scenarios to test network resilience
  • Monthly attacks would test different security aspects (51% attacks, finality reversion, censorship, P2P DoS, etc.)
  • The project appears to have originated from Vitalik's concern about "peacetime assumptions" in protocol design
  • Some team members expressed interest, particularly Nico who mentioned this aligns with concerns since the Bellatrix issue
  • Concerns were raised about publishing attack tools publicly, with suggestions to keep such capabilities in private forks

The team agreed to explore this opportunity further, particularly after current fork priorities are addressed.

Updates

Matt:

  • Successfully deployed to DevNet 6 with all three major features ready
  • Worked on refactoring, with first PR moving enums around and pushing branch for NEB classes
  • Developing a roadmap for code reviews and merging of refactored components
  • Identified remaining validator custody implementation needs with Derek and Hugh

Tuyen:

  • Identified peer stability issues in DevNet 6 with low peer counts and connection failures
  • We should cherry pick noble issue from unstable to peerdas to see if it helps with networking
  • Working on implementing BLST-Z, achieving nearly 10x performance improvement in benchmarks
  • Noted concerns about stability with the new implementation that requires further testing
  • Planning to fix tests and implement reflective changes in the Bun side for BLST-Z release

NC

  • Reviewed Tuyen's attestation packing PRs
  • Shifted focus to hierarchical state diff research
  • Studied Lighthouse implementation and reviewed Anton's hackmd notes
  • Started reviewing PRs related to state diffing

Nico

  • Reviewed other clients' attestation packing implementations compared to Tuyen's work
  • Conducted preliminary analysis showing good results from the packing optimizations
  • Working on MEV topics and researching EPBS implementation challenges
  • Planning Ethereum guild discussions on MEV topics with potential guest speakers. Aiming for after Pectra hard fork.

Nazar

  • Cleaned up linting and tooling-related issues
  • Researched bun implementation differences compared to NodeJS
  • Continuing work on state PRs based on NC's feedback
  • Breaking down state feature into smaller, reviewable pieces to facilitate merging

Katya

  • Had first PR merged
  • Worked with Jim on server-side setups, restarting beacon node and setting up Grafana dashboard
  • Currently working on validator monitor with Nico's assistance
  • Launched local beacon node in testing mode (encountering hashing issues on Intel Mac that required workarounds)
  • Note that hashtree upstream in Potuz's implementation doesn't support Darwin x86.

Gajinder

  • Reviewed PRs on PeerDAS
  • Commented on BPF4 configs and other specification-related matters

Lodestar Standup 2025-14 (April 8, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7652

Recording/Transcript: https://drive.google.com/drive/folders/1ULDPYnf--vSKuGIJY8y0W9rs2H9lExnC?usp=sharing

Planning/Discussions

PeerDAS Implementation

Validator Custody PR

  • Derek reported that the basic validator custody PR is ready to go, which implements dynamic custody group count without backfilling old columns
  • The PR will include a flag to turn the feature on/off to prevent downscore issues since backfilling isn't implemented
  • Matthew spent time reviewing the architecture and will finalize the PR review

DevNet 6 Requirements Clarification

  • Gajinder provided important clarification about distributed blob building vs. current PeerDAS implementation
  • Distributed blob building (where ELs distribute blobs based on node custody groups) is a future goal and not part of PeerDAS yet, as the spec isn't clear
  • Two critical features needed for DevNet 6:
    1. Engine get_blobs_v2 API implementation
    2. Supporting cell proofs in transaction bundles

Expanded Details on the Discussion Around get_blobs_v2 for DevNet 6

1. Purpose and Behavior of get_blobs_v2

  • The get_blobs_v2 API is designed to allow Consensus Layer (CL) clients to fetch blobs from their local Execution Layer (EL) if the EL has all the required blobs for a block. This makes it a "hit-or-miss" feature:
    • If all blobs are available locally, the API provides them.
    • If even one blob is missing, the response is empty.
  • This behavior contrasts with distributed blob publishing, which is a future goal where ELs would aggressively distribute partial blob data (columns) based on custody groups. However, distributed blob publishing is not yet part of PeerDAS or DevNet 6 due to incomplete specifications.

2. Metrics and Data Collection

  • The team emphasized the importance of collecting metrics to evaluate the effectiveness of get_blobs_v2:
    • Metrics should measure how often get_blobs_v2 would succeed in real-world scenarios by tracking hit and miss rates.
    • Current metrics only show how many blobs were fetched but do not indicate whether all blobs for a block were locally available. This makes it challenging to assess the utility of get_blobs_v2 accurately.
  • A proposal was made to add a flag to enable metric collection specifically for this purpose. This would involve making additional calls to get_blobs_v2, even when unnecessary, solely to gather data.

3. Incremental Implementation

  • There was consensus that get_blobs_v2 does not need to be a blocker for DevNet 6 and can be added incrementally:
    • The team compared this approach to how get_blobs_v1 was introduced in stages.
    • The focus for DevNet 6 should remain on validating other critical features like cell proofs in transaction bundles and basic blob handling.

4. Debate on Partial vs. Empty Responses

  • A significant debate arose regarding whether get_blobs_v2 should return partial responses if some blobs are available or an empty response if all blobs are not present:
    • One perspective argued for empty responses to maintain simplicity and avoid confusion.
    • Another perspective highlighted potential benefits of partial responses in scenarios like local block building, where some blobs might already be cached locally.
  • Ultimately, it was noted that local block building typically involves importing blocks directly with all associated data already present, reducing the relevance of partial responses in this context.

5. Mainnet and DevNet Considerations

  • For DevNet 6:
    • It was agreed that EL clients should implement get_blobs_v2, but CL clients could defer its usage depending on collected metrics.
  • For mainnet:
    • The team recognized that get_blobs_v2 might become more critical as network conditions evolve and distributed blob publishing gains traction.
    • Differences in how various EL clients handle blob propagation will likely influence its utility across implementations.

6. Broader Implications

  • The discussion highlighted broader challenges in PeerDAS development:
    • Effective coordination between CL and EL clients is crucial for features like blob handling.
    • Network topology, bandwidth constraints, and private vs. public mempools significantly impact blob availability and propagation rates.
  • The team acknowledged that further research and testing are needed to refine these mechanisms before full-scale adoption.

While get_blobs_v2 is an important feature for PeerDAS, its immediate inclusion in DevNet 6 is not critical. The focus remains on collecting accurate metrics to inform its implementation and ensuring other foundational features are ready for testing.

Mainnet Release Planning

  • Team working toward releasing RC.0, targeting Thursday for the cut
  • PRs currently merged into unstable branch
  • Awaiting Tuyen's total effective balance sorting PR
  • Waiting for Gnosis's configs, expected by the next day
  • Additional potential PRs for consideration:
    1. Reducing maximum retain epoch from three to two for the seen attester cache
    2. Updating weak subjectivity calculation from the spec
  • Team agreed these are not critical blockers but would be nice to include if ready by Thursday

Updates

Matthew

  • Spent significant time reviewing architecture and PeerDAS implementation
  • Got Tuyen's PR merged for global config
  • Researched specs for get_blobs_v2 and cell publishing
  • Reviewed Nazar's PRs (~6 of them) focused on state diffs, linting, and JSR package management
  • Made progress on refactoring code with approximately 1000 lines committed locally
  • Focused on massage implementation and updating code to match newer paradigms

Derek & Hugh (Coinbase)

  • Working on validator custody PR implementation
  • Planning to move on to distributed blob publishing and engine get_blobs_v2 once the custody PR is merged
  • Preparing for implementing cell proof support in transaction bundles

Katya

  • Completed docs PR that's ready for review
  • Finished validator monitor refactoring
  • Running a beacon node (Hoodie) but having issues with metrics visibility in Grafana
  • Will coordinate with Philip to resolve Grafana access issues

Tuyen

  • Focused on packing attestation deployment and performance testing
  • Deployed improvements over the weekend and gathered performance data
  • Started work on multi INV implementation, aiming to finish within a week
  • Encountered performance issues with BLST after SSZ 1.2 upgrade, particularly with hash computation
  • Despite 30% faster performance on mainnet, holding off on further optimization until migration to Bun

Nico

  • Helped merge PRs by Tuyen and Cayman, including IPNM (IP Number Management) implementation
  • Spent time reviewing attestation-related code
  • Tuyen's scoring PR looks promising
  • Planning to coordinate with Luca for another data collection on Hoodie to compare with other clients
  • Focusing on deeper understanding of Mev (Maximal Extractable Value) to prepare for EPBS implementation
  • Reached out to Flashbots regarding MEV discussions, though they have limited time availability

NC

  • Reported that Prysm and Lodestar fixed their consensus issue on FOCIL, now interoperating correctly
  • Multiclient network is working well with minor hiccups to resolve
  • Identified issue with Lodestar not publishing inclusion lists when slot time is set to 6 seconds
  • Shifting focus back to EPBS (Ethereum Protocol Blockchain Staking) discussions
  • Reviewing Tuyen's PRs and following attestation packing developments

Phil

  • Working with Chiemerie on setting up the next fork group to track Electra fork progress
  • This will help monitor for regressions when merging PurDAS branch
  • Reminded about quarterly reviews deadline
  • Provided information about Petra book feedback form for team contributions

Lodestar Standup 2025-11 (March 18, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7554

Planning/Discussions

1.28 Release Candidate Review

  • The team discussed the 1.28 RC currently running on beta.
  • Metrics look good overall, but comparison is difficult due to Holesky testnet.
  • Proposed approach: Deploy to canary validators on CIP to get mainnet metrics before wider release.
  • An RC1 is needed with the latest fix for pre-Electra attestations, which addresses a regression from the past two months.
  • Some optimizations on pending consolidation may be included in a future 1.29 release rather than 1.28.

Infrastructure Migration

  • Chiemerie is working on migrating genesis validators from ETH Panda Ops to Hoodi servers.
  • Goal is to migrate testnet infrastructure to Hoodi by end of the week.
  • This will provide more stable metrics for collection.

Binary Diffs Feature Branch

  • Discussion on how to approach reviewing and merging large feature branches like binary diffs.
  • Consensus that an Architecture Decision Record (ADR) or similar high-level documentation would be helpful before implementation.
  • Suggestion to focus on interfaces and overall architecture first, then implementation details.
  • Agreement that big PRs into unstable are acceptable for formal review, but process could be improved.

Tooling Updates

  • Nazar presented work on package manager and release management tool updates.
  • Team decided to table major tooling changes until after Electra release to avoid disruption.
  • Discussion on potential future adoption of Bun and compatibility with current tools.

PeerDAS Collaboration

  • A developer from Base has offered to contribute to PeerDAS implementation.
  • Team discussing how to best leverage this external help, possibly focusing on validator custody.

Updates

Matt

  • Reviewed PRs with Nazar
  • Discussed execution decoupling and range syncing ideas
  • Planning Ethereum Guild session on execution decoupling

Katya

  • Opened PRs needing review
  • Working on data client metrics
  • Planning calls with Tuyen and Nazar for onboarding

Cayman

  • Published JSLipP2Quick, which is performing well
  • Developed a Zig build tool, seeking feedback

Gajinder

  • Worked on PeerDAS spec changes for validator custody and API endpoints
  • Discussing static vs dynamic validator custody implementation

Tuyen

  • Fixed issue with pending deposits processing time
  • Syncing node from Genesis for BatchHash feature
  • Improvements to BLST-Z library build process

Nico

  • Created GitHub issues from notes
  • Merged native compute implementation
  • Addressed critical security report from a white hat

NC

  • Updated FOCIL implementation for easier testing with Electra
  • Working on lazy load pattern using generators for pending deposits and consolidation
  • Updated block and attestation rewards to support Electra

Nazar

  • Refactoring archive store
  • Preparing to open PR for binary diff feature

The team also discussed improving their process for reviewing large architectural changes and welcoming new team member Katya.

Lodestar Standup 2025-10 (March 11, 2025)

Agenda: N/A

Planning/Discussions

Release Planning

  • The team discussed the approach for the next release after the Holesky rescue efforts.
  • Initially considered a hotfix (1.27.2) but decided to aim for a full 1.28 release instead.
  • Plan to merge bug fixes and new features from the Holesky rescue branch into unstable.
  • Aim to have a 1.28 RC (Release Candidate) by the end of the week.
  • Considering a potential 1.29 release before mainnet for additional features.

Infrastructure and Testing

Performance Improvements

  • Tuyen mentioned improvements in state root computation time (now ~1 second on Holesky).
  • Plans to push optimizations that could improve performance by 5x.

Client Comparison and Achievements

  • Lodestar performed well during the Holesky rescue, showing stability and efficiency.
  • Resource usage is significantly lower compared to other clients (12GB RAM vs 60-100GB).
  • Lodestar was stable for two weeks and produced blocks when network participation was low.
  • Currently the only client that can take an unfinalized checkpoint.

Future Testing and Development

  • Suggestion to focus on checking performance metrics on a stable Holesky network.
  • Discussion about potentially using feed groups for normal testing.
  • Consideration of keeping a 300-validator key node for comparison with mainnet Lido validators.

Updates

Tuyen

  • Coordinated with Nico on Holesky Rescue.
  • Worked on BLST-Bun, completed spec tests and benchmarks.
  • Found performance issues on Linux (3x slower than BLST-TS), suspecting build optimization problems.
  • Requested Nico's help with the remaining work on unfinalized state feature.

Gajinder

  • Discussed uncoupled execution payload with Potuz.
  • Explored potential complexities with delayed availability of data blobs.
  • Worked on a simplified version of PeerDAS validator custody architecture.
  • Presented the simplified version in the PeerDAS breakout meeting.

Cayman

  • Updated js-libp2p-quic repo, an experimental napi-rs wrapper for QUIC library.
  • Helped with Holesky Rescue efforts.
  • Worked with Nico to spec out ERA file support for archive node functionality.
  • Continuing work on Zig build tooling CLI.

Katya

  • Working on reorganizing metrics.
  • Focusing on the state transition package.
  • Learning the codebase and asking questions.

Nazar

  • Has multiple PRs pending review, including one for erasable syntax.
  • Preparing to publish the first package on JSR, starting with the benchmark package.
  • Working on modernizing the publishing pipeline to support various ecosystems.

Nico

  • Led efforts in Holesky Rescue, fixing critical issues.
  • Unfortunate that we had the attestation bug, but we performed very well comparatively.
    • Resource usage was stable at around 12GB.
    • We were stable for two weeks and still produced blocks.
    • We were stable even throughout the 5% participation where the network almost collapsed.
    • We pointed 100,000 validator keys to one beacon node and still performed well.
  • Experimented with loading 100,000 validator keys in a single validator client.
    • Possible to load, the validator client had a 16GB heap memory usage
    • Goes into a death spiral though for initializing the slashing DB for all 100k validators
  • Accidentally caused some validators to be slashed during key movements due to moving keys around
  • Emphasized Lodestar's strong performance and reliability during the rescue efforts.

The team concluded the meeting by discussing the next steps for stabilizing the network and returning to normal development rhythms. They also mentioned upcoming opportunities for conference presentations and travel arrangements.

Lodestar Standup 2025-09 (March 4, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7524

Planning/Discussions

Non-Finalized Chain Handling

  • Discussed pruning previous states and rejecting blocks before a certain depth
  • Considered using an epoch state at a particular depth as the finalized state, similar to non-finalized checkpoint start
  • Tuyen's PR to delete after 100 epochs was mentioned as an example of current practice on Holesky fork. We won't be able to regen beyond 100 epochs ago
  • Concerns raised about range sync always starting from the last finalized state, potentially causing issues with long non-finalized chains

Sync Challenges

  • Lodestar experiencing issues with continuous batch generation from last finalized state during long periods of non-finalization
  • Discussion on why nodes might be re-querying batches and potentially getting banned by peers
  • Speculation on how Lodestar handles sync when connecting to new peers

Testing Infrastructure

  • Current Holesky prod nodes are functioning, but testnet infrastructure is unstable
  • Need for testing infrastructure for 1.28 release and future updates
  • Challenges with testing on Sepolia due to low validator count (around 1700)
  • Consideration of using CIP validators for beta testing

Release Planning for 1.28

  • Plan to have 1.28.0-RC by the end of the week
  • Discussion on which PRs to include, such as peer manager updates and blacklisted block check
  • Agreement to release an RC with confident changes and deploy to CIP validators for sanity checking

Updates

Matt

  • Continued work on a large code change, writing another thousand lines
  • Preparing to break down the changes into PR-able pieces
  • Reviewed Nazar's work and preparing for state diff transition

Nazar

  • Merged several PRs, working on renaming archive store
  • Preparing binary diff PR for review
  • Added support for JSR registry publishing in SSZ repo
  • Investigating Node erasable syntax for potential future TypeScript execution in Node.js

Tuyen

  • Fixed option and historical state file data store issues
  • Working on syncing from unfinalized checkpoints
  • Progress on BLST in Bun, aiming for first release next week

Katya

  • Started work on metrics, focusing on beacon metrics specs
  • Found some unused specs and planning to push initial PR after review

Nico

  • Discussed pessimistic sync implementation challenges
  • Planning to test Tuyen's PR for syncing from unfinalized states
  • Considering testing NC's pessimistic sync on Holesky

Gajinder

  • Proposed an EIP to uncouple execution payload from BeaconBlock
  • Discussed potential benefits of this approach compared to EPBS
  • Suggested a dedicated session to discuss the EIP in detail

The team also discussed plans for an upcoming Ethereum Guild meeting to debate various topics, including Gajinder's EIP proposal.

Lodestar Standup 2025-07 (February 18, 2025)

Agenda: N/A

Planning/Discussions

Pectra Hard Fork Preparation

  • The team is nearing readiness for the Pectra Hard Fork scheduled for next Monday.
    • Holesky Genesis validators and Sepolia groups are updated and ready. Awaiting mev-boost for updating
    • Execution Layer (EL) clients connected to testnet infrastructure are being upgraded.
    • Feature branches need to be rebased to v1.27 and redeployed to ensure proper forking. If not completed, version v1.27 will be deployed as a fallback at the end of the week
  • A potential consensus issue was reported by Gnosis, but its root cause remains unclear. It may relate to differences in presets or hardcoded constants between mainnet and Gnosis chain.

Pectra: Consensus Issue Investigation

  • A specific Lodestar node showed a state root mismatch during state transitions one Gnosis node only, while others were unaffected.
    • Suspected causes include caching issues or discrepancies in pre-state data. It was noted that states are different from the problem node and the healthy node. Slot 68,120 it seems like it has a different pre-state in the bad node.
    • Further investigation is required, including downloading API data and comparing states.

Pectra: Sync Aggregate Issues

  • https://github.com/ChainSafe/lodestar/issues/7299
  • Sync aggregate problems were observed on DevNet 6 and potentially Holsky/Mainnet.
    • Some blocks failed due to delayed parent block visibility of greater than 6 seconds or other unknown factors.
    • The team will continue monitoring and investigating these issues.

PeerDAS Updates

  • DevNet is being restarted due to issues with the Geth client. Fixes are pending release.
  • Optimistic syncing was identified as a cause for nodes appearing synced without validator attestations. This issue should resolve once EL fixes are applied.
  • Matt continues working on refactor here: https://github.com/ChainSafe/lodestar/pull/7474

SSZ and Future Lodestar Architecture

  • SSZ repository has been added to ChainSafe with admin access pending setup for Lodestar teams: https://github.com/ChainSafe/ssz-z
    • SSZ operations would benefit from being lightweight and fast in the native implementation.
    • The design allows for seamless data retrieval and caching between native and JavaScript layers.
  • A proposal for modernizing Lodestar using Zig was discussed:
    • Focus on native state transitions leveraging Zig's lightweight pointer management. A proposed mechanism involves storing pointers as numeric identifiers in JavaScript, which can be passed back to the native side for operations.
    • Potential benefits include faster data access and shared memory between native and JavaScript environments.
    • Concerns about memory space limitations (e.g., U32 vs. U64) were addressed with existing safeguards in place.
  • Native State Transition: This approach aims to improve performance and memory efficiency by bypassing JavaScript's limitations.
  • Pointer-Based Architecture:
    • A pointer in Zig is represented as an 8-byte value.
    • Lodestar's architecture would maintain a "BeaconState" pointer, which can be passed to native functions for state transitions, reducing overhead.
  • Shared Data Access: Both native and JavaScript environments can access shared data structures, minimizing duplication and improving performance.
  • While some features may depend on unstable Zig APIs, the design ensures compatibility with existing tools and safeguards against potential limitations. It will require careful monitoring and updates as the language evolves.

Next Steps:

  1. Begin prototyping the state transition module in Zig.
  2. Evaluate performance improvements through benchmarks.
  3. Address integration challenges with existing Lodestar components (e.g., SSZ serialization).
  4. Plan resource allocation, including potential team expansion, to support this roadmap.

Docker vs. Binary Deployments

  • Discussion on whether Docker and non-Docker deployments should be maintained across all testing groups:
    • Docker deployments generally perform better due to up-to-date system libraries (e.g., glibc).
    • Most users deploy via Docker, making it essential for testing.
    • Decision: Maintain both Docker and binary testing but simplify where possible.

Team Expansion

  • Plans to hire additional engineers to alleviate workload and support the Zig-native roadmap. Referrals are encouraged.

Updates

Cayman:

  • Developed a native version of sync committee computation using NAPI-RS, achieving a ~40x performance improvement (10–15ms computation time).
  • Worked on SSZ refactoring; seeking reviewers for progress discussion.

Matt

  • Progressed on a major refactor PR, addressing errors in sync logic and range sync. The draft PR is available for preliminary review.

Nazar

  • Four pull requests are under review; feedback will be incorporated this week despite health challenges.

Nico

  • Updated fork testing from Altair/Bellatrix to Electra payloads, resolving spec issues in the Beacon API and Builder API.
  • Released a major Beacon API update validated against Lodestar.
  • Continued debugging interop issues, particularly with Prism clients.

Tuyen

  • Investigated sync committee performance bottlenecks; native implementations show promising improvements.
  • Identified performance issues in PeerDAS deployments due to lack of unit tests; ongoing refactor expected to address this.
  • Explored callback integration for FFI but deemed it lower priority compared to other tasks.

NC:

  • Fixed block attestations issue caught on devnet-6
  • Read about ethspecify and investigate its potential benefit for Lodestar see #7477
  • Focil - caught a Prysm bug that its state.fork is not updated correctly on focil fork, which is one of the causes of state root mismatch I mentioned last week.
  • Lodestar also has fork digest issue which I had fixed
  • Regarding running a stress test of having large number of deposit requests ⁠Electra Discussions⁠ I have pinged pk910, hopefully we can get some result soon.

Lodestar Standup 2025-06 (February 11, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7436

Planning/Discussions

Pectra Testnet Release

  • The team is preparing for the Pectra testnet release, scheduled for Thursday.
  • Several PRs (e.g., Snappy and Async Aggregate) will be reverted due to unresolved issues.
  • The pruneHistory feature will not be reverted but hidden via a flag as it is not yet production-ready. Concerns were raised about its behavior during restarts, particularly with LevelDB compaction delays.

Electra and Mainnet Planning

  • The Pectra release will focus on testnets and include hard fork numbers for Sepolia and Holesky.
  • A mainnet release date will be decided after the Sepolia fork, with expectations for one or two more releases before the official Pectra mainnet release.
  • Discussions highlighted whether to integrate the PeerDAS (Fulu) code into the Pectra release. The consensus was to delay merging large changes until after mainnet to ensure stability and thorough testing.

Validator Custody and Super Nodes

  • Validator custody complexities were debated, with concerns about its implementation difficulty and enforcement. Suggestions included manually increasing custody data for nodes with more validators.
  • The role of super nodes in PeerDAS was discussed. While they can enhance network performance, their necessity for network stability was questioned. It was emphasized that availability should not hinge on super nodes alone.

Beam Chain and Fabric Initiatives

  • Preliminary discussions on Lodestar's involvement in the Beam chain highlighted ongoing work in cryptographic advancements (e.g., post-quantum signatures) and modular architecture to support future proof systems.
  • The Fabric initiative, focusing on L2 standardization (e.g., bridging, DA layers), was introduced as a potential direction for Lodestar's modular architecture.

Updates

Matt

  • Progressed on PeerDAS integration:
    • Spec tests, unit tests, and linting are nearly complete.
    • PeerDAS branch is synced on DevNet.
  • Proposed renaming fork-related types (e.g., Fork Blobs) in line with spec conventions (e.g., pre-Deneb/post-Deneb).

Nico

  • Reviewed remaining Electra tasks; identified an attestation/block production issue on DevNet.
  • Beacon API cleanup is ongoing; deprecated the deposit API following community consensus.

NC

  • Focused on Fossil development:
    • Resolved Engine API issues with Geth.
    • Ran local devnets with Lodestar-only setups, achieving finalized inclusion lists.
    • Encountered state root mismatches when testing with Prysm; debugging continues.

Tuyen

  • Worked on Proposal Boost reorg fixes; tests passed but sync tests need refinement.
  • Released BLST-Zig integration using Zig compiler for BLST bindings.
  • Addressed PeerDAS issues, including ensuring minimum match pieces per group.

Cayman

  • Finalized pruneHistory flag implementation but will hide it due to rough edges.
  • Reverted Snappy and Async Aggregate PRs due to regressions when combined; plans to test individually moving forward.
  • Continued SSZ-Zig refactoring; nearing completion despite challenges from ongoing updates.

Gajinder

  • Highlighted critical issues in delayed block proposals:
    • Suggested upfront transaction fee deductions to prevent invalid blocks from impacting coinbase payouts under deferred execution models.
    • Advocated delaying Fossil until delayed execution is fully tested.
  • Engaged in discussions on validator custody and super node roles within PeerDAS.

Helena

  • Announced an upcoming blog post on censorship resistance by Ben and Phil.
  • Encouraged team members to submit speaking proposals for Protocol Guild and ECC events.

Action Items

  1. Finalize Pectra testnet release by Thursday, ensuring reverted PRs are excluded or flagged appropriately.
  2. Investigate Electra attestation/block production issues further before testnet deployment.
  3. Continue debugging Prysm state root mismatches for Fossil compatibility testing.
  4. Prepare a unified message for Lodestar’s stance and contributions to the Beam chain initiative ahead of Friday’s call.

Lodestar Standup 2025-05 (February 4, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7422

Planning/Discussions

1.26 Release Update

  • RC1 was pushed yesterday with a change from Nico to increase block production timeouts
  • No issues reported so far on beta deployment and Nico's mainnet validators
  • Team will do a final review after the call before releasing

Infrastructure Restructuring

  • Discussion on optimizing testing infrastructure and server configurations
  • Focus on covering both Docker and binary setups due to performance differences
  • Proposal to have ARM64 servers in unstable, stable, and beta groups for sanity checking, not required in feature groups.
  • We should make sure that all servers of a similar type are actually using similar hardware.
  • Redunancies to the core use case/workload that we measure against is important
  • Consider that subscribe all subnets helps spot regressions more than it is to be performant while subscribing to all subnets
  • We should have holesky validators attached with subscribe all subnets, same hardware to do better apples to apples comparisons

Subscribe All Subnets Discussion

  • Debate on the usefulness and performance impact of subscribe all subnets feature
  • Consensus that it's not beneficial for validator effectiveness but useful for testing
    • We generally see performance of validators suffering under it and is not really beneficial at all.
    • Head votes are worse
    • Block imports are delayed
    • A worse peer overall
  • Might be worth putting in our documentation that subscribe all subnets is not recommended for normal use in any validator staking configuration
  • We've come to the conclusion that some changes improve validator performance but may regress metrics on subscribe all subnets. There is no correlation that shows better performance on subscribe all subnets means better validator effectiveness for normal users.
  • Decision to use it as a tool for spotting regressions and optimizations, but not to cater resources specifically to being performant with subscribe all subnets
  • Agreement to not advertise or advocate for subscribe all subnets as a primary feature

Monthly Pull Request Grooming

Binary Diff and State Archive

  • Expected to be ready for review soon
  • This is a breaking change but will be hidden under a flag for merging to unstable and disabled by default
  • It is not backwards compatible once upgraded
  • Once we test everything and the feature is matured, we can then think about writing a migration to completely deprecate and remove the old format which is more important for historical nodes rather than validators
  • It is also possible to just use a different bucket name to prevent it from being a breaking change
  • Consideration for making it default in Lodestar 2.0 after thorough testing where we can see the reduction of disk size usage
  • Note that computation will likely be higher for generating binary diffs

Pruning

  • Discussed having a pruning history flag to get rid of historical blocks and states (we currently store one state every 4096 slots by default) for better UX.
  • Some validator type users can have up to 500GB of worthless storage that can be cleaned up via startup using this flag to purge history
  • https://github.com/ChainSafe/lodestar/pull/7427

Code Management for Forks

  • Discussion on managing long-lived fork branches
  • Agreement to merge sooner into unstable branch to avoid conflicts
  • Emphasis on merging code that's already in consensus spec into unstable

Updates

NC

  • Working on FOCIL implementation
  • Encountering issues with Engine API and syncing
  • Planning to focus efforts on resolving these issues before the next FOCIL call

Nazar

  • Binary diff PR nearly ready for review
  • Will deploy to feature group for testing after review

Matt

  • Completed work on PeerDAS
  • Planning to look into MSM implementation
  • Working on refactoring PeerDAS code and implementing Twiyn's suggestions

General Team Updates

  • Discussion on deprecating deposit snapshot after Pectra goes live
  • Consideration of pruning old blocks for improved storage management
  • Ongoing work on various PRs including flattening fork logic and deprecating eth1 data poll

Lodestar Standup 2025-04 (January 28, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7395

Planning/Discussions

Electra and 1.26 Release

  • Version 1.26 is scheduled for RC release on Friday.
  • DevNet 6 is expected to happen this week, but the timeline is uncertain as teams are still fixing DevNet 5 bugs.
  • The team aims to include some hard fork numbers by Thursday's ACD call.
  • SSZ 1.0.2 release is facing CI issues, but the team plans to swap the token and try again.
  • Most remaining Electra tasks are optimizations and not urgent for the 1.26 release.
  • The team wants to include the Electra attester slashing handling in Friday's release.

Pectra Retrospective and Future Hard Forks

  • The team discussed the challenges faced during the Petra upgrade and how to improve for future hard forks like Fulu.
  • Key issues identified:
    1. Scope creep and difficulty in maintaining timelines
    2. Challenges in prioritizing features and EIPs
    3. The need for better communication and consensus-building mechanisms
  • Suggestions for improvement:
    1. Consider implementing a more structured approach to EIP inclusion and hard fork planning
    2. Potentially increase the frequency of hard forks (e.g., two per year) with smaller scopes
    3. Improve tooling and processes for identifying consensus and prioritizing features
    4. Balance between decentralized decision-making and the need for clear direction

Technical Considerations

  • The team discussed the difficulty of removing features from a hard fork once implementation has begun, as they are often intertwined in the codebase.
  • There was a discussion about the potential benefits of a more modular architecture that would allow for easier feature toggling, similar to what some other clients have implemented.
  • The importance of maintaining Ethereum's high reliability standards was emphasized, with the team agreeing that shortening hard fork cycles should not compromise security or stability.

Updates

  • The team is working on fixing CI issues for the SSZ 1.0.2 release.
  • Work continues on optimizing and finalizing Electra features for the upcoming 1.26 release.
  • The team is closely monitoring DevNet 5 and preparing for DevNet 6.
  • There was a discussion about recent developments in the L2 ecosystem, particularly regarding the Fabric initiative and its potential impact on Ethereum's development process.

The team will continue to work on the 1.26 release and prepare for upcoming discussions at the ACD call and town hall meeting.

Lodestar Standup 2025-03 (January 23, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7360

Planning/Discussions

Lodestar V2 Planning

  • The team discussed the roadmap for Lodestar V2, reviewing a pinned issue with the planned milestones.
  • There was a debate about whether to support old forks in V2:
    • Consensus was to keep old forks for now unless they become blockers, as removing them would require significant effort.
    • The team agreed to remove the sim merge test, as pre-merge networks are no longer relevant.

DevNet 5 Metrics Review

  • Some regressions were observed after merging DevNet 5 to Unstable:
    • Increased job wait times
    • Issues with mesh peers
    • Higher GC on the main thread
    • Increased attestation reprocessing (from 4 to 24 per slot on average)

EIP 4881 (Delayed State Root)

  • Gajinder reviewed the new EIP for delayed state root computation.
  • Concerns were raised about potential complications for CL validations:
    • Without the state root, it's harder to verify if a block was properly executed.
    • Suggested adding a commitment or hash of state diff changes to address this issue.

RC Release Schedule

  • The scheduled RC release this week may be pushed to next week, depending on the progress in fixing the regressions on unstable and to accommodate potential hard fork release epochs for public testnets.

Updates

Matthew:

  • Merged non-SIMD changes
  • Worked on PeerDAS integration with DevNet 5
  • Reviewing PeerDAS specs for discussion with Gajinder

Nazar:

  • Fixed memory leak in benchmark setup caused by Vitest runner
  • Working on migrating Lodestar repo benchmarks to the upgraded version

Cayman:

  • Worked on a PR for Tuyen's ZIG SSZ library (paused for now)
  • Analyzed metrics for libP2P 2.0 branch (feat-4), looking promising
  • Started familiarizing with sync code for potential refactoring or backfill sync work
  • Planning to revisit the Bun runtime effort for running Lodestar

Gajinder:

  • Assisted with Kurtosis configs debugging
  • Investigated NAPRS and TypeScript type interactions
  • Reviewed and commented on EIP for delayed state root (EIP 7732)

Nazar:

  • Fixed memory leak in benchmark utility related to Vitest runner
  • Working on migrating Lodestar repo benchmarks

Andrew:

  • Reaching out to node operators for feedback
  • Potential new node operator interested in testing Lodestar

Nico:

  • Discovered issues with some testing servers not running binaries as expected
  • Planning to rejig testnet infrastructure, consolidating smaller validator nodes
  • Participated in Twitter spaces event bridging builders with core devs and researchers

The team will continue to debug the Unstable branch issues asynchronously and reassess the RC release schedule based on progress.

Lodestar Standup 2025-02 (January 14, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7337

Planning/Discussions

Release Planning

  • Metrics review for 1.25 release showed no concerning issues. Release planned for later today.
  • Next RC due date discussed for 1.26:
    • Agreed to cut RC on January 24th, aiming for release on January 31st
    • Acknowledged shorter timeframe but aligns with goal of more frequent releases

Major PRs and Features

  • Plans include merging work to switch to Snappy Wasm for decompression once it is rebased and redeployed.
  • They also want to revisit the libp2p 2.0 and GossipSub updates, testing them separately from the β€œIDONTWANT” feature to address prior latency issues.
  • SIMD support for SSZ was merged and tested, aiming for inclusion in 1.26
  • Continued benchmarks for different JavaScript runtimes (e.g., Bun, Node) are underway to handle garbage collection impacts on benchmarking consistency

Devnet-5 Branch Cleanup and Integration to Unstable

The discussion around the Devnet-5 branch cleanup and integration plan back into unstable covered a few key points:

  • There is a need to clean up the DevNet-5 branch.
  • The branch contains changes that are not only related to Electra but also affect how forks are handled.
  • The DevNet-5 branch includes changes to the NEP (Network Upgrade Process), which requires caution during merging.
  • There are preset value updates in the minimal preset that affect CI and sim tests.
  • Current CI is failing due to the preset changes.
  • The team needs to wait for a Lighthouse image with updated presets to fix the CI issues.
  • As a temporary measure, they might consider running tests up to Deneb instead of Electra until the Lighthouse image is available.

Action Items

  • Review and fix any issues in the DevNet-5 branch before merging.
  • Wait for the Lighthouse image update to resolve CI problems.
  • Plan the integration carefully to avoid disrupting the main development flow.

The team emphasized the importance of not rushing the integration process and ensuring thorough testing before merging DevNet-5 changes into the unstable branch.

Electra Cleanup

  • Team to review and tag remaining Electra-related todos
  • Plan to integrate DevNet-5 changes into Unstable after 1.26 RC

Updates

Gajinder

  • Reviewed latest PeerDAS spec changes
  • Investigated TypeScript and Rust interop for potential light client snarkification

Nazar

  • Working on moving runtime setup to SSZ repo
  • Debugging benchmark issues related to garbage collection

Nico

  • Completed gas limit check PR
  • Finished remaining DevNet-5 tasks
  • Started looking into FOSSIL implementation

Matt

  • Finalized remaining work on the SSZ SIMD PR.
  • Focused on PeerDAS groups PR and a thorough review of the overall PeerDAS code.
  • Rebasing changes on Electra and merging updates for Fulu, aiming to test them in DevNet soon.

Cayman

  • Made PRs to discv5 for library improvements and removing costly crypto dependencies.
  • Reviewed the SSZ ZIG implementation and has a PR coming for enhancements.
  • Continues to review PRs such as Peer Discovery for subnet requests and related code[2].

Tuyen

  • Continued work on BLST binding for Zig and Bun testing signature, secret key, and public key classes.
  • Working on more advanced features like verifying multiple aggregate signatures.
  • Raised small PRs for PeerDAS improvements, including minimum subnet checks and batch download fixes.

NC

  • Focused on FOCIL implementation, estimated to be halfway complete.
  • Contributing to the Beacon API aspects for FOCIL and coordinating on design discussions in the open spec documents.

Phil

  • Upcoming call with Alex Stokes and Peter for quarterly feedback
  • Coordinating team attendance for upcoming interops

Andrew

  • Followed up on RPC endpoints discussion with infrastructure team
  • Reconnecting with operators for feedback, including questions on new release stability and validator effectiveness data

Peter

  • Highlighted an upcoming blog post draft for a new β€œFabric” standard for rollups, aiming to unify L2 solutions without introducing new networks or tokens
  • Mentioned synergy with potential L2 API expansions in Lodestar and continuing research on universal rollup definitions

Lodestar Standup 2025-01 (January 7, 2025)

Agenda: https://github.com/ChainSafe/lodestar/discussions/7326

Planning/Discussions

Strict Cadence for Releases

  • Implement stricter cadence for releases to prevent performance regressions from large accumulations of unreleased commits.
  • Proposed cadence: Close door to PRs on specific day (Friday), cut RC, deploy to beta group, review metrics and release after standup the following week.
  • Goal is to avoid situations like the recent accumulation of 97 PRs.

Lodestar 1.25 Planning and PR Grooming

FOCIL and PeerDAS

  • FOCIL project gaining momentum; need to assign lead and break down spec for implementation.
    • NC and Nico will take charge on this implementation
  • Spec itself is small. Changes to beacon node, validator, P2P and fork choice. Due for next breakout call January 15.
  • PeerDAS is high priority; need to get it working to avoid max blob count issues. Continue implementation and testing.
    • Matt and Gajinder will take charge on this implementation. Tuyen in support role.
    • Kurtosis configs and interop testing scheduled for two weeks from now.

Pectra Devnet-5

  • Local testing via Kurtosis is ongoing, but no public spin-up yet.
  • Builder flow will start being tested.
  • Focus on fixing bugs and ensuring stability before public release.

Updates

BlockOps Role

  • Closing role, shifting BlockOps role back towards software engineering focus.

Zig Implementation of DiscV5

  • Cayman:
    • Completed ENR encoding, decoding, signing, and verifying implementation in Zig.
    • Will share work and findings with team soon.

Zig BLS Library

  • Tuyen:
    • Making progress on getting Zig BLS library operational.
    • Expects to have good news to share next week.
Clone this wiki locally