Progress Bar Accuracy for Airbyte Connector Syncs

[carlesoctav]

I have a question regarding the "progress bar/sync bar" for connectors. On the UI, there's a sync_progress indicator that outputs something like this:

{
    "connectionId": "f6f160c1-ca98-42c4-a4c0-84b5a92666b1",
    "jobId": 112,
    "streams": [
        {
            "streamName": "user",
            "streamNamespace": "assistx_suite",
            "configType": "sync"
        }
    ],
    "recordsEmitted": 0,
    "recordsCommitted": 0,
    "bytesEmitted": 0,
    "bytesCommitted": 0,
    "syncStartedAt": 1753414774,
    "configType": "sync"
}

One issue I've encountered is that during a full_refresh (I haven't tested all sync modes), the recordsCommitted and bytesCommitted values are always 0. This is because, on the destination side, we don't send back an AirbyteStateMessage with the record count for the destination (committed records).

As far as I understand, according to the Airbyte protocol, we can only send back an AirbyteStateMessage if the input was also an AirbyteStateMessage.

However, during a full_refresh, I noticed that only one AirbyteStateMessage is sent at the end of the sync. Because of this, we can only capture a value of 0 or 1, and not a continuous progression. This makes it difficult to display a meaningful progress bar.

Does anyone have ideas on how to make the progress bar more accurate or continuous?

---

sequenceDiagram
    participant S as Airbyte Source
    participant D as Airbyte Destination  
    participant P as Airbyte Platform
    participant A as AssistXSuite

    Note over S,A: Typical Sync Flow
    S->>D: AirbyteRecordMessage (1000x)
    S->>D: AirbyteRecordMessage (1000x)
    S->>D: AirbyteRecordMessage (1000x)
    Note over S,D: No state messages during processing
    S->>D: AirbyteStateMessage (END)
    D->>P: AirbyteStateMessage + destinationStats{recordCount: 3000}
    P->>A: Progress Update: 100% (3000 records)
    
    Note over S,A: What We Want But Cannot Have
    S->>D: AirbyteRecordMessage (100x)
    Note over D: ❌ Cannot emit state without source state
    S->>D: AirbyteRecordMessage (100x)
    Note over D: ❌ Cannot emit state without source state
    S->>D: AirbyteStateMessage (END)
    D->>P: AirbyteStateMessage + destinationStats{recordCount: 200}

Loading