Should I use a sensor or an observable source asset?

[sryza]

Sensors and observable source assets are two different approaches to listening for external changes and triggering downstream computation based on them. What's the difference, and when is each one appropriate?

[sryza]

Both sensors and observable source assets listen for external changes.

The main high-level conceptual different between sensors and observable source assets is that observable source assets decouple the "discover what changed" logic from the "decide what downstream assets should be materialized" logic.

• A sensor is responsible for deciding what gets kicked off when it notices an external change.
• An observable source asset just records (observes) the change. The auto-materialize policies on downstream assets determine whether they'll be rematerialized in response to changes.

A few other differences:

• Observable source assets are currently marked experimental, while sensors are marked stable.
• When turned on, sensors are automatically evaluated every 30 seconds. To automatically evaluate an observable source asset, you use the argument auto_observe_interval_minutes=1. Source asset observations occur inside a run, whereas sensor ticks are evaluated outside a run.

[zyd14]

@sryza Are observable source assets still a recommended approach? I thought I had seen some references in communications to SourceAssets being deprecated soon in favor of AssetSpecs and External Assets.

[sryza]

Great question and yes. We are planning to rename them to avoid the deprecated name "source asset", but otherwise they should work the same. I had been hoping to push that out as part of 1.8, but there are some lingering questions about what the API should look like for partitioned observation results that we're still trying to resolve before finalizing the change: #23405.

[juliushuck]

auto_observe_interval_minutes=1 is deprecated, right? https://docs.dagster.io/api/dagster/assets#dagster.SourceAsset

I think we should use a schedule now for that kind of observability: https://docs.dagster.io/guides/build/assets/metadata-and-tags/asset-observations#observable-source-assets

[bitadmiral]

I'm wondering why one can't alternatively use a job to observe source assets? The doc-string of dagster.define_asset_job() states:

Creates a definition of a job which will either materialize a selection of assets or observe a selection of source assets. This will only be resolved to a JobDefinition once placed in a project.

I tried creating a job selecting Observable Assets exclusively using the upstream_source_assets method as below, but when running the job in the UI, it refuses with error: 'One or more observable assets are selected and cannot be materialized.'

dg.define_asset_job(
    name=f"observable_sources_group_{asset_group_name}",
    selection=dg.AssetSelection.groups(asset_group_name).upstream_source_assets(),
    description="A default job for processing assets",
)

Using the UI, I verified only observable source assets were selected.

[bitadmiral]

I'm wondering why one can't alternatively use a job to observe source assets? ...

It turns out this is just a UI bug for which there are easy workarounds (e.g. using the CLI instead of the UI). Bug #32060 has been filed.