New docs site is live!

[neverett]

Hi Dagster community!

Our new docs site is now live! Please check out https://docs.dagster.io/ and let us know what you think, either here or in the #docs-feedback channel in the community Slack.

We put in a lot of work to make our docs easier to use and contribute to—specific improvements include:

• A more user-friendly content structure
• An upgraded AI chatbot
• Migration to the Docusaurus platform, giving us lots of great features out of the box
• An improved build process, with:
  • API documentation that is created during the build process, avoiding ugly merge conflicts
  • Automatic broken link checkers
  • Fully tested code examples

We're also planning future docs improvement projects, so the docs site should only get better with age. Thanks to all who contributed your thoughts on the preview version of the site—your feedback makes our docs better for everyone.

[nigelainscoe]

In the tutorial https://docs.dagster.io/etl-pipeline-tutorial/, attempting to execute the dagster project from-example --example getting_started_etl_tutorial command raise the error Example "getting_started_etl_tutorial" not available from the dagster project CLI.

Checking the available projects using dagster project list-examples confirms this as true.

[cmpadden]

Hi @nigelainscoe , I am seeing the project listed when running list-examples.

$ dagster project list-examples
Examples available in `dagster project from-example`:
* assets_dbt_python
* assets_dynamic_partitions
* assets_modern_data_stack
* assets_pandas_pyspark
* assets_pandas_type_metadata
* assets_smoke_test
* deploy_docker
* deploy_ecs
* deploy_k8s
* development_to_production
* feature_graph_backed_assets
* getting_started_etl_tutorial
* project_analytics
* project_ask_ai_dagster
* project_atproto_dashboard
* project_dagster_modal_pipes
* project_dagster_university_start
* project_du_dbt_starter
* project_fully_featured
* quickstart_aws
* quickstart_etl
* quickstart_gcp
* quickstart_snowflake
* tutorial
* tutorial_notebook_assets
* with_great_expectations
* with_openai
* with_pyspark
* with_pyspark_emr
* with_wandb
* airlift-federation-tutorial
* airlift-migration-tutorial

And the scaffolding is working as expected.

$ dagster project from-example --example getting_started_etl_tutorial
Downloading example 'getting_started_etl_tutorial'. This may take a while.
Success! Created getting_started_etl_tutorial at /private/tmp/getting_started_etl_tutorial/.

This is on the latest version of Dagster:

$ dagster --version
dagster, version 1.9.11

I didn't think the version mattered, but could you please confirm which version you're running?

[nigelainscoe]

dagster, version 1.6.6

This is what got installed in the venv when following the instruction at https://docs.dagster.io/etl-pipeline-tutorial/

Or maybe I had an old version installed. I'll update it and see what happens.

Update: It looks like Python 3.13 doesn't support it. I downgraded to 3.11 and now everything in the world is lovely.

[cmpadden]

Perfect, thanks for the update.

[joejk2]

FWIW

I hit the same error on Python 3.11.11:

Example "getting_started_etl_tutorial" not available from the `dagster project` CLI.
Please specify the name of an official Dagster example. You can find the available examples via `dagster project list-examples`.

But everything works on Python 3.11.1:

Downloading example 'getting_started_etl_tutorial'. This may take a while.

[neverett]

@joejk2 I was able to reproduce this in a Python 3.11.11 venv, and was able to fix it by deactivating and reactivating my venv. Can you try that and let me know if it works for you as well?

[raiAmagi]

Cannot search documents

For example, if I enter “HourlyPartitionsDefinition” in the search box to search for a document on “HourlyPartitionsDefinition”, I get no results.
Specifically
For old documents:.
　Search suggestions are displayed, but even if I select a suggestion, I get a 404 error screen.
In the case of the new document: the
　No search suggestions appear, and the following error message is displayed.
　　No results for “HourlyPartitionsDefinition”.

I have done this on Chrome, firefox, and edge, and no search results appear in all of them.

Is there some kind of search technique?

We have confirmed that both new and old “HourlyPartitionsDefinition” documents exist in the API reference.

[C00ldudeNoonan]

Thanks for the callout. We have been working on adjusting our search crawler to have better coverage on the API pages and the code snippets to get you to the right spot faster. I will reply to your comment once the fix is in.

[raiAmagi]

Thank you for your reply.
I'm waiting for it to be fixed.

[C00ldudeNoonan]

I just pushed some changes so code snippets and API classes are being included. There is still more tuning to do on my end but I appreciate your feedback and I will be working on docs search for the next few days to get it working more intuitively

[raiAmagi]

Thanks for the modifications.
I was able to search.

[avneeshs1]

The new docs look very nice. One thing that would be nice to have is the [source] link in the API Reference section like the old docs use to have. I use that all the time since the docstrings at times are not very clear.

[cmpadden]

Good suggestion, I'll add that to the backlog of tasks to re-implement that feature. Thanks @avneeshs1 !

[petrusek]

In the new docs, is it possible to see for which version of Dagster the docs are?
If it is always latest (so no versioning), would it be possible to add a hint since which Dagster version it is valid (docs for the feature)?

I'm trying https://docs.dagster.io/guides/automate/declarative-automation/customizing-automation-conditions/example-customizations#updating-older-time-partitions-with-automationconditionon_cron but it gives me errors, that .replace doesn't exist... after that I realized my Dagster is v1.9.0 ... after studying the release notes I found that it is available since 1.9.10 (https://github.com/dagster-io/dagster/releases/tag/1.9.10)

[cmpadden]

Apologies for that @petrusek -

We plan to re-introduce versioning during the next release, you can see the PR for that here:

#27782

This will add a drop down to the top navigation and allow you to select version-specific documentation.

[petrusek]

Awesome, thank you!

[raiAmagi]

We are aware that the version currently indicated in the top right-hand corner of the documentation is the version of the documentation.
We believe that the old documentation was marked according to the core version.
It would be useful if the new documentation is also marked as the core version.

[aleewen]

First of all, thank you for putting in some effort to improve the documentation. This has been and continues to be the one downside to Dagster and the primary barrier to entry.

Much still needs to be improved. There continues to be holes and many areas lacking detailed explanation.

The docker-compose.yaml example is missing DAGSTER_POSTGRES_HOST alongside the user, password, and db variables. If this is not included, deployment will fail.

Documentation on concurrency is still not detailed and explicit, and is unclear all around.

[Aebel-Shajan]

This is minor, but the next button on https://docs.dagster.io/etl-pipeline-tutorial/create-and-materialize-assets should link to https://docs.dagster.io/etl-pipeline-tutorial/create-and-materialize-a-downstream-asset but instead links back to the start of the tutorial.

[cmpadden]

Thanks @Aebel-Shajan - @neverett and I have a ticket to fix this. We appreciate the feedback!

[raiAmagi]

This is about the “Python APIs” described in the “Executing a job” section of the “Job execution” document.

On the Python API description screen, the item for writing execution code is
"Loading..."
on the Python API description screen.

Is this a description error?

https://docs.dagster.io/guides/build/jobs/job-execution#python-apis