Handling STAC extension version upgrades

[duckontheweb]

Moving here from stac-spec Discussions thread.

Now that STAC extension versions are no longer tied to the core STAC version or to each other, we may run into situations where a user is working with catalogs that implement some combination of a core spec version and extension versions that are not all supported by the same version of PySTAC.

Here are a few ideas on how to handle this, but other suggestions are welcome. It would also be good to get feedback from maintainers of downstream libraries on what would work best.

1. Separate Extension Repos

STAC extension implementations would each have their own repo that would follow its own versioning (e.g. pystac-extension-label, pystac-extension-file, etc.). This would probably also require creating a pystac-extensions-base package that would be a dependency of each extension implementation.

Pros

• More accurately reflects the structure of the spec with extensions separate from the core spec
• Developers can mix and match any combination of extension versions they need
• No need to upgrade PySTAC version with each extension version update
• Simpler PySTAC core code base

Cons

• Means a lot more repos in stac-utils org
• Could lead to dependency conflicts if different extension implementations are using different versions of pystac-extensions-base
• Developers need to explicitly install each extension they want to use (or maybe we have an all-extensions extra in the pystac install that grabs all of them...)
• Sounds like a lot of work and may delay a 1.0 release

2. Version-specific Extension Classes

Instead of a single set of extension classes (e.g. EOExtension, ItemEOExtension, AssetEOExtension), we create a set of classes for each extension version (e.g. EOExtension_1_0_0, etc.). We still have a top-level extension class (e.g. EOExtension) with an ext method that would handle parsing the extension schema URI and delegating to the right version-specific class.

Pros

• Fewer repos to maintain
• Developers will continue to get extension support within PySTAC without needing to install other extension-specific dependencies
• May be able to limit the outward-facing changes if the behavior of ext stays mostly the same as it is now
• Might be able to implement this incrementally (i.e. only for extensions that have actually undergone an upgrade)?

Cons

• More complex extension code base within PySTAC (may need to create version-specific modules within each extension to keep the files from getting too bloated).
• Typing may be difficult and/or confusing (e.g. calling EOExtension.ext and getting an instance of EOExtension_1_0_0 back might be non-intuitive).

3. Only Support Latest Version (Status Quo)

We continue to update the existing extension code to support each new extension version as it is released and do not provide support for multiple extension versions within the same version of PySTAC.

Pros

• Lowest development effort
• Simpler extension code than other 2 options

Cons

• Does not solve the issue raised above. It would be on users to find a workaround.

cc @lossyrob @m-mohr @gadomski @jbants @matthewhanson

[gadomski]

Some questions:

• Do any of the currently-supported extensions add dependencies to PySTAC (i.e. if we split up to separate repos, would install_requires get smaller)?
• Right now, are extensions implemented with a plugin and/or namespace structure such that third party extensions can hook into the PySTAC API? Not saying they need to be and I'm not sure if that'd be particularly useful, but just curious.

Speaking from a stactools perspective, the best case scenario would be a PySTAC that came with "batteries included"; it would work with all core extensions without needing to depend on additional repositories. The version thing is a little trickier -- I think it's non-ideal if downstream code has to add logic to deal with different extension versions, that feels like a PySTAC thing to me. I think Option 3 (Only support latest version) is my initial instinct, with maybe some additional super-powers to "bring forward" older versions to the latest version when it makes sense.

[lossyrob]

Answers:

• No extensions add dependencies; PySTAC sticks to metadata so there shouldn't be reason to add dependencies for extensions and we've been aggressive in keeping the dependency footprint as small as possible
• Extensions are implemented as a plugin architecture. They were more so prior to the 1.0 refactor, but now there's a light(er) weight hooks that an extension can register (example hooks and registering) that extensions written inside or outside of the main project can use to hook into PySTAC inner workings in order to accomplish extension logic (e.g. the label extension adds a "source" link which should be treated as a linked STACObject when PySTAC is resolving a catalog).

with maybe some additional super-powers to "bring forward" older versions to the latest version when it makes sense.

PySTAC already migrates things to the latest supported version, including extensions. That's part of the hooks for an extension - it provides the extension the ability to upgrade an object to the latest version.

I agree Option 3 is the way to go; the additional maintenance burden and code spread of Option 1 I think would outweigh the burden to upgrade to new versions of extensions as they are released. PySTAC has gone with an approach of "use the latest version to read any version, but write the latest version. If you need to write older versions, use older versions of PySTAC". I think that also applies here. Option 2 might be a better solve, but would be a lot more complex and potentially an even greater maintenance burden, and would confuse the "always upgrade to latest" approach PySTAC takes now - if we do this for extensions, why not for core objects? Maybe Option 2 is a direction we should explore if there's enough use cases where upgrading to the latest (or needing to upgrade PySTAC to a newly released version of an extension) is a big issue for users.

[duckontheweb]

That all makes sense to me. If we're comfortable with Option 3, at least for now, I won't include this in the RC1 milestone. I mostly wanted to make sure we decided on any breaking changes prior to that release.

I've been a little uncomfortable with the fact that we automatically migrate objects when deserializing them (although I also see the benefit of it). I think that deserves a discussion of its own, however, so I'll open a separate issue for that.

[emmanuelmathot]

@duckontheweb to answer your question in previous discussion: In DotNetStac, I implemented Option 3 with a versioning mechanism reading previous versions (if implemented) with upgrading functions to latest version.

[m-mohr]

FYI: I basically do the same in STAC Browser, all data is automatically converted to the latest version (although STAC Browser only does read and no write, of course). This is done through stac-migrate, but I have not implemented e.g. updating file extension 1.0 to 2.0 (+ raster 1.0) for example.

[vincentsarago]

👋 That's an interesting discussion.

From a user POV, right now I'm a bit worried. I wanted to use pystac+extensions but I'm afraid about having to keep an eye on either pystac version and version of the extension within each extension submodule.

I was a bit surprised to find all extensions from stat-extensions available in pystac, and while this is nice it might get messy in the future. (IMO, I would have deferred the management of each extension to each repo but some maintainers might not use python at all).

I'm 👍 on Option 3, because it seems the simplest (😬 I'm not a contributor so you can ignore me).

[duckontheweb]

One possible hiccup to option 3 is the case where we cannot fully migrate fields from a previous extension version into the most recent version. The File Info Extension may be an example of this (see discussion in #472). In that case, some of the v1.0.0 fields were dropped in v2.0.0 and moved into the Raster Extension. However, there was no guarantee that those fields were being applied to a raster file originally so it may not be appropriate to create a Raster Band for them.

This does not necessarily break anything because they can still access the fields directly via extra_fields, but it might lead to some confusion.

[gadomski]

@duckontheweb resurfacing this issue as something that should probably be addressed as part of a v2.0 release. As the STAC ecosystem matures the domain of extension versions will only grow, and if we want people to use PySTAC's extension API (instead of going directly to properties), we should be able to gracefully handle older/newer versions without throwing. This problem is cropping up now in the wild w.r.t. the raster extension (initially brought to my attention by @Kirill888).

[Kirill888]

I think that exact version match approach will have significant negative impact. Effectively this makes it really hard to evolve extensions. One would need to upgrade an entire collection every time a new version of an extension your collection uses changes. Even if that change is a pure addition of new optional fields. Even if I could downgrade/upgrade installed extension versions independently per extension, it would not solve the problem of using multiple data sources within the same environment, not all collections will be in sync and sometimes you want data from many data sources.

If my currently installed software supports extension A-1.0 and I encounter data that has A-1.1, I would expect to be able to read A-1.0 fields out of the extension. And similarly A-1.1 code should be able to parse A-1.0 data.

Newer code/older data case is certainly possible, even if you have no constraints at all on format evolution, in the worst case you have older implementations included as renamed copies in the newer code that newer code can dispatch to, and possibly include on-the-fly translations (extra code to support migration will be needed).

Older code/newer data case is harder/more dangerous. For that to work one would need to agree up front on what changes are allowed as you go from A-1.0 to A-1.1. IF one promises to not change existing fields between minor version upgrades and only allow additions of new fields, then older code can safely parse newer data and you are not punished for using latest and greatest version. This would enable easier evolution of extensions as you are not forcing software or data upgrades while still allowing to capture more metadata as new use-cases are discovered. It does require some extra planning and more careful format evolution, but I don't think it will cost a lot in terms of complexity of the implementation. If you want to allow more complex format evolution than just field addition, while still maintaining forward compatibility, then you might choose to add some more complexity in the extension code, like for example being careful at interpreting some enumeration types with an idea that you might encounter newer values in newer versions and handling those gracefully. But that's a choice that can be made per-extension.

At the very least it should be possible to force extension code to interpret whatever data is present even if it's not quite the right version without having to patch stac_extensions property in code or on disk.