feat: Incremental Catalog

[andyoneal]

Feature scope

Taps (catalog, state, tests, etc.)

Description

For a salesforce tap, discovery is expensive. It can take several minutes to generate a catalog, which you might want to do on every sync in order to catch new fields or objects/streams being added. In many salesforce orgs, schema changes like this can happen daily.

Salesforce's "describe" api, which dumps the full schema as json, can return with only the schema changes made after a certain date with If-Modified-Since, but it's not obvious to me how this should be implemented with the sdk. Mainly, where is the "date last ran describe" state stored? Should the date be stored in the config file and updated on each run? Also, are there potential issues with a catalog file being supplied, but also edited and saved before sync since a tap is typically either writing a catalog file or reading it, not both?

Are there any known examples of this sort of thing being implemented with the sdk? If not, could you point me in the right direction for this might be done or what might need to change in the sdk to enable this? Happy to submit a PR.

[edgarrmondragon]

Hi @andyoneal 👋

That's an interesting idea, thanks for creating an issue.

Using the state for this would require the tap to be invoked with --discover --state=.... This is not a common, established pattern that the SDK supports, and it's also not currently supported by Meltano (though it could be updated¹ to use it).

The changes in the SDK would have to go through:

• Passing the contents of --state to the discover callback:

  sdk/singer_sdk/tap_base.py

  Lines 550 to 558 in 72f6f60

  	config_args = ctx.params.get("config", ())
  	config_files, parse_env_config = cls.config_from_cli_args(*config_args)
  	try:
  	tap = cls(
  	config=config_files, # type: ignore[arg-type]
  	parse_env_config=parse_env_config,
  	validate_config=cls.dynamic_catalog,
  	setup_mapper=False,
  	)

  similar to

  sdk/singer_sdk/tap_base.py

  Lines 524 to 530 in 72f6f60

  	tap = cls(
  	config=config_files, # type: ignore[arg-type]
  	state=state,
  	catalog=catalog,
  	parse_env_config=parse_env_config,
  	validate_config=True,
  	)

• Use Stream.tap_state to write directly to the tap state under a special key (e.g. discovery_bookmarks). Maybe the SDK should also be updated to make the pattern natively supported.

These are high-level ideas, but I think it's the direction in which things need to move to support this feature. What do you think?

Footnotes

1. https://github.com/meltano/meltano/blob/d12372af4bcf107667735edc48d28851ceb83413/src/meltano/core/plugin/singer/tap.py#L469-L474 ↩