Streamline complementary REST requests

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/93

Originally created by @aaronsteers on 2021-04-08 23:56:48

---

There are at least two use cases for developers to make additional REST API calls beyond what is handled already by the SDK:

1. During discovery - The tap may want to ping a REST API ahead of calls to get_records().
2. During data-append - The tap may need to supliment the core data with additional calls in the post_process() handler.

For both of these cases, we should establish a pattern that developers can implement with confidence.

As of today, the developer would likely implement calls against the requests library directly. The biggest downside of this approach is that those extra calls may have inconsistencies in regards to authentication and retry logic.

[MeltyBot]

View 5 previous comments from the original issue on GitLab

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Could be addressed by #1649

[ReubenFrankel]

Curious if there is any more progress on this?

[edgarrmondragon]

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

[ReubenFrankel]

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

Definitely a nice-to-have rather than a pressing issue, but same as my comment on the GitLab issue - tap-auth0 has some clunky logic to request a user export job, then poll the job until it's completed in order to fetch the resulting gzip, decompress and parse out records.

https://github.com/Matatika/tap-auth0/blob/75d5c92d57afe3443f793a3f5a671cb6da826af8/tap_auth0/streams.py#L22-L117

[edgarrmondragon]

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

Definitely a nice-to-have rather than a pressing issue, but same as my comment on the GitLab issue - tap-auth0 has some clunky logic to request a user export job, then poll the job until it's completed in order to fetch the resulting gzip, decompress and parse out records.

https://github.com/Matatika/tap-auth0/blob/75d5c92d57afe3443f793a3f5a671cb6da826af8/tap_auth0/streams.py#L22-L117

I'll revisit #1649 and see if I can a) make sure the API in that PR would truly solve this and b) get it to a merge-able state

[ReubenFrankel]

Appreciate it - no rush though. 😄

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[ReubenFrankel]

Should stay open IMO

[edgarrmondragon]

Agreed