Search for large data sources for performance profiling

[aaronsteers]

Wanted to open this discussion to locate large data sources for performance benchmarking.

Related to:

• `BATCH` message type preview feedback #963
• Tap and Target performance benchmarks #906

A few known options to kick off the discussion:

1. Sample Datasets from Snowflake and BigQuery.
   • Both have a large library of datasets available to logged in users of these platforms. We could locate a few large data sources in each, and write performance test scripts that assume connectivity to one or the other.
2. Gov open data.
   • We could use something like tap-socrata to access one ore more large datasets in the public domain.
3. Fabricated data.
   • We could alter tap-smoke-test to fabricate large randomized datasets on-demand.

[tayloramurphy]

Something in 20 GB range, show extraction on record by record, SDK-based batching, native batching on both. Seems like a good resource https://www.reddit.com/r/bigquery/wiki/datasets

[aaronsteers]

Yeah, what's nice about BigQuery is that it's super easy to get an account. Configuring the tap is a little tricky but aside from that, it is very streamlined: it integrates with people's individual (or professional) Google accounts and it gives access to some very large (and interesting!) raw data.

[BuzzCutNorman]

I am still partial to the Stack Overflow dataset. I created a tap for the xml files that Stack Overflow makes available on Archieve.org. https://github.com/BuzzCutNorman/tap-stackoverflow-sampledata.

[aaronsteers]

@BuzzCutNorman - Wow - I just looked at the tap you created and it looks great. 🤩

This creates a really nice repeatable process for anyone in the community who wants to do their own benchmarks:

1. Download the datasets using the links in your readme.
2. Install the tap and configure it with the path to the downloaded files.
3. Run the tap to a target directly, cat the output to a local buffer file, or load it into a database that you want to test as a tap.
4. Test, observe, tweak, repeat! 🎉

cc @edgarrmondragon, @kgpayne fyi

[BuzzCutNorman]

@aaronsteers Thanks, tried to keep it minimal. You can also just use one file, or two or more files to check a stream transition. The tap will detect which files are in the folder as long as you don't rename them. Please let me know if you run into any issues using it.

cc: @edgarrmondragon , @kgpayne

[BuzzCutNorman]

Just updated https://github.com/BuzzCutNorman/tap-stackoverflow-sampledata to SDK 0.12.0 and added batch config documentation.

[edgarrmondragon]

Would getting good metrics logging in the SDK help with this?

I imagine it would at least help to compare record-by-record vs batch by looking at a record count timeseries in e.g. Prometheus. For example, backpressure would become apparent.

[aaronsteers]

@edgarrmondragon - We discussed already offline, but wanted to loop back here for the thread.

I do think that the work to let metrics logs to a specific logger (and log file) would be a big step in the right direction. This replaces the grep approach to trying to find metrics logs within the context of all other logs - and for very long streams, I think that's really helpful.

Users would presumably still need to aggregate the logs, but at least then they are all in one place and ready to be consumed.

[edgarrmondragon]

@aaronsteers I started a more general discussion on metrics: #1016