Tdigestdistribution #37109
Tdigestdistribution #37109
Conversation
Demo script that generates data from different distributions (normal, bimodal, longtail), runs a Beam pipeline, and displays quantile comparisons using the TDigest implementation. Uses BundleBasedDirectRunner to avoid portable runner which loses tdigest during protobuf serialization. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Added demo script section noting tdigest_demo.py - Documented portable runner limitation where tdigest is lost - Added workaround using BundleBasedDirectRunner 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Changed from CDF (percentile markers) to PDF (density estimate) - Rotated visualization 90 degrees (now vertical bar chart) - Uses TDigest.cdf() to compute density in each bin - Shows characteristic shapes: bell curve (normal), two peaks (bimodal), exponential decay (longtail) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Investigation revealed that DirectRunner now uses PrismRunner by default, which is a Go-based portable runner. Prism does not properly preserve TDigest data in distribution metric payloads. Root cause: The Python SDK correctly encodes TDigest data (290+ bytes), but Prism truncates the payload to only 4-5 bytes (basic count/sum/min/max). This is a limitation of the Go Prism implementation, not the Python SDK. Verified: - Python SDK creates MonitoringInfos with TDigest correctly - Protobuf serialization/deserialization preserves TDigest - BundleBasedDirectRunner works perfectly Workaround: Use BundleBasedDirectRunner for local testing with TDigest metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Removed temporary test scripts used for investigating the PrismRunner TDigest issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Summary of ChangesHello @hjtran, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the Distribution metric in Apache Beam's Python SDK by incorporating TDigest functionality. This enhancement allows for the computation of approximate quantiles and percentiles (such as p50, p90, p99) directly from DistributionResult objects, providing richer insights into data distributions without requiring any changes to how users currently update their metrics. The integration leverages the fastdigest library, is designed with backward compatibility in mind, and includes comprehensive testing. A key aspect of this change is the detailed documentation of a current limitation with the default PrismRunner, which does not fully support the extended TDigest payload, along with a recommended workaround. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
1 participant
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.