Skip to content

PerfTest - Orchestrator Improvements #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
clhain opened this issue May 20, 2025 · 0 comments
Open

PerfTest - Orchestrator Improvements #467

clhain opened this issue May 20, 2025 · 0 comments
Assignees
@clhain
Labels
enhancement New feature or request pipeline Rust Pipeline Related Tasks rust Pull requests that update Rust code

Comments

@clhain
Copy link
Contributor

clhain commented May 20, 2025

parent #377

Orchestrator needs to be able to support:

  • Flexible/standardized mechanisms for plugging in arbitrary components (data senders / SUTs / data receivers).
  • Flexible mechanisms for intercepting and customizing behaviors at various points in the component lifecycles (e.g. build executables, containers, deploy infrastructure, etc)
  • Phased test(s) execution across a set of components (pipeline), with varied parameters (e.g. what load to send, how to process, where to emit it).
  • Tracking state / duration / arbitrary additional data for each test execution / phase / component lifecycle event.
  • Flexible / pluggable reporting and test pass/fail criteria definitions.
  • Customizable retry / error handling mechanisms for component lifecycle and test phase execution.
  • Tracking for component and underlying host/infrastructure resource usage and error indicators (detect when test results are likely unstable).
  • Detection of steady-state / support for warm-up periods.
  • Support for different load strategies (step up/down, constant rate), different validation criteria (simple counting, 'golden' result comparison).
@clhain clhain mentioned this issue May 20, 2025
Merged
@lquerel lquerel added enhancement New feature or request pipeline Rust Pipeline Related Tasks rust Pull requests that update Rust code labels May 20, 2025
@lquerel lquerel moved this to In Progress in OTel-Arrow May 20, 2025
@clhain clhain mentioned this issue May 20, 2025
Merged
@clhain clhain mentioned this issue May 29, 2025
Merged
lquerel pushed a commit that referenced this issue May 30, 2025
@clhain
Perf - support hook execution and runtime data on test framework elem…
e79896a 
…ents. (#493)

This change refactors hook and runtime classes and adds support for them
on Test Elements (Test Suite, Test, Test Step). This will allow us to
e.g. run global hooks in order to e.g. deploy infrastructure without
having to tie it to a particular component. Change also includes
restructuring to provide more uniformity between component and test
element naming, and introduces base classes for test elements and their
associated context classes.

- Rename LifecycleComponent -> Component
- Rename LifecyclePhase -> ComponentPhase
- Rename HookableLifecyclePhase -> HookableComponentPhase
- Rename LifecycleHookContext -> ComponentHookContext
- Move ComponentHookContext, HookableComponentPhase -> lib.core.context
- Rename ComponentRuntime -> Runtime 
- Move Runtime to lib.core.runtime (new).
- Add TestFrameworkElement as a base for all Test Element classes
(TestSuite, TestDefinition, TestStep)
- Added TestFrameworkElementContext as a base for all Test Element
related contexts.
- Added TestElementHookContext, HookableTestPhase in lib.core.context

This should be the last major change to the core. Config factory to
follow, then various strategy implementations. #467
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@clhain clhain

Labels
enhancement New feature or request pipeline Rust Pipeline Related Tasks rust Pull requests that update Rust code
Projects
OTel-Arrow
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lquerel @clhain