From 3b5347e1d9d2ca63a7fe52acc00cef45b85f3f5f Mon Sep 17 00:00:00 2001 From: Anirban Date: Wed, 27 Nov 2024 02:10:23 -0700 Subject: [PATCH 01/14] Added what I wrote so far --- .../index.qmd | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd new file mode 100644 index 00000000..cd6e5fe7 --- /dev/null +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -0,0 +1,87 @@ +--- +title: "Continuous find the best thread count!" +author: "Anirban Chetia" +date: "2024-11-11" +categories: [developer, github action, performance testing] +image: "" +draft: false +--- + +In an effort to address the need for continuous performance benchmarking in `data.table`, I created a [GitHub Action](https://github.com/Anirban166/Autocomment-atime-results/blob/main/action.yml) to facilitate testing the time/memory-based performance of the incoming changes that are introduced via Pull Requests (PRs) to the official GitHub repository. + +My motivation in taking this initiative was to help ensure that `data.table` consistently maintains its code efficiency or upholds its high performance standards as PRs keep coming and getting integrated frequently (meaning they need to be monitored for retaining the quality of code contributions performance-wise, especially to avoid regressions, and an automatic way to do that would be ideal, noh?). + +Through this post, I aim to share some insights regarding my action and discuss some implementation details, but before that, I'm happy to convey that the action has been live since over the past seven months now! +There are numerous examples of it being used to generate diagnostic performance plots for PRs that involve changes to the C and R files in the codebase, which can be found through the 'Pull requests' section of `data.table` on GitHub (aside from the ['Actions' tab](https://github.com/Rdatatable/data.table/actions/workflows/performance-tests.yml), where jobs keep running as new PRs and commits involving code changes emerge from time to time). + +## Key features + +- Predefined flexible tests + The action runs test cases (utilizing the [`atime`](https://github.com/tdhock/atime) package) from the setup defined in `.ci/atime/tests.R` (can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. + +- Automated commenting + Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. + > Fun facts + - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. + - If two or more commits are pushed together in rapid succession or before the previous job finishes, only the most recent among them is fully run to save CI time. + +- Versioning + The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as described in the table below: + + | Label Name | R Package Version Description | + |------------|-------------------------------------------------------------------------------------------| + | base | PR target | + | HEAD | PR source | + | Merge-base | Common ancestor between base and HEAD | + | CRAN | Latest version on the platform | + | Before | Pre-regression commit (predates the onset of the performance regression) | + | Regression | Commit that is either responsible for the degradation in performance or is affected by it | + | Fixed | Commit where the performance has been restored or improved beyond the point of regression | + | | (same as 'Before' or better than that version in terms of performance) | + +- Diagnostic visualization + A plot is uploaded within the comment which comprises of subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. + +- Timing information + The time taken for executing various tasks (such as setting up R, installing different `data.table` versions, running and plotting the test cases) is measured (in seconds) and organized in a table within the comment. + +- Links + A download link that retrieves a zipped file for the artifact containing all the `atime`-generated results is provisioned, aside from the hyperlink to the commit that triggered the workflow and generated that particular comment. + +## Usage + +The workflow can be directly fetched from the [Marketplace](https://github.com/marketplace/actions/autocomment-atime-results) for use in any GitHub repository of an R package. For example, one can use this template for their `.github/workflows/.yml`: + +```yml +name: Autocomment atime-based performance analysis on PRs + +on: + pull_request: + types: + - opened + - reopened + - synchronize + # Modify path filters as per need: + paths: + - 'R/**' + - 'src/**' + - '.ci/atime/**' + +jobs: + comment: + runs-on: ubuntu-latest + container: ghcr.io/iterative/cml:0-dvc2-base1 + env: + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + repo_token: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: Anirban166/Autocomment-atime-results@v1.4.1 +``` +The example I provided above can be customized further as needed, as long as a few things are kept intact: +- The workflow runs on a `pull_request` event +- `GITHUB_PAT` is supplied (required to authenticate git operations, have higher rate limits, etc.) +- The `container` and `repo_token` fields are specified as I did above (required for `cml` functionality) + +::: callout-note +The action is not constrained to be OS-specific and there is only one single job or set of steps that execute on the same runner. +::: From 4a9bc527f537e4bef77a9f27a89e06e2a0574940 Mon Sep 17 00:00:00 2001 From: Ani Date: Wed, 27 Nov 2024 02:18:20 -0700 Subject: [PATCH 02/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index cd6e5fe7..5cf51e75 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -1,5 +1,5 @@ --- -title: "Continuous find the best thread count!" +title: "Continuous performance benchmarking using GitHub Actions" author: "Anirban Chetia" date: "2024-11-11" categories: [developer, github action, performance testing] From d9ae57776ec2477cc96b7b6c42fbda05a0366fbc Mon Sep 17 00:00:00 2001 From: Ani Date: Wed, 27 Nov 2024 17:19:51 -0700 Subject: [PATCH 03/14] Update index.qmd --- .../index.qmd | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 5cf51e75..27d21608 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -16,16 +16,16 @@ There are numerous examples of it being used to generate diagnostic performance ## Key features -- Predefined flexible tests +- Predefined flexible tests
The action runs test cases (utilizing the [`atime`](https://github.com/tdhock/atime) package) from the setup defined in `.ci/atime/tests.R` (can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. -- Automated commenting +- Automated commenting
Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. > Fun facts - - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. - - If two or more commits are pushed together in rapid succession or before the previous job finishes, only the most recent among them is fully run to save CI time. + > - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. + > - If two or more commits are pushed together in rapid succession or before the previous job finishes, only the most recent among them is fully run to save CI time. -- Versioning +- Versioning
The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as described in the table below: | Label Name | R Package Version Description | @@ -39,13 +39,13 @@ There are numerous examples of it being used to generate diagnostic performance | Fixed | Commit where the performance has been restored or improved beyond the point of regression | | | (same as 'Before' or better than that version in terms of performance) | -- Diagnostic visualization +- Diagnostic visualization
A plot is uploaded within the comment which comprises of subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. -- Timing information +- Timing information
The time taken for executing various tasks (such as setting up R, installing different `data.table` versions, running and plotting the test cases) is measured (in seconds) and organized in a table within the comment. -- Links +- Links
A download link that retrieves a zipped file for the artifact containing all the `atime`-generated results is provisioned, aside from the hyperlink to the commit that triggered the workflow and generated that particular comment. ## Usage From b6b9ecd3caac5569e1be1813ff35a07bc75e07bd Mon Sep 17 00:00:00 2001 From: Ani Date: Wed, 27 Nov 2024 17:30:05 -0700 Subject: [PATCH 04/14] Update index.qmd --- .../index.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 27d21608..c51b3fd7 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -23,10 +23,10 @@ There are numerous examples of it being used to generate diagnostic performance Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. > Fun facts > - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. - > - If two or more commits are pushed together in rapid succession or before the previous job finishes, only the most recent among them is fully run to save CI time. + > - If multiple commits are pushed together in quick succession or before the previous job finishes, only the most recent one among them is fully run to save CI time. - Versioning
- The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as described in the table below: + The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as enlisted in the table below: | Label Name | R Package Version Description | |------------|-------------------------------------------------------------------------------------------| From 9528c26467029b60517218debe4482492a9300ff Mon Sep 17 00:00:00 2001 From: Ani Date: Wed, 27 Nov 2024 17:34:52 -0700 Subject: [PATCH 05/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index c51b3fd7..dc5b4186 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -40,7 +40,7 @@ There are numerous examples of it being used to generate diagnostic performance | | (same as 'Before' or better than that version in terms of performance) | - Diagnostic visualization
- A plot is uploaded within the comment which comprises of subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. + Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. - Timing information
The time taken for executing various tasks (such as setting up R, installing different `data.table` versions, running and plotting the test cases) is measured (in seconds) and organized in a table within the comment. From d1d9738fb0bc34df6094b6126fcdff69b2af5724 Mon Sep 17 00:00:00 2001 From: Ani Date: Wed, 27 Nov 2024 18:11:43 -0700 Subject: [PATCH 06/14] Update index.qmd --- .../index.qmd | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index dc5b4186..30c61dcf 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -17,7 +17,7 @@ There are numerous examples of it being used to generate diagnostic performance ## Key features - Predefined flexible tests
- The action runs test cases (utilizing the [`atime`](https://github.com/tdhock/atime) package) from the setup defined in `.ci/atime/tests.R` (can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. + The action runs test cases (utilizing the [`atime`](https://github.com/tdhock/atime) package) from the setup defined in `.ci/atime/tests.R` (the path can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. - Automated commenting
Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. @@ -28,16 +28,15 @@ There are numerous examples of it being used to generate diagnostic performance - Versioning
The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as enlisted in the table below: - | Label Name | R Package Version Description | - |------------|-------------------------------------------------------------------------------------------| - | base | PR target | - | HEAD | PR source | - | Merge-base | Common ancestor between base and HEAD | - | CRAN | Latest version on the platform | - | Before | Pre-regression commit (predates the onset of the performance regression) | - | Regression | Commit that is either responsible for the degradation in performance or is affected by it | - | Fixed | Commit where the performance has been restored or improved beyond the point of regression | - | | (same as 'Before' or better than that version in terms of performance) | + | Label Name | R Package Version Description | + |------------|------------------------------------------------------------------------------------------------| + | base | PR target | + | HEAD | PR source | + | Merge-base | Common ancestor between base and HEAD | + | CRAN | Latest version on the platform | + | Before | Pre-regression commit (predates the onset of the performance regression) | + | Regression | Commit that is either responsible for the degradation in performance or is affected by it | + | Fixed | Commit where the performance has been restored or even improved to exceed the 'Before' version | - Diagnostic visualization
Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. From 84483d5068535654e1e90f4e7b7738c12812fb3c Mon Sep 17 00:00:00 2001 From: Ani Date: Thu, 28 Nov 2024 00:31:05 -0700 Subject: [PATCH 07/14] Update index.qmd --- .../index.qmd | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 30c61dcf..81f1992a 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -11,8 +11,7 @@ In an effort to address the need for continuous performance benchmarking in `dat My motivation in taking this initiative was to help ensure that `data.table` consistently maintains its code efficiency or upholds its high performance standards as PRs keep coming and getting integrated frequently (meaning they need to be monitored for retaining the quality of code contributions performance-wise, especially to avoid regressions, and an automatic way to do that would be ideal, noh?). -Through this post, I aim to share some insights regarding my action and discuss some implementation details, but before that, I'm happy to convey that the action has been live since over the past seven months now! -There are numerous examples of it being used to generate diagnostic performance plots for PRs that involve changes to the C and R files in the codebase, which can be found through the 'Pull requests' section of `data.table` on GitHub (aside from the ['Actions' tab](https://github.com/Rdatatable/data.table/actions/workflows/performance-tests.yml), where jobs keep running as new PRs and commits involving code changes emerge from time to time). +Through this post, I aim to share some insights regarding my action and discuss some implementation details, but before that, I'm happy to convey that it has been live since over the past seven months now! There are numerous examples of it being used to generate diagnostic performance plots for PRs that involve changes to the C and R files in the codebase, which can be found through the 'Pull requests' section of `data.table` on GitHub (aside from the ['Actions' tab](https://github.com/Rdatatable/data.table/actions/workflows/performance-tests.yml), where jobs keep popping up as new PRs and commits involving code changes emerge from time to time). ## Key features From f4de6e0842b6fac1529e0a813a4e276e72c33dd7 Mon Sep 17 00:00:00 2001 From: Ani Date: Mon, 2 Dec 2024 21:35:33 -0700 Subject: [PATCH 08/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 81f1992a..10ba310a 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -38,7 +38,7 @@ Through this post, I aim to share some insights regarding my action and discuss | Fixed | Commit where the performance has been restored or even improved to exceed the 'Before' version | - Diagnostic visualization
- Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. + Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for a more proportional preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. - Timing information
The time taken for executing various tasks (such as setting up R, installing different `data.table` versions, running and plotting the test cases) is measured (in seconds) and organized in a table within the comment. From 3e96b5851b617de1c6a0b952074930ebc23e208b Mon Sep 17 00:00:00 2001 From: Ani Date: Mon, 2 Dec 2024 21:36:32 -0700 Subject: [PATCH 09/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 10ba310a..bd89ca26 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -38,7 +38,7 @@ Through this post, I aim to share some insights regarding my action and discuss | Fixed | Commit where the performance has been restored or even improved to exceed the 'Before' version | - Diagnostic visualization
- Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for a more proportional preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. + Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for a well-proportioned preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. - Timing information
The time taken for executing various tasks (such as setting up R, installing different `data.table` versions, running and plotting the test cases) is measured (in seconds) and organized in a table within the comment. From fc0953cd1c238a43d831b6f997973f84cdcda714 Mon Sep 17 00:00:00 2001 From: Ani Date: Tue, 3 Dec 2024 16:47:10 -0700 Subject: [PATCH 10/14] Update index.qmd --- .../index.qmd | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index bd89ca26..2c14a52a 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -27,15 +27,17 @@ Through this post, I aim to share some insights regarding my action and discuss - Versioning
The action computes the tests on different `data.table` versions that can be visually compared on the resultant plot. These include various labels, as enlisted in the table below: - | Label Name | R Package Version Description | - |------------|------------------------------------------------------------------------------------------------| - | base | PR target | - | HEAD | PR source | - | Merge-base | Common ancestor between base and HEAD | - | CRAN | Latest version on the platform | - | Before | Pre-regression commit (predates the onset of the performance regression) | - | Regression | Commit that is either responsible for the degradation in performance or is affected by it | - | Fixed | Commit where the performance has been restored or even improved to exceed the 'Before' version | + | Label Name | R Package Version Description | + |------------|-------------------------------------------------------------------------------------------------| + | base | PR target | + | HEAD | PR source | + | merge-base | Common ancestor between base and HEAD | + | CRAN | Latest version on the platform | + | Before | Pre-regression commit (predates the onset of the performance regression) | + | Regression | Commit that is either responsible for the degradation in performance or is affected by it | + | Fixed | Commit where the performance has been restored or even improved to exceed the 'Before' version | + | Slow | Older version with slower performance (non-regression) when compared to the latest developments | + | Fast | Newer version that demonstrates noticeable performance improvement over the 'Slow' version | - Diagnostic visualization
Plots are uploaded within the comment which contain subplots for each test case, showing the time and memory trends across different `data.table` versions. The plot shown in the PR threads will be one generated for a well-proportioned preview, meaning it is condensed to only show the top 4 tests (this number can be configured using the `N.tests.preview` variable in the `tests.R` file) based on having the most significant differences between HEAD and min. The full version (all tests) will be shown when we click/tap on the plot. From 269d0167bd53b1c329c7f54a7d4e8b5136258c66 Mon Sep 17 00:00:00 2001 From: Ani Date: Thu, 12 Dec 2024 04:25:43 -0700 Subject: [PATCH 11/14] Added my new section 'Steps' along with some references (TBU) --- .../index.qmd | 30 ++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 2c14a52a..f0c13d18 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -20,7 +20,6 @@ Through this post, I aim to share some insights regarding my action and discuss - Automated commenting
Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. - > Fun facts > - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. > - If multiple commits are pushed together in quick succession or before the previous job finishes, only the most recent one among them is fully run to save CI time. @@ -85,3 +84,32 @@ The example I provided above can be customized further as needed, as long as a f ::: callout-note The action is not constrained to be OS-specific and there is only one single job or set of steps that execute on the same runner. ::: + +## Steps + +Interested to learn more about the code behind this? +Fret not! In this section, I’ll walk you through the steps I created in my workflow, one by one - right from the actions, software, and snippets involved to how they fit into the overall logic. + +To begin, I use the `checkout` action to fetch the repository's current contents into the runner's file system. This allows my workflow to access and work with the target project's source code in the right branch. Note that I set `fetch-depth` to 0 as I want all commits, branches, and tags (the entire history basically). This is essential for running other versions of the target R package, as otherwise the default value of 0 would only fetch the latest commit on the checked-out branch. + +Next, I disable the safe directory check on my repository to bypass the restriction on running commands within foreign directories that `git` by default enables for security purposes. + +I then use two git switches (rationale[^double-git-switch]) to ensure local branch references exist and can be found by `atime` when using `git2r::revparse_single` to pick up the right environment variables for `HEAD` and `base`. + +For a standard R setup, I use the RStudio Package Manager (RSPM) to install the latest version of R. + +Next up, I perform an up-to-date system-wide installation of `libgit2` (requisite for `git2r` operations; in turn, `atime` requires `git2r`) from source. + +I then proceed to install the required R packages from a CRAN mirror. These include `atime` with its hard dependencies plus the packages required for generating the diagnostic visualizations. I follow up (within the same step) by running `atime::atime_pkg` (using the `tests.R` file from the `.ci` directory of the target package) in the workspace as allocated by my checkout step (`$GITHUB_WORKSPACE` environment variable pertains to the default checkout directory). + +All of the results that have been generated are then uploaded as an artifact using the `upload-artifact` action. v4[^upload-artifact-v4] brings this feature to the table, allowing artifacts to be identifiable within a workflow (after the have been generated and uploaded, with a reference to that step) as the action's API could now create an ID variable that is available within the succeeding steps of the workflow (I use that to construct the artifact retrieval URLs). + +Finally, its time to publish the results within a comment in PR threads via the GitHub Actions bot! Everything goes into a markdown file - Two plots (one hyperlinked to the other), the SHA for the commit everything is based upon, the link to download the artifact (which again, is being concocted using various environment variables), and an organized table with timing information for different measured phases (the calculations for which are run within this step, and the timestamp recording points are distributed accordingly throughout the workflow and collected in `$GITHUB_ENV` for ease of access in subsequent steps). + +The order of these steps and segregation of tasks as I described can be also found in my old slides[^old-gha-slides-drive-link]. + +## References + +[^double-git-switch]: https://github.com/Anirban166/Autocomment-atime-results/issues/33#issuecomment-2038431272 +[^upload-artifact-v4]: https://github.com/Anirban166/Autocomment-atime-results/issues/17 +[^old-gha-slides-drive-link]: https://drive.google.com/file/d/1_uD0k6vJMpxw9jiLQQSt2H8Da5kdCUb5/view From 089f74f6ec6c69f886d42dddb4a9d448479a6eb7 Mon Sep 17 00:00:00 2001 From: Ani Date: Thu, 12 Dec 2024 19:22:08 -0700 Subject: [PATCH 12/14] Added 'Future work' section, more references, and minor edits --- .../index.qmd | 35 ++++++++++++++----- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index f0c13d18..32229647 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -7,7 +7,7 @@ image: "" draft: false --- -In an effort to address the need for continuous performance benchmarking in `data.table`, I created a [GitHub Action](https://github.com/Anirban166/Autocomment-atime-results/blob/main/action.yml) to facilitate testing the time/memory-based performance of the incoming changes that are introduced via Pull Requests (PRs) to the official GitHub repository. +In an effort to address the need for continuous performance benchmarking in `data.table`, I created a GitHub Action[^gha] to facilitate testing the time/memory-based performance of the incoming changes that are introduced via Pull Requests (PRs) to the official GitHub repository. My motivation in taking this initiative was to help ensure that `data.table` consistently maintains its code efficiency or upholds its high performance standards as PRs keep coming and getting integrated frequently (meaning they need to be monitored for retaining the quality of code contributions performance-wise, especially to avoid regressions, and an automatic way to do that would be ideal, noh?). @@ -16,10 +16,10 @@ Through this post, I aim to share some insights regarding my action and discuss ## Key features - Predefined flexible tests
- The action runs test cases (utilizing the [`atime`](https://github.com/tdhock/atime) package) from the setup defined in `.ci/atime/tests.R` (the path can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. + The action runs test cases (utilizing the `atime`[^atime] package) from the setup defined in `.ci/atime/tests.R` (the path can be customized) on different versions of `data.table` (or the R package being tested). These tests are either based on documented historical regressions or performance improvements. - Automated commenting
- Using [`cml`](https://github.com/iterative/cml), the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. + Using `cml`[^cml], the action posts information/results in a comment on the pull request thread. The comment is automatically edited with each new push to avoid cluttering, ensuring that only one comment exists per PR, which is the updated or latest one. > - The comment is authored by a GitHub bot and operates using the `GITHUB_TOKEN` I provide to authenticate itself and interact with the GitHub API within the scope of the workflow. > - If multiple commits are pushed together in quick succession or before the previous job finishes, only the most recent one among them is fully run to save CI time. @@ -49,7 +49,7 @@ Through this post, I aim to share some insights regarding my action and discuss ## Usage -The workflow can be directly fetched from the [Marketplace](https://github.com/marketplace/actions/autocomment-atime-results) for use in any GitHub repository of an R package. For example, one can use this template for their `.github/workflows/.yml`: +The workflow can be directly fetched from the Marketplace[^marketplace-version] for use in any GitHub repository of an R package. For example, one can use this template for their `.github/workflows/.yml`: ```yml name: Autocomment atime-based performance analysis on PRs @@ -88,13 +88,13 @@ The action is not constrained to be OS-specific and there is only one single job ## Steps Interested to learn more about the code behind this? -Fret not! In this section, I’ll walk you through the steps I created in my workflow, one by one - right from the actions, software, and snippets involved to how they fit into the overall logic. +Fret not! In this section, I’ll walk you through the steps I have in my workflow, one by one - right from the actions, software, and snippets involved to how they fit into the overall logic. To begin, I use the `checkout` action to fetch the repository's current contents into the runner's file system. This allows my workflow to access and work with the target project's source code in the right branch. Note that I set `fetch-depth` to 0 as I want all commits, branches, and tags (the entire history basically). This is essential for running other versions of the target R package, as otherwise the default value of 0 would only fetch the latest commit on the checked-out branch. Next, I disable the safe directory check on my repository to bypass the restriction on running commands within foreign directories that `git` by default enables for security purposes. -I then use two git switches (rationale[^double-git-switch]) to ensure local branch references exist and can be found by `atime` when using `git2r::revparse_single` to pick up the right environment variables for `HEAD` and `base`. +I then use two git switches (rationale[^why-double-git-switch]) to ensure local branch references exist and can be found by `atime` when using `git2r::revparse_single` to pick up the right environment variables for `HEAD` and `base`. For a standard R setup, I use the RStudio Package Manager (RSPM) to install the latest version of R. @@ -102,14 +102,31 @@ Next up, I perform an up-to-date system-wide installation of `libgit2` (requisit I then proceed to install the required R packages from a CRAN mirror. These include `atime` with its hard dependencies plus the packages required for generating the diagnostic visualizations. I follow up (within the same step) by running `atime::atime_pkg` (using the `tests.R` file from the `.ci` directory of the target package) in the workspace as allocated by my checkout step (`$GITHUB_WORKSPACE` environment variable pertains to the default checkout directory). -All of the results that have been generated are then uploaded as an artifact using the `upload-artifact` action. v4[^upload-artifact-v4] brings this feature to the table, allowing artifacts to be identifiable within a workflow (after the have been generated and uploaded, with a reference to that step) as the action's API could now create an ID variable that is available within the succeeding steps of the workflow (I use that to construct the artifact retrieval URLs). +All of the results that have been generated are then uploaded as an artifact using the `upload-artifact` action. v4[^upload-artifact-v4] brings this feature (allowing artifacts to be identifiable within a workflow) to the table as the action's API can now create ID variables that are available (after the artifacts have been generated and uploaded) within the succeeding steps of the workflow (I use them to construct the artifact retrieval URLs). Finally, its time to publish the results within a comment in PR threads via the GitHub Actions bot! Everything goes into a markdown file - Two plots (one hyperlinked to the other), the SHA for the commit everything is based upon, the link to download the artifact (which again, is being concocted using various environment variables), and an organized table with timing information for different measured phases (the calculations for which are run within this step, and the timestamp recording points are distributed accordingly throughout the workflow and collected in `$GITHUB_ENV` for ease of access in subsequent steps). -The order of these steps and segregation of tasks as I described can be also found in my old slides[^old-gha-slides-drive-link]. +This specific order and segregation of tasks above can also be found in my old slides[^old-gha-slides-drive-link]. + +## Future work + +My action has come a long way since the time I created an issue[^gha-introduction-issue] to introduce it to the `data.table` community and subsequently the PR[^gha-integration-pr] through which it got integrated into the project (the follow-up[^integration-follow-up-pr] to that also included my first `atime` test!), but the main goal for me in updating it from time to time (e.g. v1.4.1[^v1.4.1] and v1.3.1[^v1.3.1]) since then has been constant - to maintain the current functionality of automatically and actively monitoring changes in PRs for noticeable impact in performance (avoiding regressions is the highlighted focus, but the same enthusiasm applies in detecting improvements or even no difference/stability). As and when required, the GHA can be expected to receive updates as long as this approach remains useful, or the needs of the `data.table` project/community in particular align with this goal. + +If reading so far has piqued your curiosity enough that you would like to contribute in terms of optimizing the workflow, and if by the laws of coincidence you also happen to be a student, I would recommend checking out the Google Summer of Code[^gsoc] program as I recently wrote in detail a project[^caching-gsoc-project] (primarily based on minifying package versions and caching/reusing ones based on historical references to save CI resources/time) for extending work on this action. Until next time, happy coding! ## References -[^double-git-switch]: https://github.com/Anirban166/Autocomment-atime-results/issues/33#issuecomment-2038431272 +[^gha]: https://github.com/Anirban166/Autocomment-atime-results/blob/main/action.yml +[^atime]: https://github.com/tdhock/atime +[^cml]: https://github.com/iterative/cml +[^marketplace-version]: https://github.com/marketplace/actions/autocomment-atime-results +[^why-double-git-switch]: https://github.com/Anirban166/Autocomment-atime-results/issues/33#issuecomment-2038431272 [^upload-artifact-v4]: https://github.com/Anirban166/Autocomment-atime-results/issues/17 [^old-gha-slides-drive-link]: https://drive.google.com/file/d/1_uD0k6vJMpxw9jiLQQSt2H8Da5kdCUb5/view +[^gha-introduction-issue]: https://github.com/Rdatatable/data.table/issues/6065 +[^gha-integration-pr]: https://github.com/Rdatatable/data.table/pull/6078 +[^integration-pr-follow-up]: https://github.com/Rdatatable/data.table/pull/6094 +[^v1.4.1]: https://github.com/Rdatatable/data.table/pull/6597 +[^v1.3.1]: https://github.com/Rdatatable/data.table/pull/6545 +[^gsoc]: https://summerofcode.withgoogle.com/ +[^caching-gsoc-project]: https://github.com/rstats-gsoc/gsoc2025/wiki/Optimizing-a-performance-testing-workflow-by-reusing-minified-R-package-versions-between-CI-runs From 244ccec2c17431145a2b2a0040b90c40f652890b Mon Sep 17 00:00:00 2001 From: Ani Date: Thu, 12 Dec 2024 19:32:08 -0700 Subject: [PATCH 13/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 32229647..9808c4bc 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -110,7 +110,7 @@ This specific order and segregation of tasks above can also be found in my old s ## Future work -My action has come a long way since the time I created an issue[^gha-introduction-issue] to introduce it to the `data.table` community and subsequently the PR[^gha-integration-pr] through which it got integrated into the project (the follow-up[^integration-follow-up-pr] to that also included my first `atime` test!), but the main goal for me in updating it from time to time (e.g. v1.4.1[^v1.4.1] and v1.3.1[^v1.3.1]) since then has been constant - to maintain the current functionality of automatically and actively monitoring changes in PRs for noticeable impact in performance (avoiding regressions is the highlighted focus, but the same enthusiasm applies in detecting improvements or even no difference/stability). As and when required, the GHA can be expected to receive updates as long as this approach remains useful, or the needs of the `data.table` project/community in particular align with this goal. +My action has come a long way since the time I created an issue[^gha-introduction-issue] to introduce it to the `data.table` community and subsequently the PR[^gha-integration-pr] through which it got integrated into the project (the follow-up[^integration-follow-up-pr] to that also included my first `atime` test!), but the main goal for me in updating it from time to time (e.g. v1.4.1[^v1.4.1] and v1.3.1[^v1.3.1]) since then has been constant - to maintain the current functionality of automatically and actively monitoring changes in PRs for noticeable impact in performance (avoiding regressions is the highlighted focus, but the same enthusiasm applies in detecting improvements or observing stability). As and when required, the GHA can be expected to receive updates (or break out of a potential plateau) as long as this approach remains useful, or the needs of the `data.table` project/community in particular align with this goal. If reading so far has piqued your curiosity enough that you would like to contribute in terms of optimizing the workflow, and if by the laws of coincidence you also happen to be a student, I would recommend checking out the Google Summer of Code[^gsoc] program as I recently wrote in detail a project[^caching-gsoc-project] (primarily based on minifying package versions and caching/reusing ones based on historical references to save CI resources/time) for extending work on this action. Until next time, happy coding! From 0afae6fe982f82d831b35401571ebbb834192970 Mon Sep 17 00:00:00 2001 From: Ani Date: Thu, 12 Dec 2024 19:44:54 -0700 Subject: [PATCH 14/14] Update index.qmd --- .../2024-11-11-performance-testing-gha-anirban_chetia/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd index 9808c4bc..c35e158b 100644 --- a/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd +++ b/posts/2024-11-11-performance-testing-gha-anirban_chetia/index.qmd @@ -1,5 +1,5 @@ --- -title: "Continuous performance benchmarking using GitHub Actions" +title: "Continuous performance testing using GitHub Actions" author: "Anirban Chetia" date: "2024-11-11" categories: [developer, github action, performance testing]