-
Notifications
You must be signed in to change notification settings - Fork 2.5k
[ci, training_utils] fix: get_nccl_backend default to nccl #4117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to fix a CI error by enabling the gloo backend for CPU parameter synchronization in the fully asynchronous policy. The change correctly adds cpu:gloo to the backend configuration for ray.util.collective.create_collective_group. However, the implementation has a critical flaw: it will cause a RuntimeError on CPU-only environments because it unconditionally attempts to get an NCCL/HCCL backend. My review includes suggestions to fix this by conditionally constructing the backend string, ensuring the code works correctly on all hardware configurations.
2d54c25 to
d29e3cc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
The pull request correctly addresses a crash on CPU-only environments by defaulting to the 'gloo' backend. However, this change introduces a naming inconsistency, as the function get_nccl_backend now returns backends other than NCCL. I've added a high-severity comment recommending a rename of the function in a follow-up change to maintain code clarity.
3c5c100 to
579bf9a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request refactors the get_nccl_backend function to get_distributed_backend, making the name more generic and improving robustness by defaulting to the 'gloo' backend for CPU-only environments instead of raising an error. The changes are applied consistently across the codebase. I have one piece of feedback regarding an outdated docstring that should be updated to align with these changes.
579bf9a to
9e2cbb7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request correctly addresses a CI failure by introducing get_distributed_backend to handle distributed training on CPU-only machines, defaulting to the gloo backend. The refactoring from get_nccl_backend is a good improvement. My review focuses on a few instances where the construction of the backend string for init_process_group could be more robust. Specifically, in a CPU-only environment, the current logic produces a redundant backend string. I've provided suggestions to handle this case more cleanly.
9e2cbb7 to
81356bb
Compare
81356bb to
e5fcce5
Compare
|
/gemini review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to fix a CI error by changing how the distributed backend is selected. However, the change introduces a critical bug by defaulting to the 'nccl' backend, which will cause failures on CPU-only systems. My review provides a code suggestion to correct this by restoring explicit hardware checks, ensuring the code works correctly across all environments.
82685e6 to
b7410b9
Compare
Fix CI error https://github.com/volcengine/verl/actions/runs/19335994615/job/55311051696 Note that in e2e_fully_async_policy CI environment, Gloo is not available, https://github.com/volcengine/verl/actions/runs/19345818399/job/55345837590 so we simply default by nccl to avoid any possible error here. Signed-off-by: Hollow Man <[email protected]>
b7410b9 to
04308e9
Compare
…e#4117) ### What does this PR do? Fix CI error https://github.com/volcengine/verl/actions/runs/19335994615/job/55311051696 Note that in e2e_fully_async_policy CI environment, Gloo is not available, https://github.com/volcengine/verl/actions/runs/19345818399/job/55345837590 so we simply default by nccl to avoid any possible error here. ### Checklist Before Starting - [X] Search for similar PRs. Paste at least one query link here: ... - [X] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [X] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [X] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [X] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [X] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [X] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) Signed-off-by: Hollow Man <[email protected]>
What does this PR do?
Fix CI error https://github.com/volcengine/verl/actions/runs/19335994615/job/55311051696
Note that in e2e_fully_async_policy CI environment, Gloo is not available, https://github.com/volcengine/verl/actions/runs/19345818399/job/55345837590 so we simply default by nccl to avoid any possible error here.
Checklist Before Starting
[{modules}] {type}: {description}(This will be checked by the CI){modules}includefsdp,megatron,sglang,vllm,rollout,trainer,ci,training_utils,recipe,hardware,deployment,ray,worker,single_controller,misc,perf,model,algo,env,tool,ckpt,doc,data,like[megatron, fsdp, doc]{type}is infeat,fix,refactor,chore,test[BREAKING]to the beginning of the title.[BREAKING][fsdp, megatron] feat: dynamic batchingTest
API and Usage Example
# Add code snippet or script demonstrating how to use thisDesign & Code Changes
Checklist Before Submitting
Important
Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.
pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=alwaysci-requestchannel in theverlSlack workspace. (If not accessible, please try the Feishu group (飞书群).)