[CI] Prevent timeouts when rebuilding containers with docker.

[larroy]

Increase timeout from 120 to 180 for pipelines
Increase timeout for docker pull as we get timeout when rebuilding the docker cache:

http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-docker-cache-refresh/job/master/1190/console

Limit parallel builds to 10

Description

Mitigation for failing CI

fixes #13817

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

[anirudhacharya]

@mxnet-label-bot add [pr-awaiting-review, Scala]

[lanking520]

It looks good to me, could you please point to some PRs that have this issue?

[larroy]

http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/clang/job/PR-13549/12/console

[larroy]

I restarted 4 PRs because of this issue

[aaronmarkham]

Does the website publish job need this increase too?
https://github.com/apache/incubator-mxnet/blob/master/docs/Jenkinsfile#L24

[larroy]

@aaronmarkham yes, could we move that into the ci/ folder for consistency? it's easy to miss if we have scripts and infrastructure in the docs folder.

[marcoabreu]

Could we hold on with the merge please. I'm not really sure whether this fixes the problem or works around another regression

[aaronmarkham]

@aaronmarkham yes, could we move that into the ci/ folder for consistency? it's easy to miss if we have scripts and infrastructure in the docs folder.

When I first put this together @marcoabreu and I discussed that, but I can't remember why it was better to have it in docs. Maybe that's changed? Marco, do you remember why? If we need to leave it there we could add some notes to the CI readme so it doesn't get overlooked.

[larroy]

Could we hold on with the merge please. I'm not really sure whether this fixes the problem or works around another regression

While we hold on CI is having time outs. It took me quite a bit to get the PR to pass CI because of the timeouts (had to manually rebuild the cache). What steps are you taking to understand if it fixes the problem? What makes you think my fix doesn't address the problem? If CI is having failures we can't merge PRs that fix CI because of protected master.

[gavinmbell]

👍

[lanking520]

A small concern there. Otherwise LGTM

[marcoabreu]

See above

[larroy]

Responded, would appreciate if this would be merged to prevent CI failures.

[lebeg]

would be great to merge