Skip to content

Commit 2cd1974

Browse files
jasonge27savingoyal
jasonge27
and
savingoyal
authored
Metaflow support for R lang
This PR introduces support for R lang. Most major features of Metaflow are supported. Documentation is hosted at https://docs.metaflow.org/v/r/ Co-authored-by: Savin <[email protected]>
1 parent f544549 commit 2cd1974

File tree

121 files changed

+17577
-7
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

121 files changed

+17577
-7
lines changed

‎.github/workflows/publish.yml

Lines changed: 50 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,31 +4,78 @@ on:
44
types: [published]
55
jobs:
66
test:
7-
name: Test on ${{ matrix.os }}
7+
name: ${{ matrix.lang }} tests on ${{ matrix.os }}
88
runs-on: ${{ matrix.os }}
99
strategy:
1010
matrix:
1111
os: [ubuntu-latest, macos-latest]
12+
lang: [Python, R]
13+
1214
steps:
1315
- uses: actions/checkout@v2
16+
1417
- name: Set up Python 2.x
18+
if: matrix.lang == 'Python'
1519
uses: actions/setup-python@v1
1620
with:
1721
python-version: '2.x'
22+
1823
- name: Install Python 2.x dependencies
24+
if: matrix.lang == 'Python'
1925
run: |
2026
python2 -m pip install --upgrade pip
2127
python2 -m pip install tox
28+
2229
- name: Set up Python 3.x
2330
uses: actions/setup-python@v1
2431
with:
2532
python-version: '3.x'
33+
2634
- name: Install Python 3.x dependencies
2735
run: |
2836
python3 -m pip install --upgrade pip
29-
python3 -m pip install tox
30-
- name: Test with tox
37+
python3 -m pip install tox numpy
38+
39+
- name: Set up R 3.6
40+
if: matrix.lang == 'R'
41+
uses: r-lib/actions/setup-r@v1
42+
with:
43+
r-version: '3.6.3'
44+
45+
- name: Install R 3.6 system dependencies
46+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
47+
run: sudo apt-get update; sudo apt-get install -y libcurl4-openssl-dev qpdf
48+
49+
- name: Install R 3.6 Rlang dependencies
50+
if: matrix.lang == 'R'
51+
run: |
52+
python3 -m pip install .
53+
Rscript -e 'install.packages("devtools", repos="https://cloud.r-project.org", Ncpus=8)'
54+
Rscript -e 'devtools::install_deps("R", dependencies=TRUE, repos="https://cloud.r-project.org", upgrade="default")'
55+
R CMD INSTALL R
56+
Rscript -e 'install.packages(c("data.table", "caret", "glmnet", "Matrix", "rjson"), repos="https://cloud.r-project.org", Ncpus=8)'
57+
58+
- name: Install latex for CRAN compatibility checks
59+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
60+
uses: r-lib/actions/setup-tinytex@v1
61+
62+
- name: Check CRAN compatibility
63+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
64+
run: |
65+
cd R/
66+
bash ./check_as_cran.sh
67+
68+
- name: Execute R tests
69+
if: matrix.lang == 'R'
70+
run: |
71+
cd R/tests
72+
Rscript testthat.R
73+
Rscript run_integration_tests.R
74+
75+
- name: Execute Python tests
76+
if: matrix.lang == 'python'
3177
run: tox
78+
3279
deploy:
3380
needs: test
3481
runs-on: ubuntu-latest

‎.github/workflows/test.yml

Lines changed: 49 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,28 +8,74 @@ on:
88
- master
99
jobs:
1010
test:
11-
name: Test on ${{ matrix.os }}
11+
name: ${{ matrix.lang }} tests on ${{ matrix.os }}
1212
runs-on: ${{ matrix.os }}
1313
strategy:
1414
matrix:
1515
os: [ubuntu-latest, macos-latest]
16+
lang: [Python, R]
17+
1618
steps:
1719
- uses: actions/checkout@v2
20+
1821
- name: Set up Python 2.x
22+
if: matrix.lang == 'Python'
1923
uses: actions/setup-python@v1
2024
with:
2125
python-version: '2.x'
26+
2227
- name: Install Python 2.x dependencies
28+
if: matrix.lang == 'Python'
2329
run: |
2430
python2 -m pip install --upgrade pip
2531
python2 -m pip install tox
32+
2633
- name: Set up Python 3.x
2734
uses: actions/setup-python@v1
2835
with:
2936
python-version: '3.x'
37+
3038
- name: Install Python 3.x dependencies
3139
run: |
3240
python3 -m pip install --upgrade pip
33-
python3 -m pip install tox
34-
- name: Test with tox
41+
python3 -m pip install tox numpy
42+
43+
- name: Set up R 3.6
44+
if: matrix.lang == 'R'
45+
uses: r-lib/actions/setup-r@v1
46+
with:
47+
r-version: '3.6.3'
48+
49+
- name: Install R 3.6 system dependencies
50+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
51+
run: sudo apt-get update; sudo apt-get install -y libcurl4-openssl-dev qpdf
52+
53+
- name: Install R 3.6 Rlang dependencies
54+
if: matrix.lang == 'R'
55+
run: |
56+
python3 -m pip install .
57+
Rscript -e 'install.packages("devtools", repos="https://cloud.r-project.org", Ncpus=8)'
58+
Rscript -e 'devtools::install_deps("R", dependencies=TRUE, repos="https://cloud.r-project.org", upgrade="default")'
59+
R CMD INSTALL R
60+
Rscript -e 'install.packages(c("data.table", "caret", "glmnet", "Matrix", "rjson"), repos="https://cloud.r-project.org", Ncpus=8)'
61+
62+
- name: Install latex for CRAN compatibility checks
63+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
64+
uses: r-lib/actions/setup-tinytex@v1
65+
66+
- name: Check CRAN compatibility
67+
if: matrix.lang == 'R' && matrix.os == 'ubuntu-latest'
68+
run: |
69+
cd R/
70+
bash ./check_as_cran.sh
71+
72+
- name: Execute R tests
73+
if: matrix.lang == 'R'
74+
run: |
75+
cd R/tests
76+
Rscript testthat.R
77+
Rscript run_integration_tests.R
78+
79+
- name: Execute Python tests
80+
if: matrix.lang == 'python'
3581
run: tox

‎.gitignore

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,12 @@ dist/
1010
.tox/
1111

1212
.ipynb_checkpoints/
13+
14+
15+
R/cran_check/
16+
R/.Rproj.user
17+
R/*.Rproj
18+
R/.Rbuildignore
19+
.Rproj.user
20+
21+
.DS_Store

‎R/DESCRIPTION

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Package: metaflow
2+
Type: Package
3+
Title: Metaflow for R-Lang
4+
Version: 2.2.0
5+
Author: Jason Ge [aut] <[email protected]>,
6+
Savin Goyal [aut, cre] <[email protected]>
7+
Maintainer: Jason Ge <[email protected]>
8+
Description: Metaflow is a human-friendly R package
9+
that helps scientists and engineers build and manage real-life data science projects.
10+
Metaflow was originally developed at Netflix to boost productivity of data scientists
11+
who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.
12+
Encoding: UTF-8
13+
License: Apache License (>= 2.0) | file LICENSE
14+
LazyData: true
15+
URL: https://metaflow.org/,
16+
https://docs.metaflow.org/,
17+
https://github.com/Netflix/metaflow
18+
BugReports: https://github.com/Netflix/metaflow/issues
19+
Imports:
20+
magrittr,
21+
R6,
22+
reticulate (>= 1.10)
23+
Suggests:
24+
cli,
25+
lubridate,
26+
testthat,
27+
knitr,
28+
rmarkdown
29+
RoxygenNote: 7.1.1
30+
Collate:
31+
'decorators.R'
32+
'flags.R'
33+
'flow.R'
34+
'metaflow_client.R'
35+
'metaflow.R'
36+
'flow_client.R'
37+
'metadata.R'
38+
'namespace.R'
39+
'parameter.R'
40+
'run.R'
41+
'utils.R'
42+
'run_client.R'
43+
'step.R'
44+
'step_client.R'
45+
'task_client.R'
46+
'zzz.R'
47+
VignetteBuilder: knitr

0 commit comments

Comments
 (0)