Order of cred_funs in gargle_env - error if not interactive gargle 1.2.0 break

[muschellij2]

I'm trying to auth with a cached token, while on GCE. I have an OAuth token, which I read in and then pass into token_fetch (technically @MarkEdmondson1234 googleAuthR is inherently calling token_fetch), but getting different behavior in interactive in RStudio (uses OAuth) vs. non-interactive (uses gce_credentials). The behavior is also different in Terminal R vs. RStudio on the same machine (due to GCE inheritance it seems).

I saw this change after upgrading to 1.2.0 (also RSPM uses it, so it bit me when using a rocker container). I don't know if the order of cred_funs would be the best fix for this, but figured that would be an easy one. I'm not sure if a Token2.0 is passed to token_fetch it is intended to use it, but I found this case it is not.

Interactive

path_to_token = "~/token.rds"
#>  
library(gargle)
token = readRDS(path_to_token)
print(token)
#> 
#> ── <Token (via gargle)> ─────────────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: XXXXXXXXXXXXXXXX
#>          email: 'my_email@email.com'
#> scopes: ...analytics, ...analytics.edit, ...analytics.manage.users, ...analytics.readonly, ...analytics.user.deletion, ...userinfo.email
#> credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
#> email = token$email
#>  
options(gargle_verbosity = "debug")
credentials_byo_oauth2(token = token, email = email)
#> → trying `credentials_byo_oauth()`
#> → putting token into the cache:
#>   ~/.cache/gargle
#> 
#> ── <Token (via gargle)> ─────────────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: XXXXXXXXXXXXXXXX
#>          email: 'my_email@email.com'
#> scopes: ...analytics, ...analytics.edit, ...analytics.manage.users, ...analytics.readonly, ...analytics.user.deletion, ...userinfo.email
#> credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
fetched = token_fetch(email = email, token = token)
#> → trying `token_fetch()`
#> → trying `credentials_service_account()`
#> → Error caught by `token_fetch()`:
#>   parse error: premature EOF
#> 
#> (right here) ------^
#>   → trying `credentials_app_default()`
#> → trying `credentials_gce()`
#> → trying `credentials_byo_oauth()`
#> → putting token into the cache:
#>   ~/.cache/gargle
fetched
#> 
#> ── <Token (via gargle)> ─────────────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: XXXXXXXXXXXXXXXX
#>          email: 'my_email@email.com'
#> scopes: ...analytics, ...analytics.edit, ...analytics.manage.users, ...analytics.readonly, ...analytics.user.deletion, ...userinfo.email
#> credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
#> > fetched$app
#> <oauth_app> XXXXXXXXXXXXX
#> key:    192019824XX-XXXXXXXX.apps.googleusercontent.com
#> secret: <hidden>

So you can see above that token_fetch gets to credentials_byo_oauth, bypassing credentials_gce. So I guess the interactive session doesn't inherit what's needed from GCE. This is not the case when running R from the terminal. Thus, interactive auth is different in RStudio vs. terminal R/Non-interactive if an OAuth token is passed on GCE. Unluckily for me, the auth in the default GCE account doesn't have the permissions needed for the scopes I need/have with the OAuth.

Non-Interactive (Reprex)

path_to_token = "~/token.rds"

library(gargle)
token = readRDS(path_to_token)
print(token)
#> 
#> ── <Token (via gargle)> ────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: XXXXXXXXXXXXXXXX
#>          email: 'my_email@email.com'
#>         scopes: ...analytics, ...analytics.edit, ...analytics.manage.users, ...analytics.readonly, ...analytics.user.deletion, ...userinfo.email
#>    credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
email = token$email

options(gargle_verbosity = "debug")
credentials_byo_oauth2(token = token, email = email)
#> trying `credentials_byo_oauth()`
#> putting token into the cache:
#> '~/.cache/gargle'
#> 
#> ── <Token (via gargle)> ────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: XXXXXXXXXXXXXXXX
#>          email: 'my_email@email.com'
#>         scopes: ...analytics, ...analytics.edit, ...analytics.manage.users, ...analytics.readonly, ...analytics.user.deletion, ...userinfo.email
#>    credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
fetched = token_fetch(email = email, token = token)
#> trying `token_fetch()`
#> trying `credentials_service_account()`
#> Error caught by `token_fetch()`:
#> parse error: premature EOF
#> 
#> (right here) ------^
#> trying `credentials_external_account()`
#> aws.ec2metadata not installed; can't detect whether running on EC2 instance
#> trying `credentials_app_default()`
#> trying `credentials_gce()`
fetched
#> <GceToken>
fetched$app
#> <oauth_app> google
#>   key:    KEY
#>   secret: <hidden>

In the terminal R/non-interactive session, it hits credentials_gce, which then succeeds, and returns the token from the GCE worker. Note, this also results in a different app for the token (which is expected)

^{Created on 2021-08-13 by the reprex package (v2.0.0)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.1.0 (2021-05-18)
#>  os       Debian GNU/Linux 10 (buster)
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language (EN)                        
#>  collate  C.UTF-8                     
#>  ctype    C.UTF-8                     
#>  tz       Etc/UTC                     
#>  date     2021-08-13                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                       
#>  askpass       1.1        2019-01-13 [2] CRAN (R 4.1.0)               
#>  cli           3.0.1      2021-07-17 [1] CRAN (R 4.1.0)               
#>  crayon        1.4.1      2021-02-08 [2] CRAN (R 4.1.0)               
#>  curl          4.3.2      2021-06-23 [1] CRAN (R 4.1.0)               
#>  digest        0.6.27     2020-10-24 [2] CRAN (R 4.1.0)               
#>  ellipsis      0.3.2      2021-04-29 [2] CRAN (R 4.1.0)               
#>  evaluate      0.14       2019-05-28 [2] CRAN (R 4.1.0)               
#>  fansi         0.5.0      2021-05-25 [2] CRAN (R 4.1.0)               
#>  fs            1.5.0      2020-07-31 [2] CRAN (R 4.1.0)               
#>  gargle      * 1.2.0.9000 2021-08-13 [1] Github (r-lib/gargle@083acb1)
#>  glue          1.4.2      2020-08-27 [2] CRAN (R 4.1.0)               
#>  highr         0.9        2021-04-16 [2] CRAN (R 4.1.0)               
#>  htmltools     0.5.1.1    2021-01-22 [2] CRAN (R 4.1.0)               
#>  httr          1.4.2      2020-07-20 [2] CRAN (R 4.1.0)               
#>  jsonlite      1.7.2      2020-12-09 [2] CRAN (R 4.1.0)               
#>  knitr         1.33       2021-04-24 [2] CRAN (R 4.1.0)               
#>  lifecycle     1.0.0      2021-02-15 [2] CRAN (R 4.1.0)               
#>  magrittr      2.0.1      2020-11-17 [2] CRAN (R 4.1.0)               
#>  openssl       1.4.4      2021-04-30 [2] CRAN (R 4.1.0)               
#>  pillar        1.6.1      2021-05-16 [2] CRAN (R 4.1.0)               
#>  pkgconfig     2.0.3      2019-09-22 [2] CRAN (R 4.1.0)               
#>  R6            2.5.0      2020-10-28 [2] CRAN (R 4.1.0)               
#>  reprex        2.0.0      2021-04-02 [2] CRAN (R 4.1.0)               
#>  rlang         0.4.11     2021-04-30 [2] CRAN (R 4.1.0)               
#>  rmarkdown     2.9        2021-06-15 [2] CRAN (R 4.1.0)               
#>  rstudioapi    0.13       2020-11-12 [2] CRAN (R 4.1.0)               
#>  sessioninfo   1.1.1      2018-11-05 [2] CRAN (R 4.1.0)               
#>  stringi       1.6.2      2021-05-17 [2] CRAN (R 4.1.0)               
#>  stringr       1.4.0      2019-02-10 [2] CRAN (R 4.1.0)               
#>  tibble        3.1.2      2021-05-16 [2] CRAN (R 4.1.0)               
#>  utf8          1.2.1      2021-03-12 [2] CRAN (R 4.1.0)               
#>  vctrs         0.3.8      2021-04-29 [2] CRAN (R 4.1.0)               
#>  withr         2.4.2      2021-04-18 [2] CRAN (R 4.1.0)               
#>  xfun          0.24       2021-06-15 [2] CRAN (R 4.1.0)               
#>  yaml          2.2.1      2020-02-01 [2] CRAN (R 4.1.0)               
#> 
#> [1] /home/jupyter/.R/library
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library

[muschellij2]

It would seem that a user can fix this by doing:

funs = gargle:::cred_funs_list()
funs = funs[c("credentials_byo_oauth2", 
              setdiff(names(funs), "credentials_byo_oauth2"))]
gargle::cred_funs_set(funs)

so I guess if you don't see this as a needed change, then you can close.

[jennybc]

I'm having some trouble parsing this problem, probably because it's really being filtered through another package.

So I'm zooming out a bit here with my answer / comment.

But the order of credential fetching functions did not change in gargle 1.2.0. I inserted a new one (re: workload identity federation), but that is neither here nor there for you (that currently only applies on AWS). However the scope logic in credentials_gce() did change and I suspect that is the correct tree to be barking up.

f6abf7d

[MarkEdmondson1234]

Perhaps via f6abf7d before the auth on GCE VM failed when it did not have cloud scopes specified (as the token is analytics only scopes), but now it does succeed via (detect_gce()) but in this case we don't want it to use the GCE VM scopes, rather the scopes in the token.

gargle/R/credentials_gce.R

Lines 129 to 132 in f6abf7d

	detect_gce <- function() {
	response <- gce_metadata_request("", stop_on_error = FALSE)
	!(inherits(response, "try-error") %||% httr::http_error(response))
	}

Its noted by @craigcitro via

Unfortunately, it does allow one new bit of confusion: a user
invoking gargle on a GCE VM without the scopes needed for an API would
previously fall through to fetching user creds; they'll now need to do so
explicitly.

So it needs that explicitly now - what is the best way to do that? I guess specify in token_fetch() - pass through the scopes in the token you are bringing.

[jennybc]

I don't think passing scopes will help you. That is, if you are assuming that some sort of scope check in credentials_gce() will fail, eventually leading to the use of user creds, I don't think that will happen now.

I think your wrapper package probably needs to insert more logic @MarkEdmondson1234. If you know you don't want to use the automatic GCE token, you could remove that cred fun from the registry. Or, you could do that conditionally, after checking and learning that the VM doesn't have the right scopes.

I'm open to discussing how or whether gargle to can offer better support here.

[jennybc]

Can you (other folks in this thread or who care / know) articulate what behaviour you actually want?

I think it's this: use the first token that has the necessary scopes.

It turns out such a condition can be hard to assess, given how scope implications actually work. But it would be helpful to reach consensus on the desired behaviour in these terms.

[jennybc]

It would also be interesting to learn more about the different path taken through the credentials_gce() code when working from RStudio vs. not, as that is probably unintended and undesirable.

[MarkEdmondson1234]

Yes, why does credentials_gce() fail when interactive but succeed when non-interactive? That seems like something that was relied upon but shouldn't have been the case?

My expected behaviour would be that if you pass a token it would also use that and not fall back on credentials_gce() or anything else. Are there horrible consequences to moving credentials_gce() to be checked last, or perhaps better move credentials_byo_oauth() to always be checked first? (as @muschellij2's fix does)

[jennybc]

Yes, why does credentials_gce() fail when interactive but succeed when non-interactive? That seems like something that was relied upon but shouldn't have been the case?

I will need one of you to do some debugging on GCE or give me your best advise on the easiest way to get into a good position to experience this for myself. It's not something I ever have an organic need for. The goal is to understand how the internals of credentials_gce() are working differently in those two contexts, specifically the stuff around GCE metadata retrieval and GCE detection.

My expected behaviour would be that if you pass a token it would also use that and not fall back on credentials_gce() or anything else...

That is a good proposal to consider. Currently, credential fetchers fail if the user-supplied input overtly breaks them. But we don't really get into the business of whether the contents of ... are clearly intended for a different fetcher. I.e. if the user has supplied input via ... that this fetcher cannot possibly honour. I will have a think on this.

[muschellij2]

I agree, and just wanted to plug that my assumption would be that if a token was passed to token_fetch that would be looked at and checked first before any auth guessing. I'm not sure if gargle should have to take the scopes into account (putting that on the user). I'm not sure if I agree that gargle shouldn't still go through cred_funs if you pass in a token and that token is malformed or has other issues.

[parthmishra]

I'm not sure if I agree that gargle shouldn't still go through cred_funs if you pass in a token and that token is malformed or has other issues.

If a token is malformed that should be a quick exit before creds_funs no? How would I know if I'm passing a bad token but silently succeeding due to another function? Seems like it could be misleading.

[parthmishra]

@jennybc a lot of us have been running into the credentials_gce() failing on 1.2.0 (Vertex AI R-notebooks env) but it seems fixed in development version due to #185 and we've seen success with it for a few weeks now. Any chance of a patch release with this included, or are there still road blockers? Thanks!

[jennybc]

I wouldn't say there are blockers, more that I don't have gargle near the very top of my todo list right now.

It's interesting / good that this seems to be fixed in the dev version.

Do those of you here still feels there's a GCE-specific angle that needs addressing?

Also I remain interested in any tips or, even better, a blog post or something about how to get myself into a situation that at least resembles the environment you're working it, so I can experience the auth for myself.

[muschellij2]

This may become a more common issue with the behavior changed in Gargle 1.2.1 and the fix of #185 from @craigcitro.

Overall, the main issue is that token_fetch exhibits inintuitive behavior when on GCE/GCP and a token is passed. I would assume that if I directly passed an auth token, it'd be used, but credentials_gce overrides it. It is more likely to override it now that the check for scopes has been removed.

Spoiler (My fix)

If someone runs into this, here is the code I use:

reorder_gargle_cred_funs_list = function(
    my_order = c("credentials_byo_oauth2", "credentials_gce")
) {
  cred_list = gargle::cred_funs_list()
  cred_list_names = names(cred_list)

  my_order = c("credentials_byo_oauth2", "credentials_gce")
  stopifnot(all(my_order %in% cred_list_names))
  indices = which(cred_list_names %in% my_order)
  stopifnot(length(indices) == length(my_order))

  cred_list_names[indices] = my_order

  reordered_list = cred_list[cred_list_names]
  gargle::cred_funs_set(reordered_list)
  return(reordered_list)
}

THis should be used in code such that the overall environment is not changed:

  cred_list = gargle::cred_funs_list()
  # reset
  on.exit({
    gargle::cred_funs_set(cred_list)
  }, add = TRUE)
reorder_gargle_cred_funs_list()

Trying to use Previous Token

Description: I would like to use a token that I generate from OAuth because it can access Google Drive and Google Sheets. I can access these things in Google Cloud Platform on a VM (Google Compute Engine (GCE)), but the metadata server that gets tokens doesn’t work the same way for Cloud Build.

Example

Here I’m going to use a previously stored token that can access drive and GCP.

library(gargle)
scopes =  c(
  "https://www.googleapis.com/auth/cloud-platform",
  "https://www.googleapis.com/auth/drive",
  "https://www.googleapis.com/auth/userinfo.email"
)

Done externally

!! Below is not reproducible because reading in token (need to run this to get outside token)

gargle::credentials_user_oauth2(scopes =  c(
  "https://www.googleapis.com/auth/cloud-platform",
  "https://www.googleapis.com/auth/drive",
  "https://www.googleapis.com/auth/userinfo.email"
))

We see the token has the required scopes are correct.

token = readRDS("token.rds")
token$email = "<removed>"
token
#> 
#> ── <Token (via gargle)> ────────────────────────────────────────────────────────
#> oauth_endpoint: google
#>            app: gargle-clio
#>          email: '<removed>'
#>         scopes: ...cloud-platform, ...drive, ...userinfo.email
#>    credentials: access_token, expires_in, refresh_token, scope, token_type, id_token
token$credentials$scope
#> [1] "openid https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email"
class(token)
#> [1] "Gargle2.0" "Token2.0"  "Token"     "R6"

Confirming we are on GCE. Cloud Build will also come up TRUE here.

gargle:::detect_gce()
#> [1] TRUE

As we see, given the default order of the credentials, credentials_gce
outranks credentials_byo_oauth2 so it will use that token over the
user-provided one. This seems counterintuitive.

gargle::with_gargle_verbosity(
  "debug",
    googledrive::drive_auth(token = token, scopes = scopes)
)
#> initializing AuthState
#> trying `token_fetch()`
#> trying `credentials_service_account()`
#> Error caught by `token_fetch()`:
#> Argument 'txt' must be a JSON string, URL or file.
#> trying `credentials_external_account()`
#> aws.ec2metadata not installed; can't detect whether running on EC2 instance
#> trying `credentials_app_default()`
#> trying `credentials_gce()`

We can confirm the token is indeed a GceToken and not of a
Gargle2.0/OAuth Token.

googledrive::drive_token()
#> <request>
#> Auth token: GceToken
class(googledrive::drive_token())
#> [1] "request"

^{Created on 2022-09-13 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Ubuntu 20.04.3 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Etc/UTC
#>  date     2022-09-13
#>  pandoc   2.17.1.1 @ /usr/lib/rstudio-server/bin/quarto/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  assertthat    0.2.1      2019-03-21 [1] RSPM (R 4.1.0)
#>  cli           3.4.0      2022-09-08 [1] RSPM (R 4.1.0)
#>  curl          4.3.2      2021-06-23 [1] RSPM (R 4.1.0)
#>  DBI           1.1.3      2022-06-18 [1] RSPM (R 4.1.0)
#>  digest        0.6.29     2021-12-01 [1] RSPM (R 4.1.0)
#>  dplyr         1.0.10     2022-09-01 [1] RSPM (R 4.1.0)
#>  evaluate      0.16       2022-08-09 [1] RSPM (R 4.1.0)
#>  fansi         1.0.3      2022-03-24 [1] RSPM (R 4.1.0)
#>  fastmap       1.1.0      2021-01-25 [1] RSPM (R 4.1.0)
#>  fs            1.5.2      2021-12-08 [1] RSPM (R 4.1.0)
#>  gargle      * 1.2.1      2022-09-08 [1] RSPM (R 4.1.0)
#>  generics      0.1.3      2022-07-05 [1] RSPM (R 4.1.0)
#>  glue          1.6.2      2022-02-24 [1] RSPM (R 4.1.0)
#>  googledrive   2.0.0      2021-07-08 [1] RSPM (R 4.1.0)
#>  highr         0.9        2021-04-16 [1] RSPM (R 4.1.0)
#>  htmltools     0.5.3      2022-07-18 [1] RSPM (R 4.1.0)
#>  httr          1.4.4      2022-08-17 [1] RSPM (R 4.1.0)
#>  jsonlite      1.8.0      2022-02-22 [1] RSPM (R 4.1.0)
#>  knitr         1.40       2022-08-24 [1] RSPM (R 4.1.0)
#>  lifecycle     1.0.1      2021-09-24 [1] RSPM (R 4.1.0)
#>  magrittr      2.0.3      2022-03-30 [1] RSPM (R 4.1.0)
#>  pillar        1.8.1      2022-08-19 [1] RSPM (R 4.1.0)
#>  pkgconfig     2.0.3      2019-09-22 [1] RSPM (R 4.1.0)
#>  purrr         0.3.4      2020-04-17 [1] RSPM (R 4.1.0)
#>  R6            2.5.1      2021-08-19 [1] RSPM (R 4.1.0)
#>  reprex        2.0.1      2021-08-05 [1] RSPM (R 4.1.0)
#>  rlang         1.0.5      2022-08-31 [1] RSPM (R 4.1.0)
#>  rmarkdown     2.16       2022-08-24 [1] RSPM (R 4.1.0)
#>  rstudioapi    0.14       2022-08-22 [1] RSPM (R 4.1.0)
#>  sessioninfo   1.2.2.9000 2022-08-24 [1] Github (r-lib/sessioninfo@0d74fe9)
#>  stringi       1.7.8      2022-07-11 [1] RSPM (R 4.1.0)
#>  stringr       1.4.1      2022-08-20 [1] RSPM (R 4.1.0)
#>  tibble        3.1.8      2022-07-22 [1] RSPM (R 4.1.0)
#>  tidyselect    1.1.2      2022-02-21 [1] RSPM (R 4.1.0)
#>  utf8          1.2.2      2021-07-24 [1] RSPM (R 4.1.0)
#>  vctrs         0.4.1      2022-04-13 [1] RSPM (R 4.1.0)
#>  withr         2.5.0      2022-03-03 [1] RSPM (R 4.1.0)
#>  xfun          0.32       2022-08-10 [1] RSPM (R 4.1.0)
#>  yaml          2.3.5      2022-02-21 [1] RSPM (R 4.1.0)
#> 
#>  [1] /usr/local/lib/R/site-library
#>  [2] /usr/local/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────