Turning GCP Terraform/Tofu into a module (#2)

Also adding pre-commit and a devcontainer

Author: James Belchamber

.devcontainer/Dockerfile

@@ -0,0 +1,25 @@
+# Dockerfile
+FROM docker.io/golang:latest
+
+RUN apt-get update &&\
+    apt-get upgrade -y &&\
+    apt-get install -y apt-transport-https ca-certificates curl wget gnupg &&\
+    apt-get clean
+
+RUN install -m 0755 -d /etc/apt/keyrings &&\
+    curl -fsSL https://get.opentofu.org/opentofu.gpg | tee /etc/apt/keyrings/opentofu.gpg >/dev/null &&\
+    curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | gpg --no-tty --batch --dearmor -o /etc/apt/keyrings/opentofu-repo.gpg >/dev/null &&\
+    chmod a+r /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg &&\
+    echo "deb [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main" | tee /etc/apt/sources.list.d/opentofu.list > /dev/null &&\
+    chmod a+r /etc/apt/sources.list.d/opentofu.list &&\
+    apt-get update &&\
+    apt-get install -y tofu &&\
+    apt-get clean
+
+RUN mkdir -p /usr/local/bin && apt-get update && apt-get install -y unzip && curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash
+
+RUN wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | tee /usr/share/keyrings/trivy.gpg > /dev/null &&\
+    echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | tee -a /etc/apt/sources.list.d/trivy.list &&\
+    apt-get update &&\
+    apt-get install -y trivy &&\
+    apt-get clean

.devcontainer/devcontainer.json

@@ -0,0 +1,16 @@
+{
+    "name": "Juicebox Software Realm",
+    "dockerFile": "Dockerfile",
+    "features": {
+        "ghcr.io/devcontainers-extra/features/pre-commit:2": {},
+        "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
+        "ghcr.io/devcontainers/features/common-utils": {
+            "installZsh": true,
+            "installOhMyZsh": true,
+            "installOhMyZshConfig": true,
+            "configureZshAsDefaultShell": true,
+            "upgradePackages": true
+        }
+    },
+    "postCreateCommand": "bash .devcontainer/post-create.sh"
+}

.devcontainer/post-create.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+# Install pre-commit hooks
+pre-commit install

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: cef0300fd0fc4d2a87a85fa2093c6b283ea36f4b # v5.0.0
+    hooks:
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: check-case-conflict
+      - id: check-merge-conflict
+      - id: detect-private-key
+  - repo: https://github.com/antonbabenko/pre-commit-terraform
+    rev: 2f8bda194a420ad77a050a9de627d77a74841fdc # v1.99.4
+    hooks:
+      - id: terraform_fmt
+      - id: terraform_validate
+      - id: terraform_tflint
+      - id: terraform_trivy

README.md

@@ -76,48 +76,26 @@ The realm software will determine which tenant key to validate on a request by a
 
 ## GCP
 
-The following instructions will help you quickly deploy a realm to Google's App Engine Flex.
-
-Before you begin, setup a project for your realm to run in on [console.cloud.google.com](https://console.cloud.google.com) and make note of its ID.
-
-Next, setup your project environment with terraform as follows:
-```sh
-cd gcp
-terraform init
-terraform plan -var='tenant_secrets={"acme":"acme-tenant-key","anotherTenant":"another-tenant-key"}'
-terraform apply -var='tenant_secrets={"acme":"acme-tenant-key","anotherTenant":"another-tenant-key"}'
-```
-
-Note: you should update the tenant secrets `var` to reflect the actual secrets you wish to support.
-
-After terraform has finished configuring your project environment, you should see an output like follows:
-```sh
-BIGTABLE_INSTANCE_ID = "jb-sw-realms"
-GCP_PROJECT_ID = "your-project-id"
-REALM_ID = "99b2da84-b707-6203-dc35-804bbbcb8cba"
-SERVICE_ACCOUNT = "[email protected]"
-```
-
-Open the `cmd/jb-sw-realm/app.yaml` file and configure it with these values where appropriate, for example:
-Replace `{{YOUR_BIGTABLE_INSTANCE_ID}}` with `jb-sw-realms`.
-
-Finally, you can deploy the realm software by running the following command from the `cmd/jb-sw-realm` directory of the repo:
-```sh
-gcloud app deploy --project {{YOUR_GCP_PROJECT_ID}}
-```
-
-Note: you will need to have the `gcloud` command line tools installed to execute this command. You can find instructions on installing these [here](https://cloud.google.com/sdk/docs/install).
-
-This may take a few minutes, but upon success you should be able to access your realm at:
-https://{{YOUR_GCP_PROJECT_ID}}.wl.r.appspot.com
-
-If all was successful, you'll see a page render that looks something like:
-```json
-{"realmID":"99b2da84-b707-6203-dc35-804bbbcb8cba"}
+The `gcp/` directory is now a Terraform module which can be leveraged in your own Terraform codebase. For example:
+
+```terraform
+module "juicebox-software-realm" {
+  source = "github.com/rpcpool/juicebox-software-realm.git//gcp"
+
+  project_id             = "your-project-id"
+  realm_id               = "99b2da84b7076203dc35804bbbcb8cba"
+  region                 = "europe-west3"
+  zone                   = "c"
+  tenant_secrets         = {"acme":"acme-tenant-key","anotherTenant":"another-tenant-key"}
+  juicebox_image_url     = "path/to/juicebox/container/image"
+  juicebox_image_version = "latest"
+  otelcol_image_url      = "path/to/otel/collector/container/image"
+  otelcol_image_version  = "latest"
+  otelcol_config_b64     = filebase64("otel-collector-config.yaml")
+}
 ```
 
-If you wish to configure a custom domain for your new realm, visit:
-https://console.cloud.google.com/appengine/settings/domains
+This will deploy an instance of Juicebox Software Realm along with the necessary infrastructure that supports it. Please note that you will either need to deploy the Juicebox and OpenTelemetry Collector containers to a repository in GCP, or else use a remote repository which caches them from GitHub.
 
 ## AWS
 

cmd/jb-sw-realm/app.yaml

@@ -10,8 +10,8 @@ crash.log
 crash.*.log
 
 # Exclude all .tfvars files, which are likely to contain sensitive data, such as
-# password, private keys, and other secrets. These should not be part of version 
-# control as they are data points which are potentially sensitive and subject 
+# password, private keys, and other secrets. These should not be part of version
+# control as they are data points which are potentially sensitive and subject
 # to change depending on the environment.
 *.tfvars
 *.tfvars.json
@@ -32,3 +32,6 @@ override.tf.json
 # Ignore CLI configuration files
 .terraformrc
 terraform.rc
+
+# This is a module, so ignore .lock.hcl
+.terraform.lock.hcl

gcp/.gitignore

@@ -0,0 +1,23 @@
+resource "google_bigtable_instance" "instance" {
+  name         = "jb-sw-realms"
+  display_name = "Juicebox Software Realms"
+
+  cluster {
+    cluster_id = "jb-sw-realms-cluster"
+    zone       = "${var.region}-${var.zone}"
+    autoscaling_config {
+      min_nodes  = 1
+      max_nodes  = 5
+      cpu_target = 80
+    }
+  }
+}
+
+resource "google_bigtable_instance_iam_binding" "access" {
+  instance = google_bigtable_instance.instance.name
+  role     = "roles/bigtable.admin"
+
+  members = [
+    "serviceAccount:${google_service_account.service_account.email}"
+  ]
+}

gcp/.terraform.lock.hcl

@@ -1,130 +1,4 @@
-# Configure the Google Cloud provider
-provider "google" {
-  project = var.project_id
-  region  = var.region
-}
-
-# Enable required APIs
-resource "google_project_service" "app_engine" {
-  service = "appengine.googleapis.com"
-}
-
-resource "google_project_service" "secrets_manager" {
-  project = var.project_id
-  service = "secretmanager.googleapis.com"
-}
-
-resource "google_project_service" "pub_sub" {
-  project = var.project_id
-  service = "pubsub.googleapis.com"
-}
-
-# Create app engine service account
 resource "google_service_account" "service_account" {
   account_id   = "jb-sw-realms"
   display_name = "Juicebox Software Realms"
 }
-
-# Create each tenant secret
-resource "google_secret_manager_secret" "secret" {
-  for_each  = var.tenant_secrets
-  project   = var.project_id
-  secret_id = "jb-sw-tenant-${each.key}"
-  replication {
-    automatic = true
-  }
-}
-
-# Add the secret data for each tenant secret
-resource "google_secret_manager_secret_version" "secret" {
-  for_each    = var.tenant_secrets
-  secret      = google_secret_manager_secret.secret[each.key].id
-  secret_data = each.value
-}
-
-# Grant access to the app engine for each tenant secret
-resource "google_secret_manager_secret_iam_binding" "access" {
-  for_each  = var.tenant_secrets
-  project   = var.project_id
-  secret_id = google_secret_manager_secret.secret[each.key].id
-  role      = "roles/secretmanager.secretAccessor"
-
-  members = [
-    "serviceAccount:${google_service_account.service_account.email}"
-  ]
-}
-
-# Create Bigtable instance
-resource "google_bigtable_instance" "instance" {
-  project      = var.project_id
-  name         = "jb-sw-realms"
-  display_name = "Juicebox Software Realms"
-
-  cluster {
-    cluster_id = "jb-sw-realms-cluster"
-    zone       = var.zone
-    autoscaling_config {
-      min_nodes  = 1
-      max_nodes  = 5
-      cpu_target = 80
-    }
-  }
-}
-
-# Grant access to the app engine for the Bigtable instance
-resource "google_bigtable_instance_iam_binding" "access" {
-  project  = var.project_id
-  instance = google_bigtable_instance.instance.name
-  role     = "roles/bigtable.admin"
-
-  members = [
-    "serviceAccount:${google_service_account.service_account.email}"
-  ]
-}
-
-# Create App Engine application
-resource "google_app_engine_application" "app" {
-  project     = var.project_id
-  location_id = var.region
-}
-
-# Grant log writer permissions to app engine
-resource "google_project_iam_binding" "logs_writer_binding" {
-  project = var.project_id
-  role    = "roles/logging.logWriter"
-  members = [
-    "serviceAccount:${google_service_account.service_account.email}"
-  ]
-}
-
-# Grant object reader permissions to app engine so it can access Google Container Registry
-resource "google_project_iam_binding" "storage_object_viewer_binding" {
-  project = var.project_id
-  role    = "roles/storage.objectViewer"
-  members = [
-    "serviceAccount:${google_service_account.service_account.email}"
-  ]
-}
-
-# Define a custom role with the specific pub/sub perms needed.
-resource "google_project_iam_custom_role" "pubsub_role" {
-  project     = var.project_id
-  role_id     = "pubsub_role"
-  title       = "Role for managing pub/sub from a software realm"
-  description = "Role for managing pub/sub from a software realm"
-  permissions = ["pubsub.subscriptions.create",
-    "pubsub.topics.attachSubscription",
-    "pubsub.topics.create",
-    "pubsub.topics.publish",
-    "pubsub.subscriptions.consume",
-  ]
-}
-
-# Grant pub/sub access to the service account
-resource "google_project_iam_binding" "pubsub_binding" {
-  project = var.project_id
-  role    = google_project_iam_custom_role.pubsub_role.name
-  members = [
-    "serviceAccount:${google_service_account.service_account.email}"
-  ]
-}