Archival Notice

This module was deprecated and archived in favor of the following modules:

• https://github.com/Ferlab-Ste-Justine/terraform-openstack-kubernetes-load-balancer
• https://github.com/Ferlab-Ste-Justine/terraform-openstack-kubernetes-node
• https://github.com/Ferlab-Ste-Justine/terraform-openstack-kubernetes-security-groups

While less concice, the above modules provide needed flexibility when operating a kubernetes cluster by allowing us to reprovision individual nodes and by decoupling the networking from the vms.

About

This package is a terraform module to provision the machines for a kubernetes cluster on openstack (without the kubernetes installation).

This includes the masters, the workers, a load balancer and accompanying security groups.

The load balancer is a vm with a dockerized haproxy, configured to load balance on the masters k8 api ports and on the workers ingresses.

The load balancer exposes the masters k8 api on port 6443 and the workers ingress http traffic on port 80 and ingress https traffic on port 443. It operates as a tcp proxy and does not perform tls termination.

The following security groups are returned by the module to provider additional access beyond what is allowed via the load balancer:

• bastion: Allows to ssh on any node in the cluster and allows access to the masters on the k8 api port
• master_client: Allows access to all ports on the masters
• worker_client: Allows access to all ports on the workers

Requirements

The module has been tested with a recent version of Ubuntu. Your mileage may vary with other distributions.

Additionally, this module is dependant on dns servers for the load balancer to resove the worker and master ips. The dns servers should be updated with the outputed ips of this module. See the example.

Usage

Variables

The module takes the following variables as input:

• namespace: A string to namespace all the vm names (ie, <vm name>-<namespace>). If this variable is omitted, a namespace suffix will not be added.
• masters_count: The number of masters to provision.
• masters_flavor_id: The id of the flavor the masters will have.
• masters_extra_security_group_ids: List of extra security groups to assign to the masters beyond those already assigned by the module. Defaults to []
• workers_count: The number of workers to provision.
• workers_flavor_id: The id of the flavor the workers will have.
• workers_extra_security_group_ids: List of extra security groups to assign to the workers beyond those already assigned by the module. Defaults to []
• load_balancer_flavor_id: The id of the flavor the load balancer will have. If you do not wish to provision a load balancer, leave this value at its blank default.
• image_id: ID of the image to use to provision all vms
• network_id: Id of the network to connect all vms to
• keypair_name: Name of the keypair that will be used to ssh on the vms
• k8_max_workers_count: Expected maximum for the number of workers (required by haproxy). Defaults to 100.
• k8_max_masters_count: Expected maximum for the number of masters (required by haproxy). Defaults to 7.
• nameserver_ips: Ips of the nameservers that provider the internal domain for your k8 masters and workers
• internal_k8_domain: Iternal domain for your k8 clusters. workers.<internal_k8_domain> should resolve to the workers in the cluster and masters.<internal_k8_domain> should resolve to the masters.
• masters_api_timeout: Amount of time a connection to the k8 api can remain idle before it times out. Defaults to 5000ms.
• masters_api_port: Port on the master nodes that the load balancer should direct api traffic to. Defaults to 6443.
• masters_max_api_connections: Maximum number of allowed concurrent connections to the k8 api. Defaults to 200.
• workers_ingress_http_timeout: Amount of time an http connection on the workers' ingress can remain idle before it times out. Defaults to 5000ms.
• workers_ingress_http_port: Port on the worker nodes that the load balancer should direct ingress http traffic to. Defaults to 30000.
• workers_ingress_max_http_connections: Maximum number of allowed concurrent ingress http connections. Defaults to 200.
• workers_ingress_https_timeout: Amount of time an https connection on the workers' ingress can remain idle before it times out. Defaults to 5000ms.
• workers_ingress_https_port: Port on the worker nodes that the load balancer should direct ingress https traffic to. Defaults to 30001.
• workers_ingress_max_https_connections: Maximum number of allowed concurrent ingress https connections. Defaults to 200.

Output

The module outputs the following variables as output:

• masters: list of the masters with each entry having the following format...

{
  id: <id of the master>
  ip: <ip address of the master>
}

• workers: list of the workers with each entry having the following format...

{
  id: <id of the master>
  ip: <ip address of the master>
}

• load_balancer: id and ip of the load balancer taking the following format:

{
  id: <id of the master>
  ip: <ip address of the master>
}

• groups: Security groups (ie, resources of type openstack_networking_secgroup_v2) that can be used to provide nodes with additional access to the cluster. It has the following 4 groups: bastion, master_client, worker_client, worker. The worker group is already associated with the k8 worker nodes and should not be associated with other nodes, although in exceptional cases, it can be extended with additional rules to give kubernetes workers access to other nodes without creating circular dependecies.

Example

Here is an example of how the module might be used:

Dns:

resource "openstack_objectstorage_container_v1" "dns" {
  name   = "dns"
  content_type = "text/plain"
}

resource "openstack_networking_port_v2" "coredns" {
  count          = 3
  name           = "coredns-${count.index + 1}"
  network_id     = module.reference_infra.networks.internal.id
  security_group_ids = [module.reference_infra.security_groups.default.id]
  admin_state_up = true
}

locals {
  nameserver_ips = [for network_port in openstack_networking_port_v2.coredns: network_port.all_fixed_ips.0]
}

module "dns_servers" {
  source = "git::https://github.com/Ferlab-Ste-Justine/openstack-coredns.git"
  image_id = module.ubuntu_image.id
  flavor_id = module.reference_infra.flavors.nano.id
  network_ports = openstack_networking_port_v2.coredns
  keypair_name = openstack_compute_keypair_v2.dns_keypair.name
  container_info = {
    name = openstack_objectstorage_container_v1.dns.name
    os_auth_url = var.auth_url
    os_region_name = var.region
    os_app_id = var.application_credential_id
    os_app_secret = var.application_credential_secret
  }
}

Cluster:

resource "openstack_networking_floatingip_v2" "k8_api_lb_floating_ip" {
  pool = module.reference_infra.networks.external.name
}

module "kubernetes_cluster" {
  source = "git::https://github.com/Ferlab-Ste-Justine/openstack-kubernetes-cluster.git"
  masters_flavor_id = module.reference_infra.flavors.micro.id
  workers_flavor_id = module.reference_infra.flavors.small.id
  load_balancer_flavor_id = module.reference_infra.flavors.micro.id
  image_id = module.ubuntu_image.id
  keypair_name = openstack_compute_keypair_v2.k8_keypair.name
  network_id = module.reference_infra.networks.internal.id
  workers_count = 6
  nameserver_ips = local.nameserver_ips
  internal_k8_domain = "k8.qa.cqdg"
}

module "k8_internal_domain" {
  source = "git::https://github.com/Ferlab-Ste-Justine/openstack-zonefile.git"
  domain = "k8.qa.cqdg"
  container = "dns"
  dns_server_name = "my.dns.com."
  a_records = concat([
    for master in module.kubernetes_cluster.masters: {
      prefix = "masters"
      ip = master.ip
    }
  ],
  [
    for worker in module.kubernetes_cluster.workers: {
      prefix = "workers"
      ip = worker.ip
    } 
  ])
}

resource "openstack_compute_floatingip_associate_v2" "k8_api_lb_ip" {
  floating_ip = openstack_networking_floatingip_v2.k8_api_lb_floating_ip.address
  instance_id = module.kubernetes_cluster.load_balancer.id
}