GKE Cluster

furyctl gke cluster provisioner

7 minute read

This provisioner deploys an GKE Kubernetes Cluster. It enables the creation and the management of all the Kubernetes infrastructural components with a simple yaml file. It provides some nice features:

  • Private GKE Kubernetes Control Plane.
    • Requires to have connectivity from the furyctl to the Subnet where the cluster will be placed.
  • Set an operator SSH key to enabling the troubleshooting of issues in the cluster nodes.
  • Configures multiple node pools
    • Taints
    • Labels
    • Tags
    • Lift and Shift updates
    • Many more…
  • Seamless cluster (control plane) updates.
  • Many more…

Configuration

kind: Cluster
metadata:
  name: # The name of the resources. Used to name the control plane and other resources...
provisioner: gke # set to `gke` to use this provisioner
spec:
  version: # GKE Control plane version. Example: 1.18.12-gke.1206
  network: # Network Identificator where the cluster contorl-plane and workers nodes will be placed
  networkProjectID: # OPTIONAL. The project ID of the shared network (enable shared vpc support)
  controlPlaneCIDR: # OPTIONAL. The IP range in CIDR notation to use for the hosted master network. Default value: 10.0.0.0/28
  additionalFirewallRules: # OPTIONAL. Create additional firewall rules. Default: true
  additionalClusterFirewallRules: # OPTIONAL. Create additional firewall rules (Upstream GKE module). Default: false.
  disalbeDefaultSNAT: # OPTIONAL. Whether to disable the default SNAT to support the private use of public IP addresses. Default: false
  subnetworks: # Subnet List (subnet ids).
  # - 'subnet-id0 # Identificator of the subnets. Index 0: Cluster Subnet'
  # - 'subnet-id1 # Identificator of the subnets. Index 1: Pod Subnet'
  # - 'subnet-id2 # Identificator of the subnets. Index 2: Service Subnet'
  dmzCIDRRange: # Secure cidr range with access to the control plane. It must be a CIDR where a bastion/vpn is available.
  sshPublicKey: # Public key used as authorized keys to the nodes
  tags: {} # Map of strings containing tags for every resource in the cluster
  nodePools: # List of object. Contains node pool definitions
    - name: # Name of the node pool
      version: # Version of the node pool. If null, cluster version will be used.
      minSize: # Min Number of nodes in the node pool.
      maxSize: # Max Number of nodes in the node pool.
      subnetworks: # availability zones (example: us-central1-a) where to place the nodes. Useful to don't create them on all zones.
      instanceType: # Instance type of the node pool. Example n1-standard-1
      maxPods: # Max number of pods per node. If null, gcp will auto generate it based on network interfaces.
      volumeSize: # Volume size of the nodes.
      labels: {} # Map of strings. It will be available as node labels in the Kubernetes API. Useful while scheduling workloads.
      taints: [] # Taint list. Useful while scheduling workloads
      tags: {} # Map of strings containing tags for this node pool. Useful to enable autoscalling in certain node pools.

Important notes

To properly deploy the GKE cluster using this provisioner, you have to solve first the connectivity problem.

This provisioner requires interacting with the GKE cluster control plane. As it is a private endpoint, the furyctl has to be able to reach the private GKE control plane via VPN or running the cli from a bastion host in the cluster network.

If you didn’t solve it, take a look to the gcp bootstrap provisioner. It fixes the connectivity issue before deploying the GKE cluster.

Resources

This provisioner creates and configures:

  • GKE: GKE Control plane in the private subnets.
    • Worker Nodes: Based on the configuration of the node pools.
      • Firewall Rules: Additional firewall rules enabling webhook to the control plane and SSH access to the nodes.

Diagram

source: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture?hl=es-419

Requirements

This provisioner requires to have previously configured:

  • GCP Credentials in form of environment variables:
    • One of: GOOGLE_CREDENTIALS | GOOGLE_CLOUD_KEYFILE_JSON | GCLOUD_KEYFILE_JSON
    • One of: GOOGLE_PROJECT | GOOGLE_CLOUD_PROJECT | GCLOUD_PROJECT | CLOUDSDK_CORE_PROJECT
    • One of: GOOGLE_REGION | GCLOUD_REGION | CLOUDSDK_COMPUTE_REGION
    • Enough permissions to deploy the above resources.
  • furyclt.
  • kubectl.
  • wget and/or curl.
  • gcloud
  • A configuration file with all the values in place. Please think it very carefully.
  • Network connetivity to the target network and subnets.

Example execution

In the following lines, you will find an example execution to deploy the bootstrap infrastructure.

First, ensure you pass the requirements. Then create a configuration file with the structure described above.

First, create a new directory:

$ mkdir demo
$ cd demo

Take this cluster.yml file as an example. We recommend you to use a non-default backend configuration:

kind: Cluster
metadata:
  name: furyctl
provisioner: gke
spec:
  version: 1.18.16-gke.502
  network: furyctl
  subnetworks:
    - furyctl-cluster-subnet
    - furyctl-cluster-pod-subnet
    - furyctl-cluster-service-subnet
  dmzCIDRRange: 10.0.0.0/8
  sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCefFo9ASM8grncpLpJr+DAeGzTtoIaxnqSqrPeSWlCyManFz5M/DDkbnql8PdrENFU28blZyIxu93d5U0RhXZumXk1utpe0L/9UtImnOGG6/dKv9fV9vcJH45XdD3rCV21ZMG1nuhxlN0DftcuUubt/VcHXflBGaLrs18DrMuHVIbyb5WO4wQ9Od/SoJZyR6CZmIEqag6ADx4aFcdsUwK1Cpc51LhPbkdXGGjipiwP45q0I6/Brjxv/Kia1e+RmIRHiltsVBdKKTL9hqu9esbAod9I5BkBtbB5bmhQUVFZehi+d/opPvsIszE/coW5r/g/EVf9zZswebFPcsNr85+x
  nodePools:
    - name: my-node-pool
      minSize: 1
      maxSize: 1
      volumeSize: 50
      instanceType: n1-standard-1
$ ls
cluster.yml

Init the cluster project

$ furyctl cluster init --reset
WARN[0000] Cleaning up the workdir                      
WARN[0000] Removing ./cluster directory
WARN[0000] could not find terraform executable
INFO[0003] [INFO] running Terraform command: cluster/bin/terraform init -no-color -force-copy -input=false -lock-timeout=0s -backend=true -get=true -get-plugins=true -lock=true -upgrade=false -verify-plugins=true
[GKE] Fury

This provisioner creates a battle-tested Google Cloud GKE Kubernetes Cluster
with a private and production-grade setup.

Requires to connect to a VPN server to deploy the cluster from this computer.
Use a bastion host (inside the GKE VPC) as an alternative method to deploy the cluster.

The provisioner requires the following software installed:
- /bin/sh
- wget or curl
- gcloud
- kubectl

[FURYCTL]

Init phase completed.

Project directory: ./cluster
Terraform logs: ./cluster/logs/terraform.logs

Everything ready to create the infrastructure; execute:

$ furyctl cluster apply

Once completed, a new directory cluster is available inside the current directory demo.

$ ls
cluster   cluster.yml

It contains all the terraform project and configuration to properly manage the cluster infrastructure.

$ tree cluster/
cluster/
├── backend.tf
├── bin
│   └── terraform
├── configuration
├── gke.tfvars
├── logs
│   └── terraform.logs
├── main.tf
├── output
├── output.tf
├── secrets
└── variables.tf

5 directories, 7 files

It didn’t create any infrastructure component, continue the example to deploy it.

Deploy the cluster project

NOTE: It can take up to 20 minutes.

$ furyctl cluster apply
ERRO[0000] Directory already exists
WARN[0000] error while initializing project subdirectories: Directory already exists
INFO[0000] terraform is up to date
INFO[0000] [INFO] running Terraform command: cluster/bin/terraform init -no-color -force-copy -input=false -lock-timeout=0s -backend=true -get=true -get-plugins=true -lock=true -upgrade=false -verify-plugins=true
INFO[0003] Updating GKE project
INFO[0003] [INFO] running Terraform command: cluster/bin/terraform fmt -no-color -write=true -list=false -diff=false ./cluster/gke.tfvars
⣾ Applying terraform project INFO[0003] [INFO] running Terraform command: cluster/bin/terraform apply -no-color -auto-approve -input=false -var-file=./cluster/gke.tfvars -lock=true -parallelism=10 -refresh=true
⣻ Applying terraform project INFO[0118] GKE Updated
INFO[0118] Gathering output file as json
INFO[0118] [INFO] running Terraform command: cluster/bin/terraform output -no-color -json
INFO[0120] Gathering output file as json
INFO[0120] [INFO] running Terraform command: cluster/bin/terraform output -no-color -json
INFO[0122] [INFO] running Terraform command: cluster/bin/terraform output -no-color -json
[GKE] Fury

All the cluster components are up to date.
GKE Kubernetes cluster ready.

GKE Cluster Endpoint: https://10.0.0.2
SSH Operator Name: ubuntu

Use the ssh ubuntu username to access the GKE instances with the configured SSH key.
Discover the instances by running

$ kubectl get nodes

Then access by running:

$ ssh ubuntu@node-name-reported-by-kubectl-get-nodes


[FURYCTL]
Apply phase completed. The Kubernetes Cluster is up to date.

Project directory: ./cluster
Terraform logs: ./cluster/logs/terraform.logs
Output file: ./cluster/output/output.json
Kubernetes configuration file: ./cluster/secrets/kubeconfig

Use it by running:
$ export KUBECONFIG=./cluster/secrets/kubeconfig
$ kubectl get nodes

Everything is up to date.
Ready to apply or destroy the infrastructure; execute:

$ furyctl cluster apply
or
$ furyctl cluster destroy

Once completed, everything is ready to start using the EKS cluster along with other cluster components. In the output message there is enough information to start using the new infrastructure:

GKE Cluster Endpoint: https://10.0.0.2
SSH Operator Name: ubuntu
$ export KUBECONFIG=./cluster/secrets/kubeconfig
$ kubectl get nodes
NAME                                     STATUS   ROLES    AGE   VERSION
gke-furyctl-my-node-pool-80c5e965-wkv2   Ready    <none>   67s   v1.18.15-gke.800
gke-furyctl-my-node-pool-c57df990-sh79   Ready    <none>   60s   v1.18.15-gke.800
gke-furyctl-my-node-pool-d1272dc7-vh94   Ready    <none>   59s   v1.18.15-gke.800

Modify the configuration

As an example modification of the stack, increase the number of minimum and maximum nodes in the my-node-pool node pool from 1 to 2. Modify the cluster.yml file adding the right number of nodes.

kind: Cluster
metadata:
  name: furyctl
provisioner: gke
spec:
  version: 1.18.15-gke.800
  network: furyctl
  subnetworks:
    - furyctl-cluster-subnet
    - furyctl-cluster-pod-subnet
    - furyctl-cluster-service-subnet
  dmzCIDRRange: 10.0.0.0/8
  sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCefFo9ASM8grncpLpJr+DAeGzTtoIaxnqSqrPeSWlCyManFz5M/DDkbnql8PdrENFU28blZyIxu93d5U0RhXZumXk1utpe0L/9UtImnOGG6/dKv9fV9vcJH45XdD3rCV21ZMG1nuhxlN0DftcuUubt/VcHXflBGaLrs18DrMuHVIbyb5WO4wQ9Od/SoJZyR6CZmIEqag6ADx4aFcdsUwK1Cpc51LhPbkdXGGjipiwP45q0I6/Brjxv/Kia1e+RmIRHiltsVBdKKTL9hqu9esbAod9I5BkBtbB5bmhQUVFZehi+d/opPvsIszE/coW5r/g/EVf9zZswebFPcsNr85+x
  nodePools:
    - name: my-node-pool
      minSize: 2 # Changed
      maxSize: 2 # Changed
      volumeSize: 50
      instanceType: n1-standard-1

Then run again the furyctl cluster apply command.

$ furyctl cluster apply

The new node can take up to five minutes to be appear in the cluster.

Destroy the cluster project

If you need to destroy the cluster project, first ensure there is nothing blocking the destroy of this infrastructure.

Then run:

$ furyctl cluster destroy
  Are you sure you want to destroy the cluster?
  Write 'yes' to continue
yes
ERRO[0001] Directory already exists
WARN[0001] error while initializing project subdirectories: Directory already exists
⣽ Initializing the terraform executor INFO[0001] terraform is up to date
INFO[0001] [INFO] running Terraform command: cluster/bin/terraform init -no-color -force-copy -input=false -lock-timeout=0s -backend=true -get=true -get-plugins=true -lock=true -upgrade=false -verify-plugins=true
INFO[0004] Destroying GKE project
INFO[0004] [INFO] running Terraform command: cluster/bin/terraform fmt -no-color -write=true -list=false -diff=false ./cluster/gke.tfvars
⣷ Destroying terraform project INFO[0004] [INFO] running Terraform command: cluster/bin/terraform destroy -no-color -auto-approve -input=false -lock-timeout=0s -var-file=./cluster/gke.tfvars -lock=true -parallelism=10 -refresh=true
⢿ Destroying terraform project INFO[0656] GKE destroyed
[GKE] Fury
All cluster components were destroyed.
GKE control plane and workers went away.

Had problems, contact us at sales@sighup.io.

[FURYCTL]
Destroy phase completed.

Project directory: ./cluster
Terraform logs: ./cluster/logs/terraform.logs