Skip to main content
Version: 1.28.6

EKSCluster Provider

The EKSCluster provider will automate the provisioning of an EKS cluster in AWS and will install the KFD modules on top of it.

This document provides a through guidance about every aspect of the configuration file. You can find the reference schema for this provider here.

Prerequisites​

Setup your AWS credentials by exporting the following environment variables:

export AWS_PROFILE=<YOUR_AWS_PROFILE_NAME>

If you don't have an AWS profile configured, you can create one by running the following command:

$ aws configure --profile <YOUR_AWS_PROFILE_NAME>
AWS Access Key ID [None]: <YOUR_AWS_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_AWS_SECRET_ACCESS_KEY>
Default region name [None]: <YOUR_AWS_REGION>
Default output format [None]: json

Configuration file creation​

Create a configuration file with:

furyctl create config --kind EKSCluster --version v1.28.6 --name <cluster-name>

This will create a new furyctl.yaml file with some default values. Read more to learn how to customize this file.

Note that the metadata.name parameter is the name of your cluster and it is used by furyctl to maintain the current status locally. It's not used by Kubernetes.

Spec section​

The spec section of the furyctl.yaml file contains parameters that configure the overall behavior of furyctl for this cluster. The available options are:

  • spec.toolsConfiguration.terraform.state.s3 section: this section defines where to store the terraform state files used by furyctl. The available parameters are:
    • bucketName: this value defines which bucket will be used to store all the states.
    • keyPrefix: this value defines which folder will be used to store all the states inside the bucket.
    • region: this value defines in which region the bucket is.
    • skipRegionValidation: this value defines if the region of the bucket should be validated or not by Terraform, useful when using a bucket in a recently added region.
  • spec.region: this value defines in which AWS region the cluster and all the related resources will be created.
  • spec.tags: this map defines which will be the common tags that will be added to all the resources created on AWS.

spec.infrastructure section​

This first section, infrastructure, defines the underlying network and eventual VPN bastions that will be provisioned.

If you already have a VPC, you can remove this key from the configuration file.

Available options are:

  • vpc.network: this key defines the VPC that will be created in AWS, with the following parameters:
    • cidr: the CIDR of the VPC that will be created. Example: 10.1.0.0/16.
    • subnnetsCidrs.private: list of CIDRs for the private subnets, where the nodes, the pods, and the private load balancers will be created.
    • subnnetsCidrs.public: list of CIDRs for the public subnets, where the public load balancers and the VPN servers will be created.
  • vpn: this object defines the creation of VPN bastions.
    • bucketNamePrefix: this value defines the prefix that will be used to create the bucket name where the VPN servers will store the states.
    • iamUserNameOverride: overrides the default IAM user name for the VPN.
    • vpcId: the VPC ID where the VPN servers will be created, required only if .spec.infrastructure.vpc is omitted.
    • instances: the number of instance to create, 0 to skip the creation.
    • port: the port used by the OpenVPN server.
    • instanceType: the size of the AWS ec2 instance, for example t3.micro.
    • diskSize: the size of the disk in GB.
    • operatorName: the username of the account to create in the bastion's operating system.
    • dhParamsBits: the dhParamsBits size used for the creation of the .pem file that will be used in the dh openvpn server.conf file.
    • vpnClientsSubnetCidr: The CIDR that will be used to assign IP addresses to the VPN clients when connected, for example 172.16.0.0/16.
    • ssh: ssh access settings. Available options:
      • githubUsersName: the github user name list that will be used to get the ssh public key that will be added as authorized key to the operatorName user
      • allowedFromCidrs: the CIDR enabled in the security group that can access the bastions in SSH. To allow acces from anywhere, insert - 0.0.0.0/0.

spec.kubernetes section​

This section describes how the EKS cluster will be created.

Inside the generated config file you will already find some guidance about the fields in the comments.

The required fields are:

  • vpcId: this value defines the VPC ID where the EKS cluster will be created, required only if .spec.infrastructure.vpc is omitted.
  • subnetIds: this list defines the subnet IDs where the EKS cluster will be created, required only if .spec.infrastructure.vpc is omitted.
  • nodeAllowedSshPublicKey: this key contains the ssh public key that can connect to the nodes via SSH using the ec2-user user.
  • nodePoolsLaunchKind: either launch_configurations, launch_templates or both. For new clusters use launch_templates, for existing cluster you'll need to migrate from launch_configurations to launch_templates using both as interim.
  • apiServer: this map defines the access to the Kubernetes API server.
    • privateAccess: enables private access to the Kubernetes API.
    • publicAccess: enables public access to the Kubernetes API.
    • privateAccessCidrs: list of source CIDRs to enable private access from, set to - 0.0.0.0/0 to enable access from every network.
    • publicAccessCidrs: list of source CIDRs to enable public access from, set to - 0.0.0.0/0 to enable access from every network.
  • nodePools: this array contains the definition of the nodepools in the cluster. The configuration for each group needs to have:
    • name: this is the name of the nodepool.
    • size.min and size.max: the max and min number of nodes in the nodepool autoscaling group.
    • instance: this map defines the characteristics of the instance that will be used in the node pool.
      • type: the instance type, for example t3.xlarge.
      • spot: boolean to define if the instance is a spot instance.
      • volumeSize: the instance disk size in GB.
      • volumeType: the instance disk type, for example gp2.
  • labels: Kubernetes labels that will be added to the nodes
  • taints: Kubernetes taints that will be added to the nodes
  • tags: AWS tags that will be added to the ASG and EC2 instances. The example provided by default shows the labels needed by cluster autoscaler.

spec.distribution section​

This section is dedicated to the configuration of the KFD core modules to be installed inside the cluster.

As usual, in the default configuration file you will find some comments to guide you editing this section.

common.nodeSelector and common.tolerations​

These parameters will be applied to every module. If you have labeled and/or tainted some nodes to host specifically KFD modules, you can put your values here to avoid placing KFD modules on all nodes.

spec.distribution.modules section​

Here you will configure the KFD core modules that will be installed in your cluster.

Ingress module​

Kubernetes Fury Ingress uses CNCF recommended, Cloud Native projects, such as Ingress NGINX an ingress controller using the well-known NGINX server as a URL path-based routing reverse proxy and load balancer, and cert-manager to automate the issuing and renewal of TLS certificates from various issuing sources.

The module also includes additional tools like Forecastle, a web-based global directory of all the services offered by your cluster.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.ingress object in the following way:

  • baseDomain: the base domain used for all the KFD ingresses, if in the nginx dual configuration, it should be the same as the .spec.distribution.modules.ingress.dns.private.name zone.
  • nginx.type: defines if the nginx should be configured as single or dual (internal + external, default) or none, with none no ingress controller will be deployed and also no ingress resource will be created.
    • dual: KFD will install two classes of the NGINX Ingress Controller, one being intended as internal use (for example by internal OPS teams) and the other for external exposure (for example towards final users). All KFD Ingresses will be using the internal class.
    • single: KFD will install a single NGINX Ingress Controller instance, with class nginx.
    • none: KFD will not install NGINX Ingress Controller. This also means that all web interfaces for other modules will not have an Ingress resource.
  • nginx.tls: the tls section defines how the tls for the ingresses should be managed. You can choose to provide the TLS certificate for the Ingress either via certManager or with a Secret resource. If you choose the provider to be secret, you must also specify the secret object with cert, key and ca contents.
  • certManager.clusterIssuer: this section lets you customize the CertManager instance that will be installed by KFD to provide certificates used by the cluster. Note that CertManager will always be installed, regardless of your choice about the Ingress, because it's used for all KFD certificates, not just the Ingress one.

The Forecastle instance will contain all Ingress resources created by KFD.

Logging module​

Kubernetes Fury Logging uses a collection of open source tools to provide the most resilient and robust logging stack for the cluster.

The central piece of the stack is the open source search engine opensearch, combined with its analytics and visualization platform opensearch-dashboards. The logs are collected using a node-level data collection and enrichment agent fluentbit, pushing it to the OpenSearch via fluentd. The fluentbit and fluentd stack is managed by Banzai Logging Operator. We are also providing an alternative to OpenSearch: loki.

All the components are deployed in the logging namespace in the cluster.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.logging object in the following way:

  • type: selects the logging stack. Choosing none will disable the centralized logging. Choosing opensearch will deploy and configure the Logging Operator and an OpenSearch cluster (can be single or triple for HA) where the logs will be stored. Choosing loki will use a distributed Grafana Loki instead of OpenSearch for storage. Choosing customOuput the Logging Operator will be deployed and installed but with no local storage, you will have to create the needed Outputs and ClusterOutputs to ship the logs to your desired storage.

  • minio.storageSize: if the backend type for the module is minio, it will install a minio instance inside the cluster. Here you specify the size for the bucket that's dedicated to the logging storage.

    info

    The minio instance will have 3 replicas with 2 PVCs each, so the storage size you specify will be multiplicated by a factor of 6.

  • minio.rootUser: here you can provide custom credentials for the Minio root user.

  • loki object: if you specified loki in type, this object lets you customize some aspects of it.

    • backend: the storage backend type for Loki. minio (default) will use an in-cluster MinIO deployment for object storage, externalEndpoint can be used to point to an external S3-compatible object storage instead of deploying an in-cluster MinIO.
    • externalEndpoint: if you specified externalEndpoint as backend, you need to specify this object to provide the needed credentials to your S3-compatible bucket.
    • resources: with this object, you can customize Loki's Pods resources.

  • opensearch object: if you specified opensearch in type, this object lets you customize some aspects of it.

    • type: you can choose between single and triple, which will install a single-instance or a HA triple-instance Opensearch respectively.
    • storageSize: here you can specify the size of the PVCs used by Opensearch's Pods.
    • resources: with this object, you can customize Opensearch's Pods resources.

  • customOutputs: when using the customOutputs logging type, you need to manually specify the spec of the several Output and ClusterOutputs that the Logging Operator expects to forward the logs collected by the pre-defined flows. Here you should follow the Output's spec object schema. You can find more info about the Flows provided by KFD here.

Monitoring module​

Kubernetes Fury Monitoring provides a fully-fledged monitoring stack for the Kubernetes Fury Distribution (KFD). This module extends and improves upon the Kube-Prometheus project.

This module is designed to give you full control and visibility over your cluster operations. Metrics from the cluster and the applications are collected and clean analytics are offered via a visualization platform, Grafana.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.monitoring object in the following way:

  • type: you can select between prometheus (default), prometheusAgent and mimir, or none to avoid installing the module.

    • prometheus will install Prometheus Operator and a preconfigured Prometheus instance, Alertmanager, a set of alert rules, exporters needed to monitor all the components of the cluster, Grafana and a series of dashboards to view the collected metrics, and more.
    • prometheusAgent will install Prometheus operator, an instance of Prometheus in Agent mode (no alerting, no queries, no storage), and all the exporters needed to get metrics for the status of the cluster and the workloads. Useful when having a centralized (remote) Prometheus where to ship the metrics and not storing them locally in the cluster.
    • mimir will install the same as the prometheus option, and in addition Grafana Mimir that allows for longer retention of metrics and the usage of Object Storage.

  • prometheus object: if you specified prometheus or mimir in type, this object lets you customize some aspects of the Prometheus instance, which is governed by Prometheus Operator.

    • remoteWrite: Set this option to ship the collected metrics to a remote Prometheus receiver. remoteWrite is an array of objects that allows configuring the remoteWrite options for Prometheus. The objects in the array follow the same schema as in the prometheus operator.
    • resources: with this object, you can customize Prometheus' Pods resources.
    • retentionSize: the retention size for the k8s Prometheus instance.
    • retentionTime: the retention time for the K8s Prometheus instance.
    • storageSize: the storage size for the k8s Prometheus instance.

  • alertmanager: if you specified prometheus or mimir in type, KFD will install Alertmanager inside the cluster. With this object, you can customize some aspects of it:

    • deadManSwitchWebhookUrl: The webhook url to send deadman switch monitoring, for example to use with healthchecks.io.
    • installDefaultRules: KFD comes with predefined Prometheus rules to generate alerts. If true, the default rules will be installed.
    • slackWebhookUrl: the Slack webhook url to send alerts.

  • grafana: if you specified prometheus or mimir in type, KFD will install Grafana inside the cluster. With this object, you can customize some aspects of it:

    • basicAuthIngress: setting this to true will deploy an additional grafana-basic-auth ingress protected with Grafana's basic auth instead of SSO. It's intended use is as a temporary ingress for when there are problems with the SSO login flow.

    • usersRoleAttributePath: JMESPath expression to retrieve the user's role. Example:

      usersRoleAttributePath: "contains(groups[*], 'beta') && 'Admin' || contains(groups[*], 'gamma') && 'Editor' || contains(groups[*], 'delta') && 'Viewer'

      More details in Grafana's documentation.

  • prometheusAgent object: if you specified prometheusAgent in type, this object lets you specify:

    • remoteWrite: you can specify a list of remote Prometheus receivers to ship metrics to.
    • resources: with this object, you can customize Prometheus' Pods resources.

  • mimir object: if you specified mimir in type, this object lets you customize some aspects of the installation:

    • backend: the storage backend type for Mimir. minio (default) will use an in-cluster MinIO deployment for object storage, externalEndpoint can be used to point to an external S3-compatible object storage instead of deploying an in-cluster MinIO.
    • externalEndpoint: if you specified externalEndpoint as backend, you need to specify this object to provide the needed credentials to your S3-compatible bucket.
    • retentionTime: The retention time for the mimir pods.

  • minio.storageSize: if you selected minio as storage backend for the module, it will install a minio instance inside the cluster. Here you specify the size for the bucket that's dedicated to the monitoring storage.

    info

    The minio instance will have 3 replicas with 2 PVCs each, so the storage size you specify will be multiplicated by a factor of 6.

  • minio.rootUser: here you can provide custom credentials for the Minio root user.

Tracing module​

Kubernetes Fury Tracing uses a collection of open source tools to provide the most resilient and robust tracing stack for the cluster.

The module contains the tempo tool from grafana.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.tracing object in the following way:

  • type: The type of tracing to use, either none or tempo.

  • tempo object: if you specified tempo in type, this object lets you customize some aspects of it.

    • backend: The storage backend type for Tempo. minio (default) will use an in-cluster MinIO deployment for object storage, externalEndpoint can be used to point to an external S3-compatible object storage instead of deploying an in-cluster MinIO.
    • retentionTime: the retention time for the tempo pods.
    • externalEndpoint: if you specified externalEndpoint as backend, you need to specify this object to provide the needed credentials to your S3-compatible bucket.

  • minio.storageSize: if you selected minio as storage backend for the module, it will install a minio instance inside the cluster. Here you specify the size for the bucket that's dedicated to the monitoring storage.

    info

    The minio instance will have 3 replicas with 2 PVCs each, so the storage size you specify will be multiplicated by a factor of 6.

OPA module​

The Kubernetes API server provides a mechanism to review every request that is made (object creation, modification, or deletion). To use this mechanism the API server allows us to create a Validating Admission Webhook that, as the name says, will validate every request and let the API server know if the request is allowed or not based on some logic (policy).

Kubernetes Fury OPA module is based on OPA Gatekeeper and Kyverno, two popular open-source Kubernetes-native policy engines that runs as a Validating Admission Webhook. It allows writing custom constraints (policies) and enforcing them at runtime.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.policy object in the following way:

  • type: the type of security to use, either none, gatekeeper (default) or kyverno.
  • gatekeeper object: if you specified gatekeeper in type, this object lets you customize some aspects of it.
    • additionalExcludedNamespaces: this parameter adds namespaces to Gatekeeper's exemption list, so it will not enforce the constraints on them.
    • installDefaultPolicies: if true, the default policies will be installed. You can find more info about the default policies here
    • enforcementAction: The enforcement action to use for the gatekeeper module. Allowed values: deny, dryrun, warn.
  • kyverno object: if you specified kyverno in type, this object lets you customize some aspects of it.
    • additionalExcludedNamespaces: this parameter adds namespaces to Kyverno's exemption list, so it will not enforce the constraints on them.
    • installDefaultPolicies: if true, the default policies will be installed. You can find more info about the default policies here
    • validationFailureAction: The validation failure action to use for the kyverno module. Allowed values: Audit, Enforce.

DR module​

Kubernetes Fury DR module is based on Velero and Velero Node Agent.

Velero allows you to:

  • backup your cluster
  • restore your cluster in case of problems
  • migrate cluster resources to other clusters
  • replicate your production environment to development and testing environment.

Together with Velero, Velero Node Agent allows you to:

  • backup Kubernetes volumes
  • restore Kubernetes volumes

The module contains also velero plugins to natively integrate with Velero with different cloud providers and use cloud provider's volumes as the storage backend.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.dr object in the following way:

  • type: The type of the DR, must be none or eks.
  • velero object: if you specified eks in type, this object lets you customize some aspects of it.
    • backend: The storage backend type for Velero. minio (default) will use an in-cluster MinIO deployment for object storage, externalEndpoint can be used to point to an external S3-compatible object storage instead of deploying an in-cluster MinIO.
    • externalEndpoint: if you specified externalEndpoint as backend, you need to specify this object to provide the needed credentials to your S3-compatible bucket.
    • schedules object: Configuration for Velero's backup schedules.
      • ttl: The Time To Live (TTL) of the backups created by the backup schedules (default 720h0m0s, 30 days). Notice that changing this value will affect only newly created backups, prior backups will keep the old TTL.
      • install: Whether to install or not the default manifests and full backups schedules. Default is true.
      • cron.full and cron.manifests: The cron expression for the full and manifests backup schedules (default 0 1 * * * for full, */15 * * * * for manifests).

Auth module​

Kubernetes Fury Auth uses CNCF recommended, Cloud Native projects, such as the Dex identity provider, and Pomerium as an identity-aware proxy to enable secure access to internal applications.

info

You can find more info about this module here.

You can customize the module using the spec.distribution.modules.auth object in the following way:

  • baseDomain: the base domain used for all the auth ingresses, if in the nginx dual configuration, it should be the same as the .spec.distribution.modules.ingress.dns.public.name zone.
  • provider.type: the authentication type used for the infrastructure ingresses (all the ingress for the distribution). It can be none (default), basicAuth, sso.
  • provider.basicAuth.username and provider.basicAuth.password: if you specified basicAuth in provider.type, these two parameter let you customize the credentials to be used.
  • dex object: if you specified sso in provider.type, this object lets you customize some aspects of the Dex installation.
    • expiry.idTokens and expiry.signingKeys: Dex ID tokens and signing key expiration time duration (default 24h and 6h respectively).
    • additionalStaticClients: The additional static clients for dex, here you can find some info about the available options.
    • connectors: The connectors for dex, see here for more info about the available options.
  • pomerium object: if you specified sso in provider.type, this object lets you customize some aspects of the Pomerium installation.

    • secrets: this object contains four parameters that need to be filled when using Pomerium:

      • COOKIE_SECRET: Cookie Secret is the secret used to encrypt and sign session cookies. To generate a random key, run the following command: head -c32 /dev/urandom | base64

      • IDP_CLIENT_SECRET: Identity Provider Client Secret is the OAuth 2.0 Secret Identifier. When auth type is SSO, this value will be the secret used to authenticate Pomerium with Dex, use a strong random value.

      • SHARED_SECRET: Shared Secret is the base64-encoded, 256-bit key used to mutually authenticate requests between Pomerium services. It's critical that secret keys are random, and stored safely. To generate a key, run the following command: head -c32 /dev/urandom | base64

      • SIGNING_KEY: Signing Key is the base64 representation of one or more PEM-encoded private keys used to sign a user's attestation JWT, which can be consumed by upstream applications to pass along identifying user information like username, id, and groups.

        To generate a P-256 (ES256) signing key:

        openssl ecparam -genkey -name prime256v1 -noout -out ec_private.pem      
        # careful! this will output your private key in terminal
        cat ec_private.pem | base64

    • routes: additional routes configuration for Pomerium. Follows Pomerium's route format.

spec.distribution.customPatches section​

This section lets you customize the manifests deployed by KFD. This is intended to apply advanced customizations where the provided configuration fields do not suffice.