Skip to main content
Version: Next

Kubernetes Fury Ingress

Kubernetes Fury Ingress provides Ingress Controllers to expose services and TLS certificate management solutions for the Kubernetes Fury Distribution (KFD).

If you are new to KFD please refer to the official documentation on how to get started with KFD.

Overview

Kubernetes Fury Ingress uses CNCF recommended, Cloud Native projects, such as Ingress NGINX an ingress controller using the well-known NGINX server as a URL path-based routing reverse proxy and load balancer, and cert-manager to automate the issuing and renewal of TLS certificates from various issuing sources.

The module also includes additional tools like Forecastle, a web-based global directory of all the services offered by your cluster.

Architecture

The reference architecture used to deploy the Fury Kubernetes Ingress Module is shown in the following figure:

Ingress Architecture

  • The traffic from end users arrives first at a Load Balancer that distributes the traffic between the available Ingress Controllers (usually, one for each availability zone).
  • Once the traffic reaches the Ingress Controller, the Ingress proxies the traffic to the Kubernetes service based on the URL path of the request.
  • The service is a Kubernetes abstraction that makes the traffic arrive at the pods where the actual application is running, usually using iptables rules.

For more information, please refer to Kubernetes Ingress offiicial documentation

Packages

Kubernetes Fury Ingress provides the following packages:

PackageVersionDescription
nginxv1.5.1The NGINX Ingress Controller for Kubernetes provides delivery services for Kubernetes applications.
dual-nginxv1.5.1It deploys two identical NGINX ingress controllers but with two different scopes: public/external and private/internal.
cert-managerv1.11.0cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources.
external-dnsv0.13.2external-dns allows you to manage DNS records natively from Kubernetes.
forecastlev1.0.119Forecastle gives you access to a control panel where you can see your ingresses and access them on Kubernetes.
aws-cert-manager-Terraform modules for managing IAM permissions on AWS for cert-manager
aws-external-dns-Terraform modules for managing IAM permissions on AWS for external-dns

Compatibility

Kubernetes VersionCompatibilityNotes
1.23.xNo known issues
1.24.xNo known issues
1.25.xNo known issues

Check the compatibility matrix for additional information on previous releases of the module.

Usage

Prerequisites

ToolVersionDescription
furyctl>=0.6.0The recommended tool to download and manage KFD modules and their packages. To learn more about furyctl read the official documentation.
kustomize>=3.5.3Packages are customized using kustomize. To learn how to create your customization layer with kustomize, please refer to the repository.

Single vs Dual Controller

As the first step, you should choose what type of ingress controller you want to use in your cluster. Kubernetes Fury Ingress provides two types, Single and Dual Ingress Controller.

The Single Controller Package deploys a single class NGINX Ingress Controller that serves all the internal (private) and external (public) traffic.

The Dual Controller Package creates two NGINX Ingress Controller classes, the internal-ingress and the external-ingress classes:

  • The internal-ingress class is in charge of serving traffic inside the cluster's network, like users accessing via VPN to internal services, for example, application's admin panels or other resources that you don't want to expose to the Internet.

  • The external-ingress class serves traffic for the applications exposed to the outside of the cluster (e.g., the frontend application to end-users).

Default Configuration

For both Single and Dual NGINX the Kubernetes Fury Ingress module has the following default configuration:

  • Maximum allowed size of the client request body: 10m
  • HTTP status code used in redirects: 301
  • Metrics are scraped by Prometheus every 10s
  • Validating Admission webhook that validates an ingress definition does not break NGINX configuration.

Additionally, the following Prometheus alerts are set up by default:

ParameterDescriptionSeverityInterval
NginxIngressDownThis alert fires if Prometheus target discovery was not able to reach ingress-nginx-metrics in the last 15 minutes.critical15m
NginxIngressFailureRateThis alert fires if the failure rate (the rate of 5xx responses) measured on a time window of 2 minutes was higher than 10% in the last 10 minutes.critical10m
NginxIngressFailedReloadThis alert fires if the ingress' configuration reload failed in the last 10 minutes.warning10m
NginxIngressLatencyTooHighThis alert fires if the ingress 99th percentile latency was more than 5 seconds in the last 10 minutes.warning10m
NginxIngressLatencyTooHighThis alert fires if the ingress 99th percentile latency was more than 10 seconds in the last 10 minutes.critical10m
NginxIngressCertificateExpirationThis alert fires if the certificate for a given host is expiring in less than 7 days.warning
NginxIngressCertificateExpirationThis alert fires if the certificate for a given host is expiring in less than 1 day.critical

Deployment

Certificates automation with cert-manager

Kubernetes Fury Ingress cert-manager package is the defacto tool to manage certificates in Kubernetes. It manages the issuing and automatic renewal of certificates for the domains served by the Ingress Controller.

The package's default configuration includes:

  • Let's Encrypt as the default certificates issuer, both staging and production.
  • ClusterIssuer as the default issuer kind

To deploy the cert-manager package:

  1. Add the package to your bases inside the Furyfile.yml:
bases:
- name: ingress/dual-nginx
version: "v1.14.1"
- name: ingress/cert-manager
version: "v1.14.1"
  1. Execute furyctl vendor -H to download the packages

  2. Inspect the download packages under ./vendor/katalog/ingress/cert-manager.

  3. Define a kustomization.yaml that includes the ./vendor/katalog/ingress/cert-manager directory as resource.

resources:
- ./vendor/katalog/ingress/cert-manager

For the dual-nginx you will need to patch the ClusterIssuer resource with the right ingress class:

---
patchesJson6902:
- target:
group: cert-manager.io
version: v1
kind: ClusterIssuer
name: letsencrypt-staging
path: patches/dual-nginx.yml
- target:
group: cert-manager.io
version: v1
kind: ClusterIssuer
name: letsencrypt-prod
path: patches/dual-nginx.yml

and in the patches/dual-nginx.yml:

---
- op: "replace"
path: "/spec/acme/solvers/0/http01/ingress/class"
value: "external"
  1. Finally, execute the following command to deploy the package:
kustomize build . | kubectl apply -f -

NGINX Ingress Controller

⚠️ You'll need to have deployed cert-manager first, the validating webhook needs to have TLS communication with the API server.

  1. Once you selected the type of ingress you want to deploy (nginx or dual-nginx) the next step is to specify this in a Furyfile.yml:

Single Ingress:

bases:
- name: ingress/nginx
version: "v1.14.1"

Dual Ingress:

dual-nginx depends on the nginx package, so we need to download both of them.

bases:
- name: ingress/nginx
version: "v1.14.1"
- name: ingress/dual-nginx
version: "v1.14.1"

See furyctl documentation for additional details about Furyfile.yml format.

  1. Execute furyctl vendor -H to download the packages

  2. Inspect the download packages under ./vendor/katalog/ingress/.

  3. Define a kustomization.yaml that includes the ./vendor/katalog/ingress directory as resource.

resources:
- ./vendor/katalog/ingress
  1. Apply the necessary patches. You can find a list of common customization here.

  2. To deploy the packages to your cluster, execute:

kustomize build . | kubectl apply -f -

Your are now ready to expose your applications using Kubernetes Ingress objects.

Common customizations

nginx package is deployed by default as a DaemonSet, meaning that it will deploy 1 ingress-controller pod on every worker node.

This is probably NOT what you want, standard Fury clusters have at least 1 infra node (nodes that are dedicated to run Fury infrastructural components, like Prometheus, elasticsearch, and the ingress controllers).

If your cluster has infra nodes you should patch the daemonset adding the NodeSelector for the infra nodes to the Ingress DaemonSet. You can do this using the following kustomize patch:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller-external
spec:
template:
spec:
nodeSelector:
node-kind.sighup.io/infra: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller-internal
spec:
template:
spec:
nodeSelector:
node-kind.sighup.io/infra: ""

If you don't have infra nodes and you don't want to run ingress-controllers on all your worker nodes, you should probably label some nodes and adjust the previous NodeSelector accordingly.

Applications directory with Forecastle

Forecastle list all the annotated ingress (applications) that exists in your cluster with an icon grouped by namesapce, in a nice web UI. It lets you search, personalize the header for the landing page (title and colors), it lets you list custom ingress and add more details to each entry.

Use Forecastle as your cluster entry point to discover the running applications easily.

To deploy the forecastle package:

  1. Add the package to your bases inside the Furyfile.yml:
bases:
- name: ingress/dual-nginx
version: "v1.14.1"
- name: ingress/cert-manager
version: "v1.14.1"
- name: ingress/forecastle
version: "v1.14.1"
  1. Execute furyctl vendor -H to download the packages

  2. Inspect the download packages under ./vendor/katalog/ingress/forecastle.

  3. Define a kustomization.yaml that includes the ./vendor/katalog/ingress/forecastle directory as resource.

resources:
- ./vendor/katalog/ingress/forecastle
  1. Finally, execute the following command to deploy the package:
kustomize build . | kubectl apply -f -
Basic usage

Forecastle looks for specific annotations on ingresses objects.

Add the following annotations to your ingresses to be discovered by Forecastle:

AnnotationDescriptionRequired
forecastle.stakater.com/exposeAdd this with value true to the ingress of the app you want to show in Forecastletrue
forecastle.stakater.com/iconIcon/Image URL of the application; An icons/logos/images collection repo Iconsfalse
forecastle.stakater.com/appNameA custom name for your application. Default is the name of the ingressfalse
forecastle.stakater.com/groupA custom group name. Use if you want the application to show in a different group than the namespace it belongs tofalse
forecastle.stakater.com/instanceA comma-separated list of name/s of the forecastle instance/s where you want this application to appear. Use when you have multiple forecastle dashboardsfalse
forecastle.stakater.com/urlA URL for the forecastle app (This will override the ingress URL). It must begin with a scheme i.e. http:// or https://false
forecastle.stakater.com/propertiesA comma separate list of key-value pairs for the properties. This will appear as an expandable list for the appfalse
forecastle.stakater.com/network-restrictedSpecify whether the application is network restricted or not (true or false)false

See Forecastle official repository for more details.