This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Ingress

1 - Ingress with Service Mesh

HTTP ingress implemented by the FSM Ingress controller

FSM can optionally use the FSM ingress controller and Pipy-based edge proxies to route external traffic to the Service Mesh backend. This guide demonstrates how to configure HTTP ingress for services managed by the FSM service mesh.

Prerequisites

  • Kubernetes cluster version v1.19.0 or higher.
  • Interact with the API server using kubectl.
  • FSM CLI installed.
  • FSM Ingress Controller installed followed by installation document

Demo

Assume that we have FSM installed under the fsm-system namespace, and named with fsm.

export FSM_NAMESPACE=fsm-system # Replace fsm-system with the namespace where FSM will be installed
export FSM_MESH_NAME=fsm # Replace fsm with the desired FSM mesh name

Save the external IP address and port of the entry gateway, which will be used later to test access to the backend application.

export ingress_host="$(kubectl -n "$FSM_NAMESPACE" get service fsm-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
export ingress_port="$(kubectl -n "$FSM_NAMESPACE" get service fsm-ingress -o jsonpath='{.spec.ports[?(@.name=="http")].port}')"

The next step is to deploy the sample httpbin service.

# Create a namespace
kubectl create ns httpbin

# Add the namespace to the mesh
fsm namespace add httpbin

# Deploy the application
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Ensure that the httpbin service and pod are up and running properly by

kubectl get pods,svc -n httpbin                                                                                                                  default/fsm-system ⎈
NAME                           READY   STATUS            RESTARTS   AGE
pod/httpbin-5c4bbfb664-xsk7j   0/2     PodInitializing   0          29s

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/httpbin   ClusterIP   10.43.83.102   <none>        14001/TCP   30s

HTTP Ingress

Next, create the necessary HTTPProxy and IngressBackend configurations to allow external clients to access port 14001 of the httpbin service under the httpbin namespace. Because TLS is not used, the link from the fsm entry gateway to the httpbin backend pod is not encrypted.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
---
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$FSM_NAMESPACE"
    name: fsm-ingress
EOF

Now we expect external clients to have access to the httpbin service, with the HOST request header of the HTTP request being httpbin.org.

curl -sI http://"$ingress_host":"$ingress_port"/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Tue, 05 Jul 2022 07:34:11 GMT
content-type: application/json
content-length: 241
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

2 - Ingress Controller - Basics

Using FSM Ingress controller to serve HTTP/HTTPS traffic

This guide demonstrate how to serve HTTP and HTTPs traffic via FSM Ingress controller.

Prerequisites

  • Kubernetes cluster version v1.19.0 or higher.
  • Interact with the API server using kubectl.
  • FSM CLI installed.
  • FSM Ingress Controller installed followed by installation document

Sample Application

The example application used here provides access through both HTTP at port 8000 and HTTPS at port 8443, with the following URI:

  • / returns a simple HTML page
  • /hi returns a 200 response with string Hi, there!
  • /api/private returns a 401 response with string Staff only

To provide HTTPS, a CA certificate and server certificate need to be issued for the application first.

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \
   -key ca-key.pem \
   -out ca-cert.pem \
   -subj '/CN=flomesh.io'

openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out server.csr -subj '/CN=example.com'
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -days 365

Before deploying the sample service, first let’s create a secret to save the certificate and key in the secret and mount it in the application pod.

kubectl create namespace httpbin
# mount self-signed cert to sample app pod via secret
kubectl create secret generic -n httpbin server-cert \
  --from-file=./server-cert.pem \
  --from-file=./server-key.pem

kubectl apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - port: 8443
    name: https
  - port: 8000
    name: http
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
        - name: pipy
          image: addozhang/httpbin:latest
          env:
            - name: PIPY_CONFIG_FILE
              value: /etc/pipy/tutorial/gateway/main.js
          ports:
            - containerPort: 8443
            - containerPort: 8000
          volumeMounts:
          - name: cert
            mountPath: "/etc/pipy/tutorial/gateway/secret"
            readOnly: true
      volumes:
      - name: cert
        secret:
          secretName: server-cert
EOF

Basic configurations

HTTP Protocol

In the following example, an Ingress resource is defined that routes requests with host example.com and path /get and / to the back-end service httpbin listening at port 8000.

Note that the Ingress resource and the back-end service should belong to the same namespace.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Exact
          backend:
            service:
              name: httpbin
              port:
                number: 8000
        - path: /hi
          pathType: Exact
          backend:
            service:
              name: httpbin
              port:
                number: 8000

Explanation of some of fields:

  • metadata.name field defines the resource name of the Ingress.
  • spec.ingressClassName field is used to specify the implementation of the entrance controller. Ingressclass is the name defined by the implementation of each entrance controller, and here we use pipy. The installed entrance controllers can be viewed through kubectl get ingressclass.
  • spec.rules field is used to define the routing resource.
    • host field defines the hostname example.com
    • paths field defines two path rules: the request matching the path /, and the uri /hi
    • backend field defines the backend service httpbin and port 8000 used to handle the path rule.

By viewing the Ingress Controller Service, you can see that its type is LoadBalancer, and its external address is 10.0.0.12, which is exactly the node’s IP address.

kubectl get svc -n fsm-system -l app=fsm-ingress
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
fsm-ingress   LoadBalancer   10.43.243.124   10.0.2.4      80:30508/TCP   16h

Applying the Ingress configuration above, when accessing the uri /hi and / endpoints of the httpbin service, we can use the node’s IP address and port 80.

export HOST_IP=$(ip addr show eth0 | grep 'inet ' | awk '{print $2}' | cut -d/ -f1)
curl http://example.com/hi --connect-to example.com:80:$HOST_IP:80
Hi, there!

curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
<!DOCTYPE html>
<html>
  <head>
    <title>Hi, Pipy!</title>
  </head>
  <body>
    <h1>Hi, Pipy!</h1>
    <p>This is a web page served from Pipy.</p>
  </body>
</html>

HTTPS protocol

This example shows how to configure an ingress controller to support HTTPS access. By default, the FSM Ingress does not enable TLS ingress, and you need to turn on the TLS ingress functionality by using the parameter --set fsm.ingress.tls.enabled=true during installation.

Or execute the command below to enable ingress TLS after installed.

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"ingress":{"tls":{"enabled":true}}}}' --type=merge

The Ingress resource in the following example configures the url https://example.com to access ingress.

  • spec.tls is the exclusive field for TLS configuration and can configure multiple HTTPS ingresses.
  • hosts field is used to configure SNI and can configure multiple SNIs. Here, example.com is used, and wildcard *.example.com is also supported.
  • secretName field is used to specify the Secret that stores the certificate and key. Note that the Ingress resource and Secret should belong to the same namespace.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8000
  tls:
  - hosts:
    - example.com
    secretName: ingress-cert

Issue TLS certificate

openssl genrsa -out ingress-key.pem 2048
openssl req -new -key ingress-key.pem -out ingress.csr -subj '/CN=example.com'
openssl x509 -req -in ingress.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out ingress-cert.pem -days 365

Create a Secret using the certificate and key.

kubectl create secret generic -n httpbin ingress-cert \
  --from-file=tls.crt=./ingress-cert.pem --from-file=tls.key=ingress-key.pem --from-file=ca.crt=./ca-cert.pem

Apply above configuration changes. Test service using the ingress-cert.pem as the CA certificate to make the request, noting that the Ingress mTLS feature is currently disabled.

curl --cacert ingress-cert.pem https://example.com/hi --connect-to example.com:443:$HOST_IP:443
Hi, there!

Advanced Configuration

Next, we will introduce the advanced configuration of FSM Ingress. Advanced configuration is set through the metadata.annotations field of the Ingress resource. The currently supported features are:

  • Path Rewrite
  • Specifying Load Balancing Algorithm
  • Session Persistence

Path Rewrite

This example demonstrates how to use the rewrite path annotation.

FSM provides two annotations, pipy.ingress.kubernetes.io/rewrite-target-from and pipy.ingress.kubernetes.io/rewrite-target-to, to configure path rewrite, both of these are required, when used.

In the following example, a route rule defines that requests with a path prefix of /httpbin will be routed to the 14001 port of the httpbin service, but the service itself does not have this path. This is where the path rewrite feature comes in.

  • pipy.ingress.kubernetes.io/rewrite-target-from: ^/httpbin/? ，This supports regular expressions, where the starting path of /httpbin/ is rewritten.
  • pipy.ingress.kubernetes.io/rewrite-target-to: /，Here, the specified rewrite content is /.

In summary, the path starting with /httpbin/ will be replaced with /, for example, /httpbin/get will be rewritten as /get.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: ^/httpbin/?
    pipy.ingress.kubernetes.io/rewrite-target-to: /
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /httpbin
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8000

After applying above Ingress configuration, we now can use the path /httpbin/hi to access the /hi endpoint of the httpbin service.

curl http://example.com/httpbin/hi --connect-to example.com:80:$HOST_IP:80
Hi, there!

Specifying Load Balancing Algorithm

This example demonstrates specifying the load balancing algorithm when you have multiple replicas running and you want to distribute load among them.

By default, FSM Ingress uses the Round-Robin load balancing algorithm, but other algorithms can be specified using the annotation pipy.ingress.kubernetes.io/lb-type annotation. Other supported load balancing algorithms are:

  • round-robin
  • hashing
  • least-work

In the following example, the hashing load balancing algorithm is specified through the annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/lb-type: 'hashing'
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8000

To demonstrate the effect of the demonstration, deploy the following example application, which has two instances and carries the hostname in the response to distinguish the response request example.

kubectl create namespace httpbin
kubectl apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: pipy
        image: addozhang/httpbin:latest
        ports:
          - containerPort: 8000
        command:
        - pipy
        - -e
        - |
          pipy()
          .listen(8000)
          .serveHTTP(new Message('Response from pod ' + os.env["HOSTNAME"]))
EOF

Apply above configuration and send requests to test the configuration.

curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-t9cxc
curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-t9cxc
curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-t9cxc

Session Persistence

This example demonstrates session persistence functionality.

FSM Ingress provides the annotation pipy.ingress.kubernetes.io/session-sticky to configure session persistence, with a default value of false (equivalent to no, 0, off, or ), meaning that the session is not kept. If you need to keep the session, you need to set the value to true, yes, 1, or on.

For example, in the following example, the annotation value is set to true, used to maintain the session between the two instances of the backend service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/session-sticky: 'true'
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8000

In order to demonstrate the effect, deploy the following example application, which has two instances and carries the host name in the response to differentiate the response request.

kubectl create namespace httpbin
kubectl apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
        - name: pipy
          image: addozhang/httpbin:latest
          ports:
            - containerPort: 8000
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8000)
              .serveHTTP(new Message('Response from pod ' + os.env["HOSTNAME"]))
EOF

Applying the above Ingress configuration, send multiple requests to test and it can be observed that it’s always the same instance responding to the request.

curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-hrvqp
curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-hrvqp
curl http://example.com/ --connect-to example.com:80:$HOST_IP:80
Response from pod httpbin-5f69c44674-hrvqp

3 - Ingress Controller - Advanced TLS

FSM Ingress Controller advanced TLS features and configuration

This guide demonstrate how to configure TLS and its related functionality.

Prerequisites

  • Kubernetes cluster version v1.19.0 or higher.
  • Interact with the API server using kubectl.
  • FSM CLI installed.
  • FSM Ingress Controller installed followed by installation document

Continuing with the previous article environment and providing examples of HTTP access at port 8000 and HTTPS access at port 8443.

Sample Application

The example application below provides access through both HTTP at port 8000 and HTTPS at port 8443, with the following URI:

  • / returns a simple HTML page
  • /hi returns a 200 response with string Hi, there!
  • /api/private returns a 401 response with string Staff only

To provide HTTPS, a CA certificate and server certificate need to be issued for the application first.

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \
   -key ca-key.pem \
   -out ca-cert.pem \
   -subj '/CN=flomesh.io'

openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out server.csr -subj '/CN=example.com'
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -days 365

Before deploying the sample service, first let’s create a secret to save the certificate and key in the secret and mount it in the application pod.

kubectl create namespace httpbin
# mount self-signed cert to sample app pod via secret
kubectl create secret generic -n httpbin server-cert \
  --from-file=./server-cert.pem \
  --from-file=./server-key.pem

kubectl apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - port: 8443
    name: https
  - port: 8000
    name: http
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
        - name: pipy
          image: addozhang/httpbin:latest
          env:
            - name: PIPY_CONFIG_FILE
              value: /etc/pipy/tutorial/gateway/main.js
          ports:
            - containerPort: 8443
            - containerPort: 8000
          volumeMounts:
          - name: cert
            mountPath: "/etc/pipy/tutorial/gateway/secret"
            readOnly: true
      volumes:
      - name: cert
        secret:
          secretName: server-cert
EOF

HTTPS Upstream

This example demonstrates how FSM Ingress can send requests to an HTTPS backend. FSM Ingress provides the following 3 annotations:

  • pipy.ingress.kubernetes.io/upstream-ssl-name：SNI of the upstream service, such as example.com
  • pipy.ingress.kubernetes.io/upstream-ssl-secret：Secret that contains the TLS certificate, formatted as SECRET_NAME or NAMESPACE/SECRET_NAME, such as httpbin/tls-cert
  • pipy.ingress.kubernetes.io/upstream-ssl-verify：Whether to verify the certificate of the upstream, defaults to false, meaning that the connection will still be established even if the certificate validation fails.

In the following Ingress resource example, the annotation pipy.ingress.kubernetes.io/upstream-ssl-secret specifies the secret tls-cert that contains the TLS certificate in the namespace httpbin.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/upstream-ssl-secret: httpbin/tls-cert
    pipy.ingress.kubernetes.io/upstream-ssl-verify: 'true'
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8443

To create the secret tls-cert with a certificate and key, you can use the following command:

kubectl create secret generic -n httpbin tls-cert \
  --from-file=tls.crt=./ca-cert.pem --from-file=ca.crt=./ca-cert.pem --from-file=tls.key=ca-key.pem

Apply the above Ingress configuration and access the HTTPS upstream application using HTTP ingress

curl http://example.com/hi --connect-to example.com:80:$HOST_IP:80
Hi, there!

Check the logs of the fsm-ingress pod to see that it is connecting to the upstream HTTPS port 8443.

kubectl logs -n fsm-system -l app=fsm-ingress | tail -5
2023-09-14 04:39:41.933 [INF] [router] Request Host:  example.com
2023-09-14 04:39:41.933 [INF] [router] Request Path:  /hi
2023-09-14 04:39:41.934 [INF] [balancer] _sourceIP 10.42.0.1
2023-09-14 04:39:41.934 [INF] [balancer] _connectTLS true
2023-09-14 04:39:41.934 [INF] [balancer] _mTLS true
2023-09-14 04:39:41.934 [INF] [balancer] _target.id 10.42.0.101:8443
2023-09-14 04:39:41.934 [INF] [balancer] _isOutboundGRPC false

Client Certificate Verification

This example demonstrates how to verify client certificates when TLS termination and mTLS are enabled.

Before using the mTLS feature, ensure that FSM Ingress is enabled and configured with TLS, by providing the parameter --set fsm.serviceLB.enabled=true during FSM installation.

Note: This can be enabled ONLY during FSM installation.

To enable the mTLS feature, you can either enable it during FSM Ingress installation by providing the parameter --set fsm.fsmIngress.tls.mTLS=true or modify the configuration after installation. The specific operation is to modify the ConfigMap fsm-mesh-config under the FSM namespace, and set the value of tls.mTLS to true. Or, enable it when enabling FSM ingress with command below:

fsm ingress enable --tls-enable --mtls

In FSM Ingress, the annotation pipy.ingress.kubernetes.io/tls-trusted-ca-secret is provided to configure trusted client certificates.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    pipy.ingress.kubernetes.io/tls-trusted-ca-secret: httpbin/trust-client-cert
spec:
  ingressClassName: pipy
  rules:
  - host: example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: httpbin
              port:
                number: 8000
  tls:
  - hosts:
    - example.com
    secretName: ingress-cert

To issue a self-signed certificate for an Ingress, you can run below command:

openssl req -x509 -newkey rsa:4096 -keyout ingress-key.pem -out ingress-cert.pem -sha256 -days 365 -nodes -subj '/CN=example.com'

Generate a Secret using generated certificate and privatey key by running below command:

kubectl create secret tls ingress-cert --cert=ingress-cert.pem --key=ingress-key.pem -n httpbin

issue a self-signed TLS certificate for client service

openssl req -x509 -newkey rsa:4096 -keyout client-key.pem -out client-cert.pem -sha256 -days 365 -nodes -subj '/CN=flomesh.io'

Generate a Secret resource by using generated self-signed client certificate.

kubectl create secret generic -n httpbin trust-client-cert \
  --from-file=ca.crt=./client-cert.pem

Apply above Ingress configurations.

curl --cacert ingress-cert.pem --cert client-cert.pem --key client-key.pem https://example.com/hi --connect-to example.com:443:$HOST_IP:443
Hi, there!

4 - Ingress with Kubernetes Nginx Ingress Controller

HTTP and HTTPS ingress with Kubernetes Nginx Ingress Controller

This guide will demonstrate how to configure HTTP and HTTPS ingress to a service part of an FSM managed service mesh when using Kubernetes Nginx Ingress Controller.

Prerequisites

  • Kubernetes cluster running Kubernetes v1.19.0 or greater.
  • Have kubectl available to interact with the API server.
  • Have FSM version >= v0.10.0 installed.
  • Have Kubernetes Nginx Ingress Controller installed. Refer to the deployment guide to install it.

Demo

First, note the details regarding FSM and Nginx installations:

fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM is installed
fsm_mesh_name=fsm # replace fsm with the mesh name (use `fsm mesh list` command)

nginx_ingress_namespace=<nginx-namespace> # replace <nginx-namespace> with the namespace where Nginx is installed
nginx_ingress_service=<nginx-ingress-controller-service> # replace <nginx-ingress-controller-service> with the name of the nginx ingress controller service
nginx_ingress_host="$(kubectl -n "$nginx_ingress_namespace" get service "$nginx_ingress_service" -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
nginx_ingress_port="$(kubectl -n "$nginx_ingress_namespace" get service "$nginx_ingress_service" -o jsonpath='{.spec.ports[?(@.name=="http")].port}')"

To restrict ingress traffic on backends to authorized clients, we will set up the IngressBackend configuration such that only ingress traffic from the endpoints of the Nginx Ingress Controller service can route traffic to the service backend. To be able to discover the endpoints of this service, we need FSM controller to monitor the corresponding namespace. However, Nginx must NOT be injected with an Pipy sidecar to function properly.

fsm namespace add "$nginx_ingress_namespace" --mesh-name "$fsm_mesh_name" --disable-sidecar-injection

Next, we will deploy the sample httpbin service.

# Create a namespace
kubectl create ns httpbin

# Add the namespace to the mesh
fsm namespace add httpbin

# Deploy the application
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Confirm the httpbin service and pod is up and running:

kubectl get pods -n httpbin
NAME                       READY   STATUS    RESTARTS   AGE
httpbin-74677b7df7-zzlm2   2/2     Running   0          11h

kubectl get svc -n httpbin
NAME      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
httpbin   ClusterIP   10.0.22.196   <none>        14001/TCP   11h

HTTP Ingress

Next, we will create the Ingress and IngressBackend configurations necessary to allow external clients to access the httpbin service on port 14001 in the httpbin namespace. The connection from the Nginx’s ingress service to the httpbin backend pod will be unencrypted since we aren’t using TLS.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
---
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$nginx_ingress_namespace"
    name: "$nginx_ingress_service"
EOF

Now, we expect external clients to be able to access the httpbin service for HTTP requests:

curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 200 OK
Date: Mon, 04 Jul 2022 06:55:26 GMT
Content-Type: application/json
Content-Length: 346
Connection: keep-alive
access-control-allow-origin: *
access-control-allow-credentials: true

HTTPS Ingress (mTLS and TLS)

To proxy connections to HTTPS backends, we will configure the Ingress and IngressBackend configurations to use https as the backend protocol, and have FSM issue a certificate that Nginx will use as the client certificate to proxy HTTPS connections to TLS backends. The client certificate and CA certificate will be stored in a Kubernetes secret that Nginx will use to authenticate service mesh backends.

To issue a client certificate for the Nginx ingress service, update the fsm-mesh-config MeshConfig resource.

kubectl edit meshconfig fsm-mesh-config -n "$fsm_namespace"

Add the ingressGateway field under spec.certificate:

certificate:
  ingressGateway:
    secret:
      name: fsm-nginx-client-cert
      namespace: <fsm-namespace> # replace <fsm-namespace> with the namespace where FSM is installed
    subjectAltNames:
    - ingress-nginx.ingress-nginx.cluster.local
    validityDuration: 24h

Note: The Subject Alternative Name (SAN) is of the form <service-account>.<namespace>.cluster.local, where the service account and namespace correspond to the Ngnix service.

Next, we need to create an Ingress and IngressBackend configuration to use TLS proxying to the backend service, while enabling proxying to the backend over mTLS. For this to work, we must create an IngressBackend resource that specifies HTTPS ingress traffic directed to the httpbin service must only accept traffic from a trusted client. FSM provisioned a client certificate for the Nginx ingress service with the Subject ALternative Name (SAN) ingress-nginx.ingress-nginx.cluster.local, so the IngressBackend configuration needs to reference the same SAN for mTLS authentication between the Nginx ingress service and the httpbin backend.

Apply the configurations:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    # proxy_ssl_name for a service is of the form <service-account>.<namespace>.cluster.local
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name "httpbin.httpbin.cluster.local";
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "fsm-system/fsm-nginx-client-cert"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
---
apiVersion: policy.flomesh.io/v1alpha1
kind: IngressBackend
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    name: "$nginx_ingress_service"
    namespace: "$nginx_ingress_namespace"
  - kind: AuthenticatedPrincipal
    name: ingress-nginx.ingress-nginx.cluster.local
EOF

Now, we expect external clients to be able to access the httpbin service for requests with HTTPS proxying over mTLS between the ingress gateway and service backend:

curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 200 OK
Date: Mon, 04 Jul 2022 06:55:26 GMT
Content-Type: application/json
Content-Length: 346
Connection: keep-alive
access-control-allow-origin: *
access-control-allow-credentials: true

To verify that unauthorized clients are not allowed to access the backend, update the sources specified in the IngressBackend configuration. Let’s update the principal to something other than the SAN encoded in the Nginx client’s certificate.

kubectl apply -f - <<EOF
apiVersion: policy.flomesh.io/v1alpha1
kind: IngressBackend
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    name: "$nginx_ingress_service"
    namespace: "$nginx_ingress_namespace"
  - kind: AuthenticatedPrincipal
    name: untrusted-client.cluster.local # untrusted
EOF

Confirm the requests are rejected with an HTTP 403 Forbidden response:

curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 403 Forbidden
Date: Wed, 18 Aug 2021 18:36:09 GMT
Content-Type: text/plain
Content-Length: 19
Connection: keep-alive

Next, we demonstrate support for disabling client certificate validation on the service backend if necessary, by updating our IngressBackend configuration to set skipClientCertValidation: true, while still using an untrusted client:

kubectl apply -f - <<EOF
apiVersion: policy.flomesh.io/v1alpha1
kind: IngressBackend
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: https
    tls:
      skipClientCertValidation: true
  sources:
  - kind: Service
    name: "$nginx_ingress_service"
    namespace: "$nginx_ingress_namespace"
  - kind: AuthenticatedPrincipal
    name: untrusted-client.cluster.local # untrusted
EOF

Confirm the requests succeed again since untrusted authenticated principals are allowed to connect to the backend:

curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 200 OK
Date: Mon, 04 Jul 2022 06:55:26 GMT
Content-Type: application/json
Content-Length: 346
Connection: keep-alive
access-control-allow-origin: *
access-control-allow-credentials: true

5 - Ingress with Traefik

Kubernetes ingress with Traefik Ingress Controller

This article demonstrates how to use Traefik Ingress to access the services hosted by the FSM service mesh.

Prerequisites 

  • Kubernetes cluster version v1.19.0 or higher.
  • Use kubectl to interact with the API server.
  • FSM is not installed, and must be removed first if installed.
  • fsm cli is installed to install FSM.
  • Helm 3 command line tool is installed for traefik installation.
  • FSM version >= v1.1.0.

Demo

Install Traefik

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace

Verify that the pod is up and running.

kubectl get po -n traefik
NAME READY STATUS RESTARTS AGE
traefik-69fb598d54-9v9vf 1/1 Running 0 24s

Retrieve and store external IP address and port of the entry gateway to environment variables, which will be used later to access the application.

export ingress_host="$(kubectl -n traefik get service traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
export ingress_port="$(kubectl -n traefik get service traefik -o jsonpath='{.spec.ports[? (@.name=="web")].port}')"

Install FSM

export fsm_namespace=fsm-system 
export fsm_mesh_name=fsm 

fsm install \
    --mesh-name "$fsm_mesh_name" \
    --fsm-namespace "$fsm_namespace" \
    --set=fsm.enablePermissiveTrafficPolicy=true

Confirm that the pod is up and running.

kubectl get po -n fsm-system
NAME READY STATUS RESTARTS AGE
fsm-bootstrap-6477f776cc-d5r89 1/1 Running 0 2m51s
fsm-injector-5696694cf6-7kvpt 1/1 Running 0 2m51s
fsm-controller-86d68c557b-tvgtm 2/2 Running 0 2m51s

Deploy sample service

kubectl create ns httpbin
fsm namespace add httpbin
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Confirm that the service has been created and the pod is up and running.

kubectl get svc -n httpbin
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
httpbin ClusterIP 10.43.51.114 <none> 14001/TCP 9s

kubectl get po -n httpbin
NAME READY STATUS RESTARTS AGE
httpbin-69dc7d545c-bsjxx 2/2 Running 0 77s

HTTP Ingress

Next, create an ingress to expose the 14001 port of the httpbin service under the httpbin namespace.

kubectl apply -f - <<EOF
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: httpbin.org
     http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Using the entry gateway address and port saved earlier to access the service, you should receive a response of 502 at this point. This is normal, because you still need to create IngressBackend to allow the entry gateway to access the httpbin service.

curl -sI http://"$ingress_host":"$ingress_port"/get -H "Host: httpbin.org"
HTTP/1.1 502 Bad Gateway
Date: Tue, 09 Aug 2022 13:17:11 GMT
Content-Length: 11
Content-Type: text/plain; charset=utf-8

Execute the following command to create IngressBackend.

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: traefik
    name: traefik
EOF

Now, re-visit httpbin and you will be able to access it successfully.

curl -sI http://"$ingress_host":"$ingress_port"/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 338
Content-Type: application/json
Date: Tue, 09 Aug 2022 13:17:41 GMT
fsm-Stats-Kind: Deployment
fsm-Stats-Name: httpbin
fsm-Stats-Namespace: httpbin
fsm-Stats-Pod: httpbin-69dc7d545c-bsjxx
Server: gunicorn/19.9.0

6 - FSM Ingress Controller - SSL Passthrough

How to use SSL passthrough feature of FSM Ingress

This guide demonstrate how to configure SSL passthrough feature of FSM Ingress

Prerequisites

  • Kubernetes cluster version v1.19.0 or higher.
  • Interact with the API server using kubectl.
  • FSM CLI installed.
  • TLS passthrough enabled following by installation document

Setup

Once all done, let’s retrieve Ingress host IP and port information.

export FSM_NAMESPACE=fsm-system #change this to the namespace your FSM ingress installed in

export ingress_host="$(kubectl -n "$FSM_NAMESPACE" get service fsm-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
export ingress_port="$(kubectl -n "$FSM_NAMESPACE" get service fsm-ingress -o jsonpath='{.spec.ports[?(@.name=="https")].port}')"

Test

For simplicity, we will not deploy an upstream service here, but instead use https://httpbin.org directly as the upstream, and resolve it to the ingress address obtained above through the curl’s revolve parameter. If the port of ingress is not 433, you can use the connect-to parameter --connect-to httpbin.org:443:$ingress_host:$ingress_port.

curl https://httpbin.org/get -i --resolve httpbin.org:443:$ingress_host:$ingress_port

HTTP/2 200
date: Tue, 31 Jan 2023 11:21:41 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.68.0",
    "X-Amzn-Trace-Id": "Root=1-63d8f9c5-5af02436470161040dc68f1e"
  },
  "origin": "20.205.11.203",
  "url": "https://httpbin.org/get"
}