This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Security

1: Permissive Traffic Policy Mode
2: Bi-direction TLS with FSM Ingress
3: Service-based access control
4: IP range-based access control
5: Cert-manager Certificate Provider

1 - Permissive Traffic Policy Mode

Set up application connectivity using service discovery without explicit SMI policies

This guide demonstrates a client and server application within the service mesh communicating using FSM’s permissive traffic policy mode, which configures application connectivity using service discovery without the need for explicit SMI traffic access policies.

Prerequisites

Kubernetes cluster running Kubernetes v1.19.0 or greater.
Have FSM installed.
Have kubectl available to interact with the API server.
Have fsm CLI available for managing the service mesh.

Demo

The following demo shows an HTTP curl client making HTTP requests to the httpbin service using permissive traffic policy mode.

Enable permissive mode if not enabled.

export FSM_NAMESPACE=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}'  --type=merge

Deploy the httpbin service into the httpbin namespace after enrolling its namespace to the mesh. The httpbin service runs on port 14001.

# Create the httpbin namespace
kubectl create namespace httpbin

# Add the namespace to the mesh
fsm namespace add httpbin

# Deploy httpbin service in the httpbin namespace
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Confirm the httpbin service and pods are up and running.

$ kubectl get svc -n httpbin
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
httpbin   ClusterIP   10.96.198.23   <none>        14001/TCP   20s

$ kubectl get pods -n httpbin
NAME                     READY   STATUS    RESTARTS   AGE
httpbin-5b8b94b9-lt2vs   2/2     Running   0          20s

Deploy the curl client into the curl namespace after enrolling its namespace to the mesh.

# Create the curl namespace
kubectl create namespace curl

# Add the namespace to the mesh
fsm namespace add curl

# Deploy curl client in the curl namespace
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/curl/curl.yaml -n curl

Confirm the curl client pod is up and running.

$ kubectl get pods -n curl
NAME                    READY   STATUS    RESTARTS   AGE
curl-54ccc6954c-9rlvp   2/2     Running   0          20s

Confirm the curl client is able to access the httpbin service on port 14001.

$ kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Wed, 29 Jun 2022 08:50:33 GMT
content-type: text/html; charset=utf-8
content-length: 9593
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

A 200 OK response indicates the HTTP request from the curl client to the httpbin service was successful.

Confirm the HTTP requests fail when permissive traffic policy mode is disabled.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}'  --type=merge

$ kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001
curl: (52) Empty reply from server
command terminated with exit code 52

2 - Bi-direction TLS with FSM Ingress

Configuring different TLS certificates for Ingress and Egress

This guide will demonstrate with multiple scenarios on how to configure different TLS certificates for Flomesh Service Mesh (FSM) Ingress and Egress communication.

Prerequisites

Kubernetes cluster running Kubernetes v1.19.0 or greater.
Have FSM installed.
Have kubectl available to interact with the API server.
Have fsm CLI available for managing the service mesh.
Have FSM Ingress Controller installed.

Install FSM Ingress Controller

if you haven’t yet installed FSM Ingress controller, you can install that quickly via

fsm install \
    --set=fsm.fsmIngress.enabled=true \
    --set=fsm.fsmIngress.tls.enabled=true \
    --set=fsm.fsmIngress.tls.mTLS=true

kubectl wait --namespace fsm-system \
  --for=condition=ready pod \
  --selector=app=fsm-ingress \
  --timeout=300s

Deploy demo pods

#Sample server service
kubectl create namespace egress-server

kubectl apply -n egress-server -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/server.yaml

#Sample middle-ware service
kubectl create namespace egress-middle
fsm namespace add egress-middle
kubectl apply -n egress-middle -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/middle.yaml

#Sample client
kubectl create namespace egress-client
kubectl apply -n egress-client -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/client.yaml

#Wait for POD to start properly
kubectl wait --for=condition=ready pod -n egress-server -l app=server --timeout=180s
kubectl wait --for=condition=ready pod -n egress-middle -l app=middle --timeout=180s
kubectl wait --for=condition=ready pod -n egress-client -l app=client --timeout=180s

Scenario#1: Client HTTP & HTTP Ingress & mTLS Egress

Test commands

Traffic flow:

Client –http–> ingress-pipy Controller

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test results

The correct return result is similar to :

HTTP/1.1 404 Not Found
Server: pipy/0.90.0
content-length: 17
connection: keep-alive

Service Not Found

Setup Ingress Rules

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egress-middle
  namespace: egress-middle
spec:
  ingressClassName: pipy
  rules:
  - host: fsm-ingress.fsm-system
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: middle
            port:
              number: 8080
EOF

Setup IngressBackend

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: egress-middle
  namespace: egress-middle
spec:
  backends:
  - name: middle
    port:
      number: 8080 # targetPort of middle service
      protocol: http
  sources:
  - kind: Service
    namespace: fsm-system
    name: fsm-ingress
EOF

Test Commands

Traffic Flow:

Client –http–> FSM Ingress –http –> sidecar –> Middle

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test Results

The correct return result is similar to :

HTTP/1.1 200 OK
date: Fri, 17 Nov 2023 09:10:45 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 13
connection: keep-alive

hello world.

Disable Egress Permissive mode

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":false}}}' --type=merge

Enable Egress Policy

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableEgressPolicy":true}}}' --type=merge

Create Egress mTLS Secret

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.crt -o middle.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.key -o middle.key

kubectl create secret generic -n fsm-system egress-middle-cert \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./middle.crt \
  --from-file=tls.key=./middle.key

Setup Egress Policy

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: server-8443
  namespace: egress-middle
spec:
  sources:
  - kind: ServiceAccount
    name: middle
    namespace: egress-middle
    mtls:
      issuer: other
      cert:
        sn: 1
        subjectAltNames: 
        - flomesh.io
        expiration: 2030-1-1 00:00:00
        secret:
          name: egress-middle-cert
          namespace: fsm-system
  hosts:
  - server.egress-server.svc.cluster.local
  ports:
  - number: 8443
    protocol: http
EOF

Test Commands

Traffic Flow:

Client –http–> FSM Ingress –http–> sidecar –> Middle –> sidecar –egress mtls–> Server

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/time

Test Results

The correct return result is similar to :

HTTP/1.1 200 OK
date: Fri, 17 Nov 2023 09:11:53 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 74
connection: keep-alive

The current time: 2023-11-17 09:11:53.67111584 +0000 UTC m=+110.875627674

This business scenario is tested and the strategy is cleaned up to avoid affecting subsequent tests

kubectl delete ingress -n egress-middle egress-middle
kubectl delete ingressbackend -n egress-middle egress-middle
kubectl delete egress -n egress-middle server-8443
kubectl delete secrets -n fsm-system egress-middle-cert

Scenario#2: HTTP FSM & mTLS Ingress & mTLS Egress

Test Commands

Traffic flow:

Client –http–> FSM Ingress Controller

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test Results

The correct return result is similar to :

HTTP/1.1 404 Not Found
Server: pipy/0.90.0
content-length: 17
connection: keep-alive

Service Not Found

Setup Ingress Controller TLS Certificate

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p \
'{
  "spec":{
    "certificate":{
      "ingressGateway":{
        "secret":{
          "name":"ingress-controller-cert",
          "namespace":"fsm-system"
        },
        "subjectAltNames":["fsm.fsm-system.cluster.local"],
        "validityDuration":"24h"
      }
    }
  }
}' \
--type=merge

Note: The Subject Alternative Name (SAN) is of the form ..cluster.local, where the service account and namespace correspond to the ingress-pipy service.

Setup Ingress Rules

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egress-middle
  namespace: egress-middle
  annotations:
    # upstream-ssl-name for a service is of the form <service-account>.<namespace>.cluster.local
    pipy.ingress.kubernetes.io/upstream-ssl-name: "middle.egress-middle.cluster.local"
    pipy.ingress.kubernetes.io/upstream-ssl-secret: "fsm-system/ingress-controller-cert"
    pipy.ingress.kubernetes.io/upstream-ssl-verify: "on"
spec:
  ingressClassName: pipy
  rules:
  - host: fsm-ingress.fsm-system
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: middle
            port:
              number: 8080
EOF

Setup IngressBackend Policy

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: egress-middle
  namespace: egress-middle
spec:
  backends:
  - name: middle
    port:
      number: 8080 # targetPort of middle service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: fsm-system
    name: fsm-ingress
  - kind: AuthenticatedPrincipal
    name: fsm.fsm-system.cluster.local
EOF

Test Commands

Traffic flow:

Client –http–> FSM Ingress –mtls –> sidecar –> Middle

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test Results

The correct return result is similar to :

HTTP/1.1 200 OK
date: Fri, 17 Nov 2023 09:12:39 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 13
connection: keep-alive

hello world.

Disable Egress Permissive mode

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":false}}}' --type=merge

Enable Egress Policy

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableEgressPolicy":true}}}' --type=merge

Create Egress mTLS Secret

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.crt -o middle.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.key -o middle.key

kubectl create secret generic -n fsm-system egress-middle-cert \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./middle.crt \
  --from-file=tls.key=./middle.key

Setup Egress Policy

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: server-8443
  namespace: egress-middle
spec:
  sources:
  - kind: ServiceAccount
    name: middle
    namespace: egress-middle
    mtls:
      issuer: other
      cert:
        sn: 1
        subjectAltNames: 
        - flomesh.io
        expiration: 2030-1-1 00:00:00
        secret:
          name: egress-middle-cert
          namespace: fsm-system
  hosts:
  - server.egress-server.svc.cluster.local
  ports:
  - number: 8443
    protocol: http
EOF

Test Commands

Traffic flow:

Client –http–> FSM Ingress –mtls–> sidecar –> Middle –> sidecar –egress mtls–> Server

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/time

Test Results

The correct return result is similar to :

HTTP/1.1 200 OK
date: Fri, 17 Nov 2023 09:13:09 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 72
connection: keep-alive

The current time: 2023-11-17 09:13:09.478407 +0000 UTC m=+186.682918839

This business scenario is tested and the strategy is cleaned up to avoid affecting subsequent tests

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"certificate":{"ingressGateway":null}}}' --type=merge

kubectl delete ingress -n egress-middle egress-middle
kubectl delete ingressbackend -n egress-middle egress-middle
kubectl delete egress -n egress-middle server-8443
kubectl delete secrets -n fsm-system egress-middle-cert

Scenario#3：TLS FSM Ingress & mTLS Ingress & mTLS Egress

Test Commands

Traffic flow:

Client –http–> FSM Ingress Controller

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test Results

The correct return result is similar to :

HTTP/1.1 404 Not Found
Server: pipy/0.90.0
content-length: 17
connection: keep-alive

Service Not Found

Setup Ingress Controller Cert

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"certificate":{"ingressGateway":{"secret":{"name":"ingress-controller-cert","namespace":"fsm-system"},"subjectAltNames":["fsm.fsm-system.cluster.local"],"validityDuration":"24h"}}}}' --type=merge

Create Ingress TLS Secret

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/ingress-pipy.crt -o ingress-pipy.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/ingress-pipy.key -o ingress-pipy.key

kubectl create secret generic -n egress-middle ingress-pipy-cert-secret \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./ingress-pipy.crt \
  --from-file=tls.key=./ingress-pipy.key

Create Egress mTLS Secret

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.crt -o middle.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.key -o middle.key

kubectl create secret generic -n fsm-system egress-middle-cert \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./middle.crt \
  --from-file=tls.key=./middle.key

Setup Ingress Rules

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egress-middle
  namespace: egress-middle
  annotations:
    # upstream-ssl-name for a service is of the form <service-account>.<namespace>.cluster.local
    pipy.ingress.kubernetes.io/upstream-ssl-name: "middle.egress-middle.cluster.local"
    pipy.ingress.kubernetes.io/upstream-ssl-secret: "fsm-system/ingress-controller-cert"
    pipy.ingress.kubernetes.io/upstream-ssl-verify: "on"
spec:
  ingressClassName: pipy
  rules:
  - host: fsm-ingress.fsm-system
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: middle
            port:
              number: 8080
  tls:
  - hosts:
    - fsm-ingress.fsm-system
    secretName: ingress-pipy-cert-secret
EOF

Setup IngressBackend Policy

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: egress-middle
  namespace: egress-middle
spec:
  backends:
  - name: middle
    port:
      number: 8080 # targetPort of middle service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: fsm-system
    name: fsm-ingress
  - kind: AuthenticatedPrincipal
    name: fsm.fsm-system.cluster.local
EOF

Replace client TLS certificate

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/client.crt -o client.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/client.key -o client.key

kubectl create secret generic -n egress-client egress-client-secret \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./client.crt \
  --from-file=tls.key=./client.key 

kubectl -n egress-client patch deploy client -p \
  '
  {
    "spec": {
      "template": {
        "spec": {
          "containers": [{
            "name": "client",
            "volumeMounts": [{
              "mountPath": "/client",
              "name": "client-certs"
            }]
          }],
          "volumes": [{
            "secret": {
              "secretName": "egress-client-secret"
            },
            "name": "client-certs"
          }]
        }
      }
    }
  }
  '

FSM disable inbound mTLS

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"ingress":{"tls":{"mTLS": false}}}}' --type=merge

Test Commands

Traffic flow:

Client –tls–> Ingress FSM –mtls –> sidecar –> Middle

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si https://fsm-ingress.fsm-system/hello --cacert /client/ca.crt

Test Results

The correct return result is similar to :

HTTP/2 200
date: Fri, 17 Nov 2023 09:17:43 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 13

hello world.

Disable Egress Permissive mode

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":false}}}' --type=merge

Enable Egress Policy

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableEgressPolicy":true}}}' --type=merge

Setup Egress Policy

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: server-8443
  namespace: egress-middle
spec:
  sources:
  - kind: ServiceAccount
    name: middle
    namespace: egress-middle
    mtls:
      issuer: other
      cert:
        sn: 1
        expiration: 2030-1-1 00:00:00
        subjectAltNames: 
        - flomesh.io
        secret:
          name: egress-middle-cert
          namespace: fsm-system
  hosts:
  - server.egress-server.svc.cluster.local
  ports:
  - number: 8443
    protocol: http
EOF

Test Commands

Traffic flow:

Client –tls–> Ingress FSM –mtls–> sidecar –> Middle –> sidecar –egress mtls–> Server

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si https://fsm-ingress.fsm-system/time --cacert /client/ca.crt

Test Results

The correct return result is similar to :

HTTP/2 200
date: Fri, 17 Nov 2023 09:18:02 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 75

The current time: 2023-11-17 09:18:02.826626944 +0000 UTC m=+480.031138782

This business scenario is tested and the strategy is cleaned up to avoid affecting subsequent tests

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"certificate":{"ingressGateway":null}}}' --type=merge

kubectl delete ingress -n egress-middle egress-middle
kubectl delete ingressbackend -n egress-middle egress-middle
kubectl delete egress -n egress-middle server-8443
kubectl delete secrets -n fsm-system egress-middle-cert
kubectl delete secrets -n egress-middle ingress-pipy-cert-secret
kubectl delete secrets -n egress-client egress-client-secret

Scenario#4：mTLS FSM & mTLS Ingress & mTLS Egress

Test Commands

Traffic flow:

Client –http–> FSM Ingress Controller

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si http://fsm-ingress.fsm-system/hello

Test Results

The correct return result is similar to :

HTTP/1.1 404 Not Found
Server: pipy/0.90.0
content-length: 17
connection: keep-alive

Service Not Found

Setup Ingress Controller Cert

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"certificate":{"ingressGateway":{"secret":{"name":"ingress-controller-cert","namespace":"fsm-system"},"subjectAltNames":["fsm.fsm-system.cluster.local"],"validityDuration":"24h"}}}}' --type=merge

Create FSM TLS Secret and CA Secret

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/ingress-pipy.crt -o ingress-pipy.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/ingress-pipy.key -o ingress-pipy.key

kubectl create secret generic -n egress-middle ingress-pipy-cert-secret \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./ingress-pipy.crt \
  --from-file=tls.key=./ingress-pipy.key

kubectl create secret generic -n egress-middle ingress-controller-ca-secret \
  --from-file=ca.crt=./pipy-ca.crt

Replace client TLS certificate

curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/client.crt -o client.crt
curl -s https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/client.key -o client.key

kubectl create secret generic -n egress-client egress-client-secret \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./client.crt \
  --from-file=tls.key=./client.key 

kubectl -n egress-client patch deploy client -p \
  '
  {
    "spec": {
      "template": {
        "spec": {
          "containers": [{
            "name": "client",
            "volumeMounts": [{
              "mountPath": "/client",
              "name": "client-certs"
            }]
          }],
          "volumes": [{
            "secret": {
              "secretName": "egress-client-secret"
            },
            "name": "client-certs"
          }]
        }
      }
    }
  }
  '

Setup Ingress Rules

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egress-middle
  namespace: egress-middle
  annotations:
    # mTLS
    pipy.ingress.kubernetes.io/tls-trusted-ca-secret: egress-middle/ingress-controller-ca-secret
    pipy.ingress.kubernetes.io/tls-verify-client: "on"
    pipy.ingress.kubernetes.io/tls-verify-depth: "1"

    # upstream-ssl-name for a service is of the form <service-account>.<namespace>.cluster.local
    pipy.ingress.kubernetes.io/upstream-ssl-name: "middle.egress-middle.cluster.local"
    pipy.ingress.kubernetes.io/upstream-ssl-secret: "fsm-system/ingress-controller-cert"
    pipy.ingress.kubernetes.io/upstream-ssl-verify: "on"
spec:
  ingressClassName: pipy
  rules:
  - host: fsm-ingress.fsm-system
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: middle
            port:
              number: 8080
  tls:
  - hosts:
    - fsm-ingress.fsm-system
    secretName: ingress-pipy-cert-secret
EOF

Setup IngressBackend Policy

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: egress-middle
  namespace: egress-middle
spec:
  backends:
  - name: middle
    port:
      number: 8080 # targetPort of middle service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: fsm-system
    name: fsm-ingress
  - kind: AuthenticatedPrincipal
    name: fsm.fsm-system.cluster.local
EOF

FSM enable inbound mTLS

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"ingress":{"tls":{"mTLS": true}}}}' --type=merge

Test Commands

Traffic flow:

Client –mtls–> Ingress FSM –mtls –> sidecar –> Middle

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si https://fsm-ingress.fsm-system/hello  --cacert /client/ca.crt --key /client/tls.key --cert /client/tls.crt

Test Results

The correct return result is similar to :

HTTP/2 200
date: Fri, 17 Nov 2023 09:19:57 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 13

hello world.

Disable Egress Permissive mode

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":false}}}' --type=merge

Enable Egress Policy

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableEgressPolicy":true}}}' --type=merge

Create Egress mTLS Secret

curl https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/pipy-ca.crt -o pipy-ca.crt
curl https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.crt -o middle.crt
curl https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/bidir-mtls/certs/middle.key -o middle.key

kubectl create secret generic -n fsm-system egress-middle-cert \
  --from-file=ca.crt=./pipy-ca.crt \
  --from-file=tls.crt=./middle.crt \
  --from-file=tls.key=./middle.key

Setup Egress Policy

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: server-8443
  namespace: egress-middle
spec:
  sources:
  - kind: ServiceAccount
    name: middle
    namespace: egress-middle
    mtls:
      issuer: other
      cert:
        sn: 1
        subjectAltNames: 
        - flomesh.io
        expiration: 2030-1-1 00:00:00
        secret:
          name: egress-middle-cert
          namespace: fsm-system
  hosts:
  - server.egress-server.svc.cluster.local
  ports:
  - number: 8443
    protocol: http
EOF

Test Commands

Traffic flow:

Client –mtls–> Ingress FSM –mtls–> sidecar –> Middle –> sidecar –egress mtls–> Server

kubectl exec "$(kubectl get pod -n egress-client -l app=client -o jsonpath='{.items..metadata.name}')" -n egress-client -- curl -si https://fsm-ingress.fsm-system/time --cacert /client/ca.crt --key /client/tls.key --cert /client/tls.crt

Test Results

The correct return result is similar to :

HTTP/2 200
date: Fri, 17 Nov 2023 09:20:24 GMT
content-type: text/plain; charset=utf-8
fsm-stats: egress-middle,Deployment,middle,middle-7965485977-nlnl2
content-length: 75

The current time: 2023-11-17 09:20:24.101929396 +0000 UTC m=+621.306441226

This business scenario is tested and the strategy is cleaned up to avoid affecting subsequent tests

export FSM_NAMESPACE=fsm-system
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"certificate":{"ingressGateway":null}}}' --type=merge
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"ingress":{"tls":{"mTLS": false}}}}' --type=merge

kubectl delete ingress -n egress-middle egress-middle
kubectl delete ingressbackend -n egress-middle egress-middle
kubectl delete egress -n egress-middle server-8443
kubectl delete secrets -n fsm-system egress-middle-cert
kubectl delete secrets -n egress-middle ingress-pipy-cert-secret
kubectl delete secrets -n egress-middle ingress-controller-ca-secret
kubectl delete secrets -n egress-client egress-client-secret

3 - Service-based access control

Managing access to services

This guide demonstrates an access control mechanism applied at Service level to services within the mesh.

Prerequisites

Kubernetes cluster running Kubernetes v1.19.0 or greater.
Have FSM installed.
Have kubectl available to interact with the API server.
Have fsm CLI available for managing the service mesh.

Demo

Deploy the sample services httpbin and curl.

#Mock target service
kubectl create namespace httpbin
fsm namespace add httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml

#Mock external service
kubectl create namespace curl
kubectl apply -n curl -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/curl/curl.yaml

#Wait for the dependent POD to start normally
kubectl wait --for=condition=ready pod -n httpbin -l app=httpbin --timeout=180s
kubectl wait --for=condition=ready pod -n curl -l app=curl --timeout=180s

At this point, we send a request from service curl to target service httpbin by executing the following command.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
# You will get error as below
command terminated with exit code 56

The access fails because by default the services outside the mesh cannot access the services inside the mesh and we need to apply an access control policy.

Before applying the policy, you need to enable the access control feature, which is disabled by default.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableAccessControlPolicy":true}}}' --type=merge

Plaintext transfer

Data can be transferred in plaintext or with two-way TLS encryption. Plaintext transfer is relatively simple, so let’s demonstrate the plaintext transfer scenario first.

Service-based access control

First, create a Service for service curl.

kubectl apply -n curl -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
EOF

Next, create an access control policy with the source Service curl and the target service httpbin.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: curl
    name: curl
EOF

Execute the command again to send the authentication request, and you can see that this time an HTTP 200 response is received.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
#Response as below
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 08:47:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
fsm-stats-namespace: httpbin
fsm-stats-kind: Deployment
fsm-stats-name: httpbin
fsm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive

Remember to execute kubectl delete accesscontrol httpbin -n httpbin to clean up the policy.

The previous ones we used were plaintext transfers, next we look at encrypted transfers.

Encrypted transfers

The default access policy certificate feature is off, turn it on by executing the following command.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableAccessCertPolicy":true}}}' --type=merge

Create AccessCert for the access source to assign a certificate for data encryption. The controller will store the certificate information in Secret curl-mtls-secret under the namespace curl, and here also assign SAN curl.curl.cluster.local for the access source.

kubectl apply -f - <<EOF
kind: AccessCert
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: curl-mtls-cert
  namespace: httpbin
spec:
  subjectAltNames:
  - curl.curl.cluster.local
  secret:
    name: curl-mtls-secret
    namespace: curl
EOF

Redeploy curl and mount the system-assigned Secret to the pod.

kubectl apply -n curl -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
        volumeMounts:
        - name: curl-mtls-secret
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: curl-mtls-secret
          secret:
            secretName: curl-mtls-secret
EOF

4 - IP range-based access control

Managing access to services based on IP ranges

This guide demonstrates an access control mechanism applied to services at IP Range level within the FSM mesh.

Prerequisites

Kubernetes cluster running Kubernetes v1.19.0 or greater.
Have FSM installed.
Have kubectl available to interact with the API server.
Have fsm CLI available for managing the service mesh.

Demo

deploy the sample services httpbin and curl

#Mock target service
kubectl create namespace httpbin
fsm namespace add httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml

#Mock external service
kubectl create namespace curl
kubectl apply -n curl -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/curl/curl.yaml

#Wait for the dependent POD to start normally
kubectl wait --for=condition=ready pod -n httpbin -l app=httpbin --timeout=180s
kubectl wait --for=condition=ready pod -n curl -l app=curl --timeout=180s

At this point, we send a request from service curl to target service httpbin by executing the following command.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
# You will get error as below
command terminated with exit code 56

The access fails because by default the services outside the mesh cannot access the services inside the mesh and we need to apply an access control policy.

Before applying the policy, you need to enable the access control feature, which is disabled by default.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableAccessControlPolicy":true}}}' --type=merge

IP Range-Based Access Control

IP range-based control is simple. You just need to specify the type of access source as IPRange and specify the IP address of the access source. As the application curl is redeployed, its IP address needs to be retrieved (perhaps you have discovered the drawbacks of IP range-based access control).

curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"

Plaintext transfer

Data can be transferred in plaintext or with two-way TLS encryption. Plaintext transfer is relatively simple, so let’s demonstrate the plaintext transfer scenario first.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF

Send the request again for testing.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
#Response as below
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:58:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
fsm-stats-namespace: httpbin
fsm-stats-kind: Deployment
fsm-stats-name: httpbin
fsm-stats-pod: httpbin-69dc7d545c-qphrh

Remember to execute kubectl delete accesscontrol httpbin -n httpbin to clean up the policy.

The previous ones we used were plaintext transfers, next we look at encrypted transfers.

Encrypted transfers

The default access policy certificate feature is off, turn it on by executing the following command.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"featureFlags":{"enableAccessCertPolicy":true}}}' --type=merge

kubectl apply -f - <<EOF
kind: AccessCert
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: curl-mtls-cert
  namespace: httpbin
spec:
  subjectAltNames:
  - curl.curl.cluster.local
  secret:
    name: curl-mtls-secret
    namespace: curl
EOF

Redeploy curl and mount the system-assigned Secret to the pod.

kubectl apply -n curl -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
        volumeMounts:
        - name: curl-mtls-secret
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: curl-mtls-secret
          secret:
            secretName: curl-mtls-secret
EOF

5 - Cert-manager Certificate Provider

Using cert-manager as a certificate provider

This guide demonstrates the usage of cert-manager as a certificate provider to manage and issue certificates in FSM.

Prerequisites

Kubernetes cluster running Kubernetes v1.19.0 or greater.
Have kubectl available to interact with the API server.
Have fsm CLI available for installing and managing the service mesh.

Demo

The following demo uses cert-manager as the certificate provider to issue certificates to the curl and httpbin applications communicating over Mutual TLS (mTLS) in an FSM managed service mesh.

Install cert-manager. This demo uses cert-manager v1.6.1.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

Confirm the pods are ready and running in the cert-manager namespace.

kubectl get pod -n cert-manager
NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-55658cdf68-pdnzg             1/1     Running   0          2m33s
cert-manager-cainjector-967788869-prtjq   1/1     Running   0          2m33s
cert-manager-webhook-6668fbb57d-vzm4j     1/1     Running   0          2m33s

Configure cert-manager Issuer and Certificate resources required by cert-manager to be able to issue certificates in FSM. These resources must be created in the namespace where FSM will be installed later.

Note: cert-manager must first be installed, with an issuer ready, before FSM can be installed using cert-manager as the certificate provider.

Create the namespace where FSM will be installed.

export FSM_NAMESPACE=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl create namespace "$FSM_NAMESPACE"

Next, we use a SelfSigned issuer to bootstrap a custom root certificate. This will create a SelfSigned issuer, issue a root certificate, and use that root as a CA issuer for certificates issued to workloads within the mesh.

# Create Issuer and Certificate resources
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: "$FSM_NAMESPACE"
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: fsm-ca
  namespace: "$FSM_NAMESPACE"
spec:
  isCA: true
  duration: 87600h # 365 days
  secretName: fsm-ca-bundle
  commonName: fsm-system
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: fsm-ca
  namespace: "$FSM_NAMESPACE"
spec:
  ca:
    secretName: fsm-ca-bundle
EOF

Confirm the fsm-ca-bundle CA secret is created by cert-manager in FSM’s namespace.
```
kubectl get secret fsm-ca-bundle -n "$FSM_NAMESPACE"
NAME            TYPE                DATA   AGE
fsm-ca-bundle   kubernetes.io/tls   3      84s
```
The CA certificate saved in this secret will be used by FSM upon install to bootstrap its ceritifcate provider utility.

Install FSM with its certificate provider kind set to cert-manager.

fsm install --set fsm.certificateProvider.kind="cert-manager"

Confirm the FSM control plane pods are ready and running.

kubectl get pod -n "$FSM_NAMESPACE"
NAME                              READY   STATUS    RESTARTS   AGE
fsm-bootstrap-7ddc6f9b85-k8ptp    1/1     Running   0          2m52s
fsm-controller-79b777889b-mqk4g   1/1     Running   0          2m52s
fsm-injector-5f96468fb7-p77ps     1/1     Running   0          2m52s

Enable permissive traffic policy mode to set up automatic application connectivity.
Note: this is not a requirement to use cert-manager but simplifies the demo by not requiring explicit traffic policies for application connectivity.
```
kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}'  --type=merge
```

Deploy the httpbin service into the httpbin namespace after enrolling its namespace to the mesh. The httpbin service runs on port 14001.

# Create the httpbin namespace
kubectl create namespace httpbin

# Add the namespace to the mesh
fsm namespace add httpbin

# Deploy httpbin service in the httpbin namespace
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Confirm the httpbin service and pods are up and running.

kubectl get svc -n httpbin
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
httpbin   ClusterIP   10.96.198.23   <none>        14001/TCP   20s

kubectl get pods -n httpbin
NAME                     READY   STATUS    RESTARTS   AGE
httpbin-5b8b94b9-lt2vs   2/2     Running   0          20s

Deploy the curl client into the curl namespace after enrolling its namespace to the mesh.

# Create the curl namespace
kubectl create namespace curl

# Add the namespace to the mesh
fsm namespace add curl

# Deploy curl client in the curl namespace
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/curl/curl.yaml -n curl

Confirm the curl client pod is up and running.

kubectl get pods -n curl
NAME                    READY   STATUS    RESTARTS   AGE
curl-54ccc6954c-9rlvp   2/2     Running   0          20s

Confirm the curl client is able to access the httpbin service on port 14001.

kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001
#Response as below
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 04 Jul 2022 09:34:11 GMT
content-type: text/html; charset=utf-8
content-length: 9593
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

A 200 OK response indicates the HTTP request from the curl client to the httpbin service was successful. The traffic between the application sidecar proxies is encrypted and authenticated using Mutual TLS (mTLS) by leverging the certificates issued by the cert-manager certificate provider.