How-to Guides

• 1: Operating guides
• 1.1: Install the FSM CLI
• 1.2: Install the FSM Control Plane
• 1.3: Upgrade the FSM Control Plane
• 1.4: Uninstall the FSM Control Plane and Components
• 1.5: Mesh configuration
• 1.6: Reconciler Guide
• 1.7: Extending FSM
• 2: Application onboarding
• 2.1: Prerequisites
• 2.2: Namespace addition
• 2.3: Sidecar Injection
• 2.4: Application Protocol Selection
• 3: Traffic Management
• 3.1: Permissive Mode
• 3.2: Traffic Redirection
• 3.2.1: Iptables Redirection
• 3.2.2: eBPF Redirection
• 3.3: Traffic Splitting
• 3.4: Circuit Breaking
• 3.5: Retry
• 3.6: Rate Limiting
• 3.7: Ingress
• 3.7.1: Ingress to Mesh
• 3.7.2: Service Loadbalancer
• 3.7.3: FSM Ingress Controller
• 3.7.3.1: Installation
• 3.7.3.2: Basics
• 3.7.3.3: Advanced TLS
• 3.7.3.4: TLS Passthrough
• 3.7.4: FSM Gateway
• 3.7.4.1: Installation
• 3.7.4.2: HTTP Routing
• 3.7.4.3: HTTP URL Rewrite
• 3.7.4.4: HTTP Redirect
• 3.7.4.5: HTTP Request Header Manipulate
• 3.7.4.6: HTTP Response Header Manipulate
• 3.7.4.7: TCP Routing
• 3.7.4.8: TLS Termination
• 3.7.4.9: TLS Passthrough
• 3.7.4.10: gRPC Routing
• 3.7.4.11: UDP Routing
• 3.7.4.12: Fault Injection
• 3.7.4.13: Access Control
• 3.7.4.14: Rate Limit
• 3.7.4.15: Retry
• 3.7.4.16: Session Sticky
• 3.7.4.17: Health Check
• 3.7.4.18: Loadbalancing Algorithm
• 3.7.4.19: Upstream TLS
• 3.7.4.20: Gateway mTLS
• 3.7.4.21: Traffic Mirroring
• 3.8: Egress
• 3.8.1: Egress
• 3.8.2: Egress Gateway
• 3.9: Multi-cluster services
• 4: Observability
• 4.1: Metrics
• 4.2: Tracing
• 4.3: Logs
• 5: Health Checks
• 5.1: Configure Health Probes
• 5.2: FSM Control Plane Health Probes
• 6: Integrations
• 6.1: Integrate Dapr with FSM
• 6.2: Integrate Prometheus with FSM
• 6.3: Microservice Discovery Integration
• 7: Security
• 7.1: Bi-directional mTLS
• 7.2: Access Control Management
• 7.3: Certificate Management
• 7.4: Traffic Access Control
• 8: Troubleshooting
• 8.1: Application Container Lifecycle
• 8.2: Error Codes
• 8.3: Prometheus
• 8.4: Grafana
• 8.5: Uninstall
• 8.6: Traffic Troubleshooting
• 8.6.1: Iptables Redirection
• 8.6.2: Permissive Traffic Policy Mode
• 8.6.3: Ingress
• 8.6.4: Egress Troubleshooting
• 9: Data plane benchmark
• 9.1: Service Mesh Data Plane Benchmark

1 - Operating guides

1.1 - Install the FSM CLI

Prerequisites

• Kubernetes cluster running Kubernetes v1.19.0 or greater

Set up the FSM CLI

From the Binary Releases

Download platform specific compressed package from the Releases page.

Unpack the fsm binary and add it to $PATH to get started.

Linux and macOS

In a bash-based shell on Linux/macOS or Windows Subsystem for Linux, use curl to download the FSM release and then extract with tar as follows:

# Specify the FSM version that will be leveraged throughout these instructions
FSM_VERSION=v1.2.3

# Linux curl command only
curl -sL "https://github.com/flomesh-io/fsm/releases/download/$FSM_VERSION/fsm-$FSM_VERSION-linux-amd64.tar.gz" | tar -vxzf -

# macOS curl command only
curl -sL "https://github.com/flomesh-io/fsm/releases/download/$FSM_VERSION/fsm-$FSM_VERSION-darwin-amd64.tar.gz" | tar -vxzf -

The fsm client binary runs on your client machine and allows you to manage FSM in your Kubernetes cluster. Use the following commands to install the FSM fsm client binary in a bash-based shell on Linux or Windows Subsystem for Linux. These commands copy the fsm client binary to the standard user program location in your PATH.

sudo mv ./linux-amd64/fsm /usr/local/bin/fsm

For macOS use the following commands:

sudo mv ./darwin-amd64/fsm /usr/local/bin/fsm

You can verify the fsm client library has been correctly added to your path and its version number with the following command.

fsm version

From Source (Linux, MacOS)

Building FSM from source requires more steps but is the best way to test the latest changes and useful in a development environment.

You must have a working Go environment and Helm 3 installed.

git clone https://github.com/flomesh-io/fsm.git
cd fsm
make build-fsm

make build-fsm will fetch any required dependencies, compile fsm and place it in bin/fsm. Add bin/fsm to $PATH so you can easily use fsm.

Install FSM

FSM Configuration

By default, the control plane components are installed into a Kubernetes Namespace called fsm-system and the control plane is given a unique identifier attribute mesh-name defaulted to fsm.

During installation, the Namespace and mesh-name can be configured through flags when using the fsm CLI or by editing the values file when using the helm CLI.

The mesh-name is a unique identifier assigned to an fsm-controller instance during install to identify and manage a mesh instance.

The mesh-name should follow RFC 1123 DNS Label constraints. The mesh-name must:

• contain at most 63 characters
• contain only lowercase alphanumeric characters or ‘-’
• start with an alphanumeric character
• end with an alphanumeric character

Using the FSM CLI

Use the fsm CLI to install the FSM control plane on to a Kubernetes cluster.

Run fsm install.

# Install fsm control plane components
fsm install                                                                    
fsm-preinstall[fsm-preinstall-4vb8n] Done
fsm-bootstrap[fsm-bootstrap-cdbccf694-nwm74] Done
fsm-injector[fsm-injector-7c9f5f9cdf-tw99v] Done
fsm-controller[fsm-controller-6d5984fb9f-2nj7s] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

Run fsm install --help for more options.

1.2 - Install the FSM Control Plane

Prerequisites

• Kubernetes cluster running Kubernetes v1.19.0 or greater
• The FSM CLI or the helm 3 CLI or the OpenShift oc CLI.

Kubernetes support

FSM can be run on Kubernetes versions that are supported at the time of the FSM release. The current support matrix is:

Using the FSM CLI

Use the fsm CLI to install the FSM control plane on to a Kubernetes cluster.

FSM CLI and Chart Compatibility

Each version of the FSM CLI is designed to work only with the matching version of the FSM Helm chart. Many operations may still work when some version skew exists, but those scenarios are not tested and issues that arise when using different CLI and chart versions may not get fixed even if reported.

Running the CLI

Run fsm install to install the FSM control plane.

fsm install
fsm-preinstall[fsm-preinstall-xsmz4] Done
fsm-bootstrap[fsm-bootstrap-7f59b7bf7-rs55z] Done
fsm-injector[fsm-injector-787bc867db-54gl6] Done
fsm-controller[fsm-controller-58d758b7fb-2zrr8] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

Run fsm install --help for more options.

Note: Installing FSM via the CLI enforces deploying only one mesh in the cluster. FSM installs and manages the CRDs by adding a conversion webhook field to all the CRDs to support multiple API versions, which ties the CRDs to a specific instance of FSM. Hence, for FSM’s correct operation it is strongly recommended to have only one FSM mesh per cluster.

Using the Helm CLI

The FSM chart can be installed directly via the Helm CLI.

Editing the Values File

You can configure the FSM installation by overriding the values file.

1. Create a copy of the values file (make sure to use the version for the chart you wish to install).

2. Change any values you wish to customize. You can omit all other values.

   • To see which values correspond to the MeshConfig settings, see the FSM MeshConfig documentation

   • For example, to set the logLevel field in the MeshConfig to info, save the following as override.yaml:

     fsm:
       sidecarLogLevel: info
     

Helm install

Then run the following helm install command. The chart version can be found in the Helm chart you wish to install here.

helm install <mesh name> fsm --repo https://flomesh-io.github.io/fsm --version <chart version> --namespace <fsm namespace> --create-namespace --values override.yaml

Omit the --values flag if you prefer to use the default settings.

Run helm install --help for more options.

OpenShift

To install FSM on OpenShift:

1. Enable privileged init containers so that they can properly program iptables. The NET_ADMIN capability is not sufficient on OpenShift.

   fsm install --set="fsm.enablePrivilegedInitContainer=true"
   

   • If you have already installed FSM without enabling privileged init containers, set enablePrivilegedInitContainer to true in the FSM MeshConfig and restart any pods in the mesh.

2. Add the privileged security context constraint to each service account in the mesh.

   • Install the oc CLI.

   • Add the security context constraint to the service account

      oc adm policy add-scc-to-user privileged -z <service account name> -n <service account namespace>
     

Pod Security Policy

Deprecated: PSP support has been deprecated in FSM since v0.10.0

PSP support will be removed in FSM 1.0.0

If you are running FSM in a cluster with PSPs enabled, pass in --set fsm.pspEnabled=true to your fsm install or helm install CLI command.

Enable Reconciler in FSM

If you wish to enable a reconciler in FSM, pass in --set fsm.enableReconciler=true to your fsm install or helm install CLI command. More information on the reconciler can be found in the Reconciler Guide.

Inspect FSM Components

A few components will be installed by default. Inspect them by using the following kubectl command:

# Replace fsm-system with the namespace where FSM is installed
kubectl get pods,svc,secrets,meshconfigs,serviceaccount --namespace fsm-system

A few cluster wide (non Namespaced components) will also be installed. Inspect them using the following kubectl command:

kubectl get clusterrolebinding,clusterrole,mutatingwebhookconfiguration,validatingwebhookconfigurations -l app.kubernetes.io/name=flomesh.io

Under the hood, fsm is using Helm libraries to create a Helm release object in the control plane Namespace. The Helm release name is the mesh-name. The helm CLI can also be used to inspect Kubernetes manifests installed in more detail. Goto https://helm.sh for instructions to install Helm.

# Replace fsm-system with the namespace where FSM is installed
helm get manifest fsm --namespace fsm-system

Next Steps

Now that the FSM control plane is up and running, add services to the mesh.

1.3 - Upgrade the FSM Control Plane

This guide describes how to upgrade the FSM control plane.

How upgrades work

FSM’s control plane lifecycle is managed by Helm and can be upgraded with Helm’s upgrade functionality, which will patch or replace control plane components as needed based on changed values and resource templates.

Resource availability during upgrade

Since upgrades may include redeploying the fsm-controller with the new version, there may be some downtime of the controller. While the fsm-controller is unavailable, there will be a delay in processing new SMI resources, creating new pods to be injected with a proxy sidecar container will fail, and mTLS certificates will not be rotated.

Already existing SMI resources will be unaffected, this means that the data plane (which includes the Pipy sidecar configs) will also be unaffected by upgrading.

Data plane interruptions are expected if the upgrade includes CRD changes. Streamlining data plane upgrades is being tracked in issue #512.

Policy

Only certain upgrade paths are tested and supported.

Note: These plans are tentative and subject to change.

Breaking changes in this section refer to incompatible changes to the following user-facing components:

• fsm CLI commands, flags, and behavior
• SMI CRDs and controllers

This implies the following are NOT user-facing and incompatible changes are NOT considered “breaking” as long as the incompatibility is handled by user-facing components:

• Chart values.yaml
• fsm-mesh-config MeshConfig
• Internally-used labels and annotations (monitored-by, injection, metrics, etc.)

Upgrades are only supported between versions that do not include breaking changes, as described below.

For FSM versions 0.y.z:

• Breaking changes will not be introduced between 0.y.z and 0.y.z+1
• Breaking changes may be introduced between 0.y.z and 0.y+1.0

For FSM versions x.y.z where x >= 1:

• Breaking changes will not be introduced between x.y.z and x.y+1.0 or between x.y.z and x.y.z+1
• Breaking changes may be introduced between x.y.z and x+1.0.0

How to upgrade FSM

The recommended way to upgrade a mesh is with the fsm CLI. For advanced use cases, helm may be used.

CRD Upgrades

Because Helm does not manage CRDs beyond the initial installation, FSM leverages an init-container on the fsm-bootstrap pod to to update existing and add new CRDs during an upgrade. If the new release contains updates to existing CRDs or adds new CRDs, the init-fsm-bootstrap on the fsm-bootstrap pod will update the CRDs. The associated Custom Resources will remain as is, requiring no additional action prior to or immediately after the upgrade.

Please check the CRD Updates section of the release notes to see if any updates have been made to the CRDs used by FSM. If the version of the Custom Resources are within the versions the updated CRD supports, no immediate action is required. FSM implements a conversion webhook for all of its CRDs, ensuring support for older versions and providing the flexibilty to update Custom Resources at a later point in time.

Upgrading with the FSM CLI

Pre-requisites

• Kubernetes cluster with the FSM control plane installed
  • Ensure that the Kubernetes cluster has the minimum Kubernetes version required by the new FSM chart. This can be found in the Installation Pre-requisites
• fsm CLI installed
  • By default, the fsm CLI will upgrade to the same chart version that it installs. e.g. v0.9.2 of the fsm CLI will upgrade to v0.9.2 of the FSM Helm chart. Upgrading to any other version of the Helm chart than the version matching the CLI may work, but those scenarios are not tested and issues that arise may not get fixed even if reported.

The fsm mesh upgrade command performs a helm upgrade of the existing Helm release for a mesh.

Basic usage requires no additional arguments or flags:

fsm mesh upgrade
FSM successfully upgraded mesh fsm

This command will upgrade the mesh with the default mesh name in the default FSM namespace. Values from the previous release will NOT carry over to the new release by default, but may be passed individually with the --set flag on fsm mesh upgrade.

See fsm mesh upgrade --help for more details

Upgrading with Helm

Pre-requisites

• Kubernetes cluster with the FSM control plane installed
• The helm 3 CLI

FSM Configuration

When upgrading, any custom settings used to install or run FSM may be reverted to the default, this only includes any metrics deployments. Please ensure that you carefully follow the guide to prevent these values from being overwritten.

To preserve any changes you’ve made to the FSM configuration, use the helm --values flag. Create a copy of the values file (make sure to use the version for the upgraded chart) and change any values you wish to customize. You can omit all other values.

**Note: Any configuration changes that go into the MeshConfig will not be applied during upgrade and the values will remain as is prior to the upgrade. If you wish to update any value in the MeshConfig you can do so by patching the resource after an upgrade.

For example, if the logLevel field in the MeshConfig was set to info prior to upgrade, updating this in override.yaml will during an upgrade will not cause any change.

Warning: Do NOT change fsm.meshName or fsm.fsmNamespace

Helm Upgrade

Then run the following helm upgrade command.

helm upgrade <mesh name> fsm --repo https://flomesh-io.github.io/fsm --version <chart version> --namespace <fsm namespace> --values override.yaml

Omit the --values flag if you prefer to use the default settings.

Run helm upgrade --help for more options.

Upgrading Third Party Dependencies

Pipy

Pipy versions can be updated by modifying the value of the sidecarImage variable in fsm-mesh-config. For example, to update Pipy image to latest (this is for example only, the latest image is not recommended), the next command should be run.

export fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl patch meshconfig fsm-mesh-config -n $fsm_namespace -p '{"spec":{"sidecar":{"sidecarImage": "flomesh/pipy:latest"}}}' --type=merge

After the MeshConfig resource has been updated, all Pods and deployments that are part of the mesh must be restarted so that the updated version of the Pipy sidecar can be injected onto the Pod as part of the automated sidecar injection performed by FSM. This can be done with the kubectl rollout restart deploy command.

Prometheus, Grafana, and Jaeger

If enabled, FSM’s Prometheus, Grafana, and Jaeger services are deployed alongside other FSM control plane components. Though these third party dependencies cannot be updated through the meshconfig like Pipy, the versions can still be updated in the deployment directly. For instance, to update prometheus to v2.19.1, the user can run:

export fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl set image deployment/fsm-prometheus -n $fsm_namespace prometheus="prom/prometheus:v2.19.1"

To update to Grafana 8.1.0, the command would look like:

kubectl set image deployment/fsm-grafana -n $fsm_namespace grafana="grafana/grafana:8.1.0"

And for Jaeger, the user would run the following to update to 1.26.0:

kubectl set image deployment/jaeger -n $fsm_namespace jaeger="jaegertracing/all-in-one:1.26.0"

FSM Upgrade Troubleshooting Guide

FSM Mesh Upgrade Timing Out

Insufficient CPU

If the fsm mesh upgrade command is timing out, it could be due to insufficient CPU.

1. Check the pods to see if any of them aren’t fully up and running

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl get pods -n fsm-system

2. If there are any pods that are in Pending state, use kubectl describe to check the Events section

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl describe pod <pod-name> -n fsm-system

If you see the following error, then please increase the number of CPUs Docker can use.

`Warning  FailedScheduling  4s (x15 over 19m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.`

Error Validating CLI Parameters

If the fsm mesh upgrade command is still timing out, it could be due to a CLI/Image Version mismatch.

1. Check the pods to see if any of them aren’t fully up and running

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl get pods -n fsm-system

2. If there are any pods that are in Pending state, use kubectl describe to check the Events section for Error Validating CLI parameters

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl describe pod <pod-name> -n fsm-system

3. If you find the error, please check the pod’s logs for any errors

kubectl logs -n fsm-system <pod-name> | grep -i error

If you see the following error, then it’s due to a CLI/Image Version mismatch.

`"error":"Please specify the init container image using --init-container-image","reason":"FatalInvalidCLIParameters"`

Workaround is to set the container-registry and fsm-image-tag flag when running fsm mesh upgrade.

fsm mesh upgrade --container-registry $CTR_REGISTRY --fsm-image-tag $CTR_TAG --enable-egress=true

Other Issues

If you’re running into issues that are not resolved with the steps above, please open a GitHub issue.

1.4 - Uninstall the FSM Control Plane and Components

This guide describes how to uninstall FSM from a Kubernetes cluster. This guide assumes there is a single FSM control plane (mesh) running. If there are multiple meshes in a cluster, repeat the process described for each control plane in the cluster before uninstalling any cluster wide resources at the end of the guide. Taking into consideration both the control plane and dataplane, this guide aims to walk through uninstalling all remnants of FSM with minimal downtime.

Prerequisites

• Kubernetes cluster with FSM installed
• The kubectl CLI
• The FSM CLI or the Helm 3 CLI

Remove Pipy Sidecars from Application Pods and Pipy Secrets

The first step to uninstalling FSM is to remove the Pipy sidecar containers from application pods. The sidecar containers enforce traffic policies. Without them, traffic will flow to and from Pods according in accordance with default Kubernetes networking unless there are Kubernetes Network Policies applied.

FSM Pipy sidecars and related secrets will be removed in the following steps:

1. Disable automatic sidecar injection
2. Restart pods

Disable Automatic Sidecar Injection

FSM Automatic Sidecar Injection is most commonly enabled by adding namespaces to the mesh via the fsm CLI. Use the fsm CLI to see which namespaces have sidecar injection enabled. If there are multiple control planes installed, be sure to specify the --mesh-name flag.

View namespaces in a mesh:

fsm namespace list --mesh-name=<mesh-name>
NAMESPACE          MESH           SIDECAR-INJECTION
<namespace1>       <mesh-name>    enabled
<namespace2>       <mesh-name>    enabled

Remove each namespace from the mesh:

fsm namespace remove <namespace> --mesh-name=<mesh-name>
Namespace [<namespace>] successfully removed from mesh [<mesh-name>]

This will remove the flomesh.io/sidecar-injection: enabled annotation and flomesh.io/monitored-by: <mesh name> label from the namespace.

Alternatively, if sidecar injection is enabled via annotations on pods instead of per namespace, please modify the pod or deployment spec to remove the sidecar injection annotation.

Restart Pods

Restart all pods running with a sidecar:

# If pods are running as part of a Kubernetes deployment
# Can use this strategy for daemonset as well
kubectl rollout restart deployment <deployment-name> -n <namespace>

# If pod is running standalone (not part of a deployment or replica set)
kubectl delete pod <pod-name> -n namespace
k apply -f <pod-spec> # if pod is not restarted as part of replicaset

Now, there should be no FSM Pipy sidecar containers running as part of the applications that were once part of the mesh. Traffic is no longer managed by the FSM control plane with the mesh-name used above. During this process, your applications may experience some downtime as all the Pods are restarting.

Uninstall FSM Control Plane and Remove User Provided Resources

The FSM control plane and related components will be uninstalled in the following steps:

• Prerequisites
• Remove Pipy Sidecars from Application Pods and Pipy Secrets
  • Disable Automatic Sidecar Injection
  • Restart Pods
• Uninstall FSM Control Plane and Remove User Provided Resources
  • Uninstall the FSM control plane
  • Remove User Provided Resources
  • Delete FSM Namespace
  • Removal of FSM Cluster Wide Resources

Uninstall the FSM control plane

Use the fsm CLI to uninstall the FSM control plane from a Kubernetes cluster. The following step will remove:

1. FSM controller resources (deployment, service, mesh config, and RBAC)
2. Prometheus, Grafana, Jaeger, and Fluent Bit resources installed by FSM
3. Mutating webhook and validating webhook
4. The conversion webhook fields patched by FSM to the CRDs installed/required by FSM: CRDs for FSM will be unpatched. To delete cluster wide resources refer to Removal of FSM Cluster Wide Resources for more details.

Run fsm uninstall mesh:

# Uninstall fsm control plane components
fsm uninstall mesh --mesh-name=<mesh-name>
Uninstall FSM [mesh name: <mesh-name>] ? [y/n]: y
FSM [mesh name: <mesh-name>] uninstalled

Run fsm uninstall mesh --help for more options.

Alternatively, if you used Helm to install the control plane, run the following helm uninstall command:

helm uninstall <mesh name> --namespace <fsm namespace>

Run helm uninstall --help for more options.

Remove User Provided Resources

If any resources were provided or created for FSM at install time, they can be deleted at this point.

For example, if Hashicorp Vault was deployed for the sole purpose of managing certificates for FSM, all related resources can be deleted.

Delete FSM Namespace

When installing a mesh, the fsm CLI creates the namespace the control plane is installed into if it does not already exist. However, when uninstalling the same mesh, the namespace it lives in does not automatically get deleted by the fsm CLI. This behavior occurs because there may be resources a user created in the namespace that they may not want automatically deleted.

If the namespace was only used for FSM and there is nothing that needs to be kept around, the namespace can be deleted at the time of uninstall or later using the following command.

fsm uninstall mesh --delete-namespace

Warning: Only delete the namespace if resources in the namespace are no longer needed. For example, if fsm was installed in kube-system, deleting the namespace may delete important cluster resources and may have unintended consequences.

Removal of FSM Cluster Wide Resources

On installation FSM ensures that all the CRDs mentioned here exist in the cluster at install time. During installation, if they are not already installed, the fsm-bootstrap pod will install them before the rest of the control plane components are running. This is the same behavior when using the Helm charts to install FSM as well.

Uninstalling the mesh in both unmanaged and managed environments:

1. removes FSM control plane components, including control plane pods
2. removes/un-patches the conversion webhook fields from all the CRDs (which FSM adds to support multiple CR versions)

leaving behind certain FSM resources to prevent unintended consequences for the cluster after uninstalling FSM.The resources that are left behind will depend on whether FSM was uninstalled from a managed or unmanaged cluster environment.

When uninstalling FSM, both the fsm uninstall mesh command and Helm uninstallation will not delete any FSM or SMI CRD in any cluster environment (managed and unmanaged) for primarily two reasons:

1. CRDs are cluster-wide resources and may be used by other service meshes or resources running in the same cluster
2. deletion of a CRD will cause all custom resources corresponding to that CRD to also be deleted

To remove cluster wide resources that FSM installs (i.e. the meshconfig, secrets, FSM CRDs, SMI CRDs, and webhook configurations), the following command can be run during or after FSM’s uninstillation.

fsm uninstall mesh --delete-cluster-wide-resources

Warning: Deletion of a CRD will cause all custom resources corresponding to that CRD to also be deleted.

To troubleshoot FSM uninstallation, refer to the uninstall troubleshooting section

1.5 - Mesh configuration

FSM deploys a MeshConfig resource fsm-mesh-config as a part of its control plane (in the same namespace as that of the fsm-controller pod) which can be updated by the mesh owner/operator at any time. The purpose of this MeshConfig is to provide the mesh owner/operator the ability to update some of the mesh configurations based on their needs.

At the time of install, the FSM MeshConfig is deployed from a preset MeshConfig (preset-mesh-config) which can be found under charts/fsm/templates.

First, set an environment variable to refer to the namespace where fsm was installed.

export FSM_NAMESPACE=fsm-system # Replace fsm-system with the namespace where FSM is installed

To view your fsm-mesh-config in CLI use the kubectl get command.

kubectl get meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -o yaml

Note: Values in the MeshConfig fsm-mesh-config are persisted across upgrades.

Configure FSM MeshConfig

Kubectl Patch Command

Changes to fsm-mesh-config can be made using the kubectl patch command.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":true}}}'  --type=merge

Refer to the Config API reference for more information.

If an incorrect value is used, validations on the MeshConfig CRD will prevent the change with an error message explaining why the value is invalid.

For example, the below command shows what happens if we patch enableEgress to a non-boolean value.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":"no"}}}'  --type=merge
# Validations on the CRD will deny this change
The MeshConfig "fsm-mesh-config" is invalid: spec.traffic.enableEgress: Invalid value: "string": spec.traffic.enableEgress in body must be of type boolean: "string"

Kubectl Patch Command for Each Key Type

Note: <fsm-namespace> refers to the namespace where the fsm control plane is installed. By default, the fsm namespace is fsm-system.

1.6 - Reconciler Guide

This guide describes how to enable the reconciler in FSM.

How the reconciler works

The goal of building a reconciler in FSM is to ensure resources required for the correct operation of FSM’s control plane are in their desired state at all times. Resources that are installed as a part of FSM install and have the labels flomesh.io/reconcile: true and app.kubernetes.io/name: flomesh.io will be reconciled by the reconciler.

Note: The reconciler will not operate as desired if the lables flomesh.io/reconcile: true and app.kubernetes.io/name: flomesh.io are modified or deleted on the reconcilable resources.

An update or delete event on the reconcilable resources will trigger the reconciler and it will reconcile the resource back to its desired state. Only metadata changes (excluding a name change) will be permitted on the reconcilable resources.

Resources reconciled

The resources that FSM reconciles are:

• CRDs : The CRDs installed/required by FSM CRDs for FSM will be reconciled. Since FSM manages the installation and upgrade of the CRDs it needs, FSM will also reconcile them to ensure that their spec, stored and served verions are always in the state that is required by FSM.
• MutatingWebhookConfiguration : A MutatingWebhookConfiguration is deployed as a part of FSM’s control plane to enable automatic sidecar injection. As this is a very critical component for pods joining the mesh, FSM reconciles this resource.
• ValidatingWebhookConfiguration : A ValidatingWebhookConfiguration is deployed as a part of FSM’s control plane to validate various mesh configurations. This resources validates configurations being applied to the mesh, hence FSM will reconcile this resource.

How to install FSM with the reconciler

To install FSM with the reconciler, use the below command:

fsm install --set fsm.enableReconciler=true
fsm-preinstall[fsm-preinstall-zqmxm] Done
fsm-bootstrap[fsm-bootstrap-7f59b7bf7-vf96p] Done
fsm-injector[fsm-injector-787bc867db-m5wxk] Done
fsm-controller[fsm-controller-58d758b7fb-46v4k] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

1.7 - Extending FSM

Extending FSM with Plugin Interface

In the latest 1.3.0 version of Flomesh service mesh FSM, we have introduced a significant feature: Plugin. This feature aims to provide developers with a way to extend the functionality of the service mesh without changing the FSM itself.

Nowadays, service mesh seems to be developing in two directions. One is like Istio, which provides a lot of ready-to-use functions and is very rich in features. The other like Linkerd, Flomesh FSM, and others that uphold the principle of simplicity and provide a minimum functional set that meets the user’s needs. There is no superiority or inferiority between the two: the former is rich in features but inevitably has the additional overhead of proxy, not only in resource consumption but also in the cost of learning and maintenance; the latter is easy to learn and use, consumes fewer resources, but the provided functions might not be enough for the immediate need of user desired functionality.

It is not difficult to imagine that the ideal solution is the low cost of the minimum functional set + the flexibility of scalability. The core of the service mesh is in the data plane, and the flexibility of scalability requires a high demand for the physique of the sidecar proxy. This is also why the Flomesh service mesh chose programmable proxy Pipy as the sidecar proxy.

Pipy is a programmable network proxy for cloud, edge, and IoT. It is flexible, fast, small, programmable, and open-source. The modular design of Pipy provides a large number of reusable filters that can be assembled into pipelines to process network data. Pipy provides a set of api and small usable filters to achieve business objectives while hiding the underlying details. Additionally, Pipy scripts (programming code that implements functional logic) can be dynamically delivered to Pipy instances over the network, enabling the proxy to be extended with new features without the need for compilation or restart.

Flomesh FSM extension solution

FSM provides three new CRDs for extensibility:

• Plugin: The plugin contains the code logic for the new functionality. The default functions provided by FSM are also available as plugins, but not in the form of a Plugin resource. These plugins can be adjusted through the Helm values file when installing FSM. For more information, refer to the built-in plugin list in the Helm values.yaml file.
• PluginChain: The plugin chain is the execution of plugins in sequence. The system provides four plugin chains: inbound-tcp, inbound-http, outbound-tcp, outbound-http. They correspond to the OSI layer-4 and layer-7 processing stages of inbound and outbound traffic, respectively.
• PluginConfig: The plugin configuration provides the configuration required for the plugin logic to run, which will be sent to the FSM sidecar proxy in JSON format.

For detailed information on plugin CRDs, refer to the Plugin API document.

Built-in variables

Below is a list of built-in PipyJS variables which can be imported into your custom plugins via PipyJS import keyword.

Demo

For a simple demonstration of how to extend FSM via Plugins, refer to below demo:

• Adding Identity and Access Management functionality

2 - Application onboarding

The following guide describes how to onboard a Kubernetes microservice to an FSM instance.

1. Refer to the application requirements guide before onboarding applications.

2. Configure and Install Service Mesh Interface (SMI) policies

   FSM conforms to the SMI specification. By default, FSM denies all traffic communications between Kubernetes services unless explicitly allowed by SMI policies. This behavior can be overridden with the --set=fsm.enablePermissiveTrafficPolicy=true flag on the fsm install command, allowing SMI policies not to be enforced while allowing traffic and services to still take advantage of features such as mTLS-encrypted traffic, metrics, and tracing.

   For example SMI policies, please see the following examples:

   • demo/deploy-traffic-specs.sh
   • demo/deploy-traffic-split.sh
   • demo/deploy-traffic-target.sh

3. If an application in the mesh needs to communicate with the Kubernetes API server, the user needs to explicitly allow this either by using IP range exclusion or by creating an egress policy as outlined below.

   First get the Kubernetes API server cluster IP:

   $ kubectl get svc -n default
   NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
   kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   1d
   

   Option 1: add the Kubernetes API server’s address to the list of Global outbound IP ranges for exclusion. The IP address could be a cluster IP address or a public IP address and should be appropriately excluded for connectivity to the Kubernetes API server.

   Add this IP to the MeshConfig so that outbound traffic to it is excluded from interception by FSM’s sidecar:

   $ kubectl patch meshconfig fsm-mesh-config -n <fsm-namespace> -p '{"spec":{"traffic":{"outboundIPRangeExclusionList":["10.0.0.1/32"]}}}'  --type=merge
   meshconfig.config.flomesh.io/fsm-mesh-config patched
   

   Restart the relevant pods in monitored namespaces for this change to take effect.

   Option 2: apply an Egress policy to allow access to the Kubernetes API server over HTTPS

   Note: when using an Egress policy, the Kubernetes API service must not be in a namespace that FSM manages

   1. Enable egress policy if not enabled:

   kubectl patch meshconfig fsm-mesh-config -n <fsm-namespace> -p '{"spec":{"featureFlags":{"enableEgressPolicy":true}}}'  --type=merge
   

   2. Apply an Egress policy to allow the application’s ServiceAccount to access the Kubernetes API server cluster IP found above. For example:

   kubectl apply -f - <<EOF
   kind: Egress
   apiVersion: policy.flomesh.io/v1alpha1
   metadata:
       name: k8s-server-egress
       namespace: test
   spec:
       sources:
       - kind: ServiceAccount
         name: <app pod's service account name>
         namespace: <app pod's service account namespace>
       ipAddresses:
       - 10.0.0.1/32
       ports:
       - number: 443
         protocol: https
   EOF
   

4. Onboard Kubernetes Namespaces to FSM

   To onboard a namespace containing applications to be managed by FSM, run the fsm namespace add command:

   $ fsm namespace add <namespace> --mesh-name <mesh-name>
   

   By default, the fsm namespace add command enables automatic sidecar injection for pods in the namespace.

   To disable automatic sidecar injection as a part of enrolling a namespace into the mesh, use fsm namespace add <namespace> --disable-sidecar-injection. Once a namespace has been onboarded, pods can be enrolled in the mesh by configuring automatic sidecar injection. See the Sidecar Injection document for more details.

5. Deploy new applications or redeploy existing applications

   By default, new deployments in onboarded namespaces are enabled for automatic sidecar injection. This means that when a new Pod is created in a managed namespace, FSM will automatically inject the sidecar proxy to the Pod. Existing deployments need to be restarted so that FSM can automatically inject the sidecar proxy upon Pod re-creation. Pods managed by a Deployment can be restarted using the kubectl rollout restart deploy command.

   In order to route protocol specific traffic correctly to service ports, configure the application protocol to use. Refer to the application protocol selection guide to learn more.

Note: Removing Namespaces

Namespaces can be removed from the FSM mesh with the fsm namespace remove command:

fsm namespace remove <namespace>

Please Note: The fsm namespace remove command only tells FSM to stop applying updates to the sidecar proxy configurations in the namespace. It does not remove the proxy sidecars. This means the existing proxy configuration will continue to be used, but it will not be updated by the FSM control plane. If you wish to remove the proxies from all pods, remove the pods’ namespaces from the FSM mesh with the CLI and reinstall all the pod workloads.

2.1 - Prerequisites

Security Contexts

• Do not run applications with the user ID (UID) value of 1500. This is reserved for the Pipy proxy sidecar container injected into pods by FSM’s sidecar injector.
• If security context runAsNonRoot is set to true at the pod level, a runAsUser value must be provided either for the pod or for each container. For example:

    securityContext:
      runAsNonRoot: true
      runAsUser: 1200
  

  If the UID is omitted, application containers may attempt to run as root user by default, causing conflict with the pod’s security context.
• Additional capabilities are not required.

Note: the FSM init container is programmed to run as root and add capability NET_ADMIN as it requires these security contexts to finish scheduling. These values are not changed by application security contexts.

Ports

Do not use the following ports as they are used by the Pipy sidecar.

2.2 - Namespace addition

Overview

When setting up an FSM control plane (also referred to as a “mesh”), one can also enroll a set of Kubernetes namespaces to the mesh. Enrolling a namespace to FSM allows FSM to monitor the resources within that Namespace whether they be applications deployed in Pods, Services, or even traffic policies represented as SMI resources.

Only one mesh can monitor a namespace, so this is something to watch out for when there are multiple instances of FSM within the same Kubernetes cluster. When applying policies to applications, FSM will only assess resources in either monitored namespaces so it is important to enroll namespaces where your applications are deployed to the correct instance of FSM with the correct mesh name. Enrolling a namespace also optionally allows for metrics to be collected for resources in the given namespace and for Pods in the namespace to be automatically injected with sidecar proxy containers. These are all features that help FSM provide functionality for traffic management and observability. Scoping this functionality at the namespace level allows teams to organize which segments of their cluster should be part of which mesh.

Namespace monitoring, automatic sidecar injection, and metrics collection is controlled by adding certain labels and annotations to a Kubernetes namespace. This can be done manually or using the fsm CLI although using the fsm CLI is the recommended approach. The presence of the label flomesh.io/monitored-by=<mesh-name> allows an FSM control plane with the given mesh-name to monitor all resources within that namespace. The annotation flomesh.io/sidecar-injection=enabled enables FSM to automatically inject sidecar proxy containers in all Pods created within that namespace. The metrics annotation flomesh.io/metrics=enabled allows FSM to collect metrics on resources within a Namespace.

See how to use the FSM CLI to manage namespace monitoring below.

Adding a Namespace to the FSM Control Plane

Add a namespace for monitoring and sidecar injection to the mesh with the following command:

fsm namespace add <namespace>

Explicitly disable sidecar injection while adding the namespace using --disable-sidecar-injection flag as shown here.

Remove a Namespace from the FSM control plane

Remove a namespace from being monitored by the mesh and disable sidecar injection with the following command:

fsm namespace remove <namespace>

This command will remove the FSM specific labels and annotations on the namespace thus removing it from the mesh.

Enable Metrics for a Namespace

fsm metrics enable --namespace <namespace>

Ignore a Namespace

There may be namespaces in a cluster that should never be part of a mesh. To explicity exclude a namespace from FSM:

fsm namespace ignore <namespace>

List Namespaces Part of a Mesh

To list namespaces within a specific mesh:

fsm namespace list --mesh-name=<mesh-name>

Troubleshooting Guide

Policy Issues

If you’re not seeing changes in SMI policies being applied to resources in a namespace, ensure the namespace is enrolled in the correct mesh:

fsm namespace list --mesh-name=<mesh-name>

NAMESPACE         MESH   SIDECAR-INJECTION
<namespace>       fsm    enabled

If the namespace does not show up, check the labels on the namespace using kubectl:

kubectl get namespace <namespace> --show-labels

NAME          STATUS   AGE   LABELS
<namespace>   Active   36s   flomesh.io/monitored-by=<mesh-name>

If the label value is not the expected mesh-name, remove the namespace from the mesh and add it back using the correct mesh-name.

fsm namespace remove <namespace> --mesh-name=<current-mesh-name>
fsm namespace add <namespace> --mesh-name=<expected-mesh-name>

If the monitored-by label is not present, it was either not added to the mesh or there was an error when adding it to the mesh. Add the namespace to the mesh either with the fsm CLI or using kubectl:

fsm namespace add <namespace> --mesh-name=<mesh-name>

kubectl label namespace <namespace> flomesh.io/monitored-by=<mesh-name>

Issues with Automatic Sidecar Injection

If you’re not seeing your Pods being automatically injected with sidecar containers, ensure that sidecar injection is enabled:

fsm namespace list --mesh-name=<mesh-name>

NAMESPACE         MESH   SIDECAR-INJECTION
<namespace>       fsm    enabled

If the namespace does not show up, check the annotations on the namespace using kubectl:

kubectl get namespace <namespace> -o=jsonpath='{.metadata.annotations.flomesh\.io\/sidecar-injection}'

If the output is anything other than enabled, either add namespace using the fsm CLI or add the annotation with kubectl:

fsm namespace add <namespace> --mesh-name=<mesh-name> --disable-sidecar-injection=false

kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=enabled --overwrite

Issues with Metrics Collection

If you’re not seeing metrics for resources in a particular namespace, ensure metrics are enabled:

kubectl get namespace <namespace> -o=jsonpath='{.metadata.annotations.flomesh\.io\/metrics}'

If the output is anything other than enabled, enable the namespace usng the fsm CLI or add the annotation with kubectl:

fsm metrics enable --namespace <namespace>

kubectl annotate namespace <namespace> flomesh.io/metrics=enabled --overwrite

Other Issues

If you’re running into issues that have not been resolved with the debugging techniques above, please open a GitHub issue on the repository.

2.3 - Sidecar Injection

Services participating in the service mesh communicate via sidecar proxies installed on pods backing the services. The following sections describe the sidecar injection workflow in FSM.

Automatic Sidecar Injection

Automatic sidecar injection is currently the only way to inject sidecars into the service mesh. Sidecars can be automatically injected into applicable Kubernetes pods using a mutating webhook admission controller provided by FSM.

Automatic sidecar injection can be configured per namespace as a part of enrolling a namespace into the mesh, or later using the Kubernetes API. Automatic sidecar injection can be enabled either on a per namespace or per pod basis by annotating the namespace or pod resource with the sidecar injection annotation. Individual pods and namespaces can be explicitly configured to either enable or disable automatic sidecar injection, giving users the flexibility to control sidecar injection on pods and namespaces.

Enabling Automatic Sidecar Injection

Prerequisites:

• The namespace to which the pods belong must be a monitored namespace that is added to the mesh using the fsm namespace add command.
• The namespace to which the pods belong must not be set to be ignored using the fsm namespace ignore command.
• The namespace to which the pods belong must not have a label with key name and value corresponding to the FSM control plane namespace. For example, a namespace with a label name: fsm-system where fsm-system is the control plane namespace cannot have sidecar injection enabled for pods in this namespace.
• The pod must not have hostNetwork: true in the pod spec. Pods with hostNetwork: true are not injected with a sidecar since doing so can result in routing failures in the host network.

Automatic Sidecar injection can be enabled in the following ways:

• While enrolling a namespace into the mesh using fsm cli: fsm namespace add <namespace>: Automatic sidecar injection is enabled by default with this command.

• Using kubectl to annotate individual namespaces and pods to enable sidecar injection:

  # Enable sidecar injection on a namespace
  $ kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=enabled
  

  # Enable sidecar injection on a pod
  $ kubectl annotate pod <pod> flomesh.io/sidecar-injection=enabled
  

• Setting the sidecar injection annotation to enabled in the Kubernetes resource spec for a namespace or pod:

  metadata:
    name: test
    annotations:
      'flomesh.io/sidecar-injection': 'enabled'
  

  Pods will be injected with a sidecar ONLY if the following conditions are met:

  1. The namespace to which the pod belongs is a monitored namespace.
  2. The pod is explicitly enabled for the sidecar injection, OR the namespace to which the pod belongs is enabled for the sidecar injection and the pod is not explicitly disabled for sidecar injection.

Explicitly Disabling Automatic Sidecar Injection on Namespaces

Namespaces can be disabled for automatic sidecar injection in the following ways:

• While enrolling a namespace into the mesh using fsm cli: fsm namespace add <namespace> --disable-sidecar-injection: If the namespace was previously enabled for sidecar injection, it will be disabled after running this command.

• Using kubectl to annotate individual namespaces to disable sidecar injection:

  # Disable sidecar injection on a namespace
  $ kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=disabled
  

Explicitly Disabling Automatic Sidecar Injection on Pods

Individual pods can be explicitly disabled for sidecar injection. This is useful when a namespace is enabled for sidecar injection but specific pods should not be injected with sidecars.

• Using kubectl to annotate individual pods to disable sidecar injection:

  # Disable sidecar injection on a pod
  $ kubectl annotate pod <pod> flomesh.io/sidecar-injection=disabled
  

• Setting the sidecar injection annotation to disabled in the Kubernetes resource spec for the pod:

  metadata:
    name: test
    annotations:
      'flomesh.io/sidecar-injection': 'disabled'
  

Automatic sidecar injection is implicitly disabled for a namespace when it is removed from the mesh using the fsm namespace remove command.

2.4 - Application Protocol Selection

FSM is capable of routing different application protocols such as HTTP, TCP, and gRPC differently. The following guide describes how to configure service ports to specify the application protocol to use for traffic filtering and routing.

Configuring the application protocol

Kubernetes services expose one or more ports. A port exposed by an application running the service can serve a specific application protocol such as HTTP, TCP, gRPC etc. Since FSM filters and routes traffic for different application protocols differently, a configuration on the Kubernetes service object is necessary to convey to FSM how traffic directed to a service port must be routed.

In order to determine the application protocol served by a service’s port, FSM expects the appProtocol field on the service’s port to be set.

FSM supports the following application protocols for service ports:

1. http: For HTTP based filtering and routing of traffic
2. tcp: For TCP based filtering and routing of traffic
3. tcp-server-first: For TCP based filtering and routing of traffic where the server initiates communication with a client, such as mySQL, PostgreSQL, and others
4. gRPC: For HTTP2 based filtering and routing of gRPC traffic

The application protocol configuration described is applicable to both SMI and Permissive traffic policy modes.

Examples

Consider the following SMI traffic access and traffic specs policies:

• A TCPRoute resource named tcp-route that specifies the port TCP traffic should be allowed on.
• An HTTPRouteGroup resource named http-route that specifies the HTTP routes for which HTTP traffic should be allowed.
• A TrafficTarget resource named test that allows pods in the service account sa-2 to access pods in the service account sa-1 for the specified TCP and HTTP rules.

kind: TCPRoute
metadata:
  name: tcp-route
spec:
  matches:
    ports:
    - 8080
---
kind: HTTPRouteGroup
metadata:
  name: http-route
spec:
  matches:
  - name: version
    pathRegex: "/version"
    methods:
    - GET
---
kind: TrafficTarget
metadata:
  name: test
  namespace: default
spec:
  destination:
    kind: ServiceAccount
    name: sa-1 # There are 2 services under this service account:  service-1 and service-2
    namespace: default
  rules:
  - kind: TCPRoute
    name: tcp-route
  - kind: HTTPRouteGroup
    name: http-route
  sources:
  - kind: ServiceAccount
    name: sa-2
    namespace: default

Kubernetes service resources should explicitly specify the application protocol being served by the service’s ports using the appProtocol field.

A service service-1 backed by a pod in service account sa-1 serving http application traffic should be defined as follows:

kind: Service
metadata:
  name: service-1
  namespace: default
spec:
  ports:
  - port: 8080
    name: some-port
    appProtocol: http

A service service-2 backed by a pod in service account sa-1 serving raw tcp application traffic shold be defined as follows:

kind: Service
metadata:
  name: service-2
  namespace: default
spec:
  ports:
  - port: 8080
    name: some-port
    appProtocol: tcp

3 - Traffic Management

3.1 - Permissive Mode

Permissive traffic policy mode in FSM is a mode where SMI traffic access policy enforcement is bypassed. In this mode, FSM automatically discovers services that are a part of the service mesh and programs traffic policy rules on each Pipy proxy sidecar to be able to communicate with these services.

When to use permissive traffic policy mode

Since permissive traffic policy mode bypasses SMI traffic access policy enforcement, it is suitable for use when connectivity between applications within the service mesh should flow as before the applications were enrolled into the mesh. This mode is suitable in environments where explicitly defining traffic access policies for connectivity between applications is not feasible.

A common use case to enable permissive traffic policy mode is to support gradual onboarding of applications into the mesh without breaking application connectivity. Traffic routing between application services is automatically set up by FSM controller through service discovery. Wildcard traffic policies are set up on each Pipy proxy sidecar to allow traffic flow to services within the mesh.

The alternative to permissive traffic policy mode is SMI traffic policy mode, where traffic between applications is denied by default and explicit SMI traffic policies are necessary to allow application connectivity. When policy enforcement is necessary, SMI traffic policy mode must be used instead.

Configuring permissive traffic policy mode

Permissive traffic policy mode can be enabled or disabled at the time of FSM install, or after FSM has been installed.

Enabling permissive traffic policy mode

Enabling permissive traffic policy mode implicitly disables SMI traffic policy mode.

During FSM install using the --set flag:

fsm install --set fsm.enablePermissiveTrafficPolicy=true

After FSM has been installed:

# Assumes FSM is installed in the fsm-system namespace
kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}'  --type=merge

Disabling permissive traffic policy mode

Disabling permissive traffic policy mode implicitly enables SMI traffic policy mode.

During FSM install using the --set flag:

fsm install --set fsm.enablePermissiveTrafficPolicy=false

After FSM has been installed:

# Assumes FSM is installed in the fsm-system namespace
kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}'  --type=merge

How it works

When permissive traffic policy mode is enabled, FSM controller discovers all services that are a part of the mesh and programs wildcard traffic routing rules on each Pipy proxy sidecar to reach every other service in the mesh. Additionally, each proxy fronting workloads that are associated with a service is configured to accept all traffic destined to the service. Depending on the application protocol of the service (HTTP, TCP, gRPC etc.), appropriate traffic routing rules are configured on the Pipy sidecar to allow all traffic for that particular type.

Refer to the Permissive traffic policy mode demo to learn more.

Pipy configurations

In permissive mode, FSM controller programs wildcard routes for client applications to communicate with services. Following are the Pipy inbound and outbound filter and route configuration snippets from the curl and httpbin sidecar proxies.

1. Outbound Pipy configuration on the curl client pod:

   Outbound HTTP filter chain corresponding to the httpbin service:

    {
     "Outbound": {
       "TrafficMatches": {
         "14001": [
           {
             "DestinationIPRanges": [
               "10.43.103.59/32"
             ],
             "Port": 14001,
             "Protocol": "http",
             "HttpHostPort2Service": {
               "httpbin": "httpbin.app.svc.cluster.local",
               "httpbin.app": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc.cluster": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc.cluster.local": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc.cluster.local:14001": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc.cluster:14001": "httpbin.app.svc.cluster.local",
               "httpbin.app.svc:14001": "httpbin.app.svc.cluster.local",
               "httpbin.app:14001": "httpbin.app.svc.cluster.local",
               "httpbin:14001": "httpbin.app.svc.cluster.local"
             },
             "HttpServiceRouteRules": {
               "httpbin.app.svc.cluster.local": {
                 ".*": {
                   "Headers": null,
                   "Methods": null,
                   "TargetClusters": {
                     "app/httpbin|14001": 100
                   },
                   "AllowedServices": null
                 }
               }
             },
             "TargetClusters": null,
             "AllowedEgressTraffic": false,
             "ServiceIdentity": "default.app.cluster.local"
           }
         ]
       }
     }
   }
   

   Outbound route configuration:

   "HttpServiceRouteRules": {
           "httpbin.app.svc.cluster.local": {
             ".*": {
               "Headers": null,
               "Methods": null,
               "TargetClusters": {
                 "app/httpbin|14001": 100
               },
               "AllowedServices": null
             }
           }
         }
   

2. Inbound Pipy configuration on the httpbin service pod:

   Inbound HTTP filter chain corresponding to the httpbin service:

   {
     "Inbound": {
       "TrafficMatches": {
         "14001": {
           "SourceIPRanges": null,
           "Port": 14001,
           "Protocol": "http",
           "HttpHostPort2Service": {
             "httpbin": "httpbin.app.svc.cluster.local",
             "httpbin.app": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc.cluster": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc.cluster.local": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc.cluster.local:14001": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc.cluster:14001": "httpbin.app.svc.cluster.local",
             "httpbin.app.svc:14001": "httpbin.app.svc.cluster.local",
             "httpbin.app:14001": "httpbin.app.svc.cluster.local",
             "httpbin:14001": "httpbin.app.svc.cluster.local"
           },
           "HttpServiceRouteRules": {
             "httpbin.app.svc.cluster.local": {
               ".*": {
                 "Headers": null,
                 "Methods": null,
                 "TargetClusters": {
                   "app/httpbin|14001|local": 100
                 },
                 "AllowedServices": null
               }
             }
           },
           "TargetClusters": null,
           "AllowedEndpoints": null
         }
       }
     }
   }
   

   Inbound route configuration:

   "HttpServiceRouteRules": {
     "httpbin.app.svc.cluster.local": {
       ".*": {
         "Headers": null,
         "Methods": null,
         "TargetClusters": {
           "app/httpbin|14001|local": 100
         },
         "AllowedServices": null
       }
     }
   }
   

3.2 - Traffic Redirection

iptables is a traffic interception tool based on the Linux kernel. It can control traffic by filtering rules. Its advantages include:

• Universality: The iptables tool has been widely used in Linux operating systems, so most Linux users are familiar with its usage.
• Stability: iptables has long been part of the Linux kernel, so it has a high degree of stability.
• Flexibility: iptables can be flexibly configured according to needs to control network traffic.

However, iptables also has some disadvantages:

• Difficult to debug: Due to the complexity of the iptables tool itself, it is relatively difficult to debug.
• Performance issues: Unpredictable latency and reduced performance as the number of services grows.
• Issues with handling complex traffic: When it comes to handling complex traffic, iptables may not be suitable because its rule processing is not flexible enough.

eBPF is an advanced traffic interception tool that can intercept and analyze traffic in the Linux kernel through custom programs. The advantages of eBPF include:

• Flexibility: eBPF can use custom programs to intercept and analyze traffic, so it has higher flexibility.
• Scalability: eBPF can dynamically load and unload programs, so it has higher scalability.
• Efficiency: eBPF can perform processing in the kernel space, so it has higher performance.

However, eBPF also has some disadvantages:

• Higher learning curve: eBPF is relatively new compared to iptables, so it requires some learning costs.
• Complexity: Developing custom eBPF programs may be more complex.

Overall, iptables is more suitable for simple traffic filtering and management, while eBPF is more suitable for complex traffic interception and analysis scenarios that require higher flexibility and performance.

3.2.1 - Iptables Redirection

FSM leverages iptables to intercept and redirect traffic to and from pods participating in the service mesh to the Pipy proxy sidecar container running on each pod. Traffic redirected to the Pipy proxy sidecar is filtered and routed based on service mesh traffic policies.

For more details of comparison between iptables and eBPF, you can refer to Traffic Redirection.

How it works

FSM sidecar injector service fsm-injector injects an Pipy proxy sidecar on every pod created within the service mesh. Along with the Pipy proxy sidecar, fsm-injector also injects an init container, a specialized container that runs before any application containers in a pod. The injected init container is responsible for bootstrapping the application pods with traffic redirection rules such that all outbound TCP traffic from a pod and all inbound traffic TCP traffic to a pod are redirected to the pipy proxy sidecar running on that pod. This redirection is set up by the init container by running a set of iptables commands.

Ports reserved for traffic redirection

FSM reserves a set of port numbers to perform traffic redirection and provide admin access to the Pipy proxy sidecar. It is essential to note that these port numbers must not be used by application containers running in the mesh. Using any of these reserved port numbers will lead to the Pipy proxy sidecar not functioning correctly.

Following are the port numbers that are reserved for use by FSM:

1. 15000: used by the Pipy admin interface exposed over localhost to return current configuration files.
2. 15001: used by the Pipy outbound listener to accept and proxy outbound traffic sent by applications within the pod
3. 15003: used by the Pipy inbound listener to accept and proxy inbound traffic entering the pod destined to applications within the pod
4. 15010: used by the Pipy inbound Prometheus listener to accept and proxy inbound traffic pertaining to scraping Pipy’s Prometheus metrics
5. 15901: used by Pipy to serve rewritten HTTP liveness probes
6. 15902: used by Pipy to serve rewritten HTTP readiness probes
7. 15903: used by Pipy to serve rewritten HTTP startup probes

The following are the port numbers that are reserved for use by FSM and allow traffic to bypass Pipy:

1. 15904: used by fsm-healthcheck to serve tcpSocket health probes rewritten to httpGet health probes

Application User ID (UID) reserved for traffic redirection

FSM reserves the user ID (UID) value 1500 for the Pipy proxy sidecar container. This user ID is of utmost importance while performing traffic interception and redirection to ensure the redirection does not result in a loop. The user ID value 1500 is used to program redirection rules to ensure redirected traffic from Pipy is not redirected back to itself!

Application containers must not used the reserved user ID value of 1500.

Types of traffic intercepted

Currently, FSM programs the Pipy proxy sidecar on each pod to only intercept inbound and outbound TCP traffic. This includes raw TCP traffic and any application traffic that uses TCP as the underlying transport protocol, such as HTTP, gRPC etc. This implies UDP and ICMP traffic which can be intercepted by iptables are not intercepted and redirected to the Pipy proxy sidecar.

Iptables chains and rules

FSM’s fsm-injector service programs the init container to set up a set of iptables chains and rules to perform traffic interception and redirection. The following section provides details on the responsibility of these chains and rules.

FSM leverages four chains to perform traffic interception and redirection:

1. PROXY_INBOUND: chain to intercept inbound traffic entering the pod
2. PROXY_IN_REDIRECT: chain to redirect intercepted inbound traffic to the sidecar proxy’s inbound listener
3. PROXY_OUTPUT: chain to intercept outbound traffic from applications within the pod
4. PROXY_REDIRECT: chain to redirect intercepted outbound traffic to the sidecar proxy’s outbound listener

Each of the chains above are programmed with rules to intercept and redirect application traffic via the Pipy proxy sidecar.

Outbound IP range exclusions

Outbound TCP based traffic from applications is by default intercepted using the iptables rules programmed by FSM, and redirected to the Pipy proxy sidecar. In some cases, it might be desirable to not subject certain IP ranges to be redirected and routed by the Pipy proxy sidecar based on service mesh policies. A common use case to exclude IP ranges is to not route non-application logic based traffic via the Pipy proxy, such as traffic destined to the Kubernetes API server, or traffic destined to a cloud provider’s instance metadata service. In such scenarios, excluding certain IP ranges from being subject to service mesh traffic routing policies becomes necessary.

Outbound IP ranges can be excluded at a global mesh scope or per pod scope.

1. Global outbound IP range exclusions

FSM provides the means to specify a global list of IP ranges to exclude from outbound traffic interception applicable to all pods in the mesh, as follows:

1. During FSM install using the --set option:

   # To exclude the IP ranges 1.1.1.1/32 and 2.2.2.2/24 from outbound interception
   fsm install --set=fsm.outboundIPRangeExclusionList="{1.1.1.1/32,2.2.2.2/24}"
   

2. By setting the outboundIPRangeExclusionList field in the fsm-mesh-config resource:

   ## Assumes FSM is installed in the fsm-system namespace
   kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"outboundIPRangeExclusionList":["1.1.1.1/32", "2.2.2.2/24"]}}}'  --type=merge
   

   When IP ranges are set for exclusion post-install, make sure to restart the pods in monitored namespaces for this change to take effect.

Globally excluded IP ranges are stored in the fsm-mesh-config MeshConfig custom resource and are read at the time of sidecar injection by fsm-injector. These dynamically configurable IP ranges are programmed by the init container along with the static rules used to intercept and redirect traffic via the Pipy proxy sidecar. Excluded IP ranges will not be intercepted for traffic redirection to the Pipy proxy sidecar. Refer to the outbound IP range exclusion demo to learn more.

2. Pod scoped outbound IP range exclusions

Outbound IP range exclusions can be configured at pod scope by annotating the pod to specify a comma separated list of IP CIDR ranges as flomesh.io/outbound-ip-range-exclusion-list=<comma separated list of IP CIDRs>.

# To exclude the IP ranges 10.244.0.0/16 and 10.96.0.0/16 from outbound interception on the pod
kubectl annotate pod <pod> flomesh.io/outbound-ip-range-exclusion-list="10.244.0.0/16,10.96.0.0/16"

When IP ranges are annotated post pod creation, make sure to restart the corresponding pods for this change to take effect.

Outbound IP range inclusions

Outbound TCP based traffic from applications is by default intercepted using the iptables rules programmed by FSM, and redirected to the Pipy proxy sidecar. In some cases, it might be desirable to only subject certain IP ranges to be redirected and routed by the Pipy proxy sidecar based on service mesh policies, and have remaining traffic not proxied to the sidecar. In such scenarios, inclusion IP ranges can be specified.

Outbound inclusion IP ranges can be specified at a global mesh scope or per pod scope.

1. Global outbound IP range inclusions

FSM provides the means to specify a global list of IP ranges to include for outbound traffic interception applicable to all pods in the mesh, as follows:

1. During FSM install using the --set option:

   # To include the IP ranges 1.1.1.1/32 and 2.2.2.2/24 for outbound interception
   fsm install --set=fsm.outboundIPRangeInclusionList="[1.1.1.1/32,2.2.2.2/24]"
   

2. By setting the outboundIPRangeInclusionList field in the fsm-mesh-config resource:

   ## Assumes FSM is installed in the fsm-system namespace
   kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"outboundIPRangeInclusionList":["1.1.1.1/32", "2.2.2.2/24"]}}}'  --type=merge
   

   When IP ranges are set for inclusion post-install, make sure to restart the pods in monitored namespaces for this change to take effect.

Globally included IP ranges are stored in the fsm-mesh-config MeshConfig custom resource and are read at the time of sidecar injection by fsm-injector. These dynamically configurable IP ranges are programmed by the init container along with the static rules used to intercept and redirect traffic via the Pipy proxy sidecar. IP addresses outside the specified inclusion IP ranges will not be intercepted for traffic redirection to the Pipy proxy sidecar.

2. Pod scoped outbound IP range inclusions

Outbound IP range inclusions can be configured at pod scope by annotating the pod to specify a comma separated list of IP CIDR ranges as flomesh.io/outbound-ip-range-inclusion-list=<comma separated list of IP CIDRs>.

# To include the IP ranges 10.244.0.0/16 and 10.96.0.0/16 for outbound interception on the pod
kubectl annotate pod <pod> flomesh.io/outbound-ip-range-inclusion-list="10.244.0.0/16,10.96.0.0/16"

When IP ranges are annotated post pod creation, make sure to restart the corresponding pods for this change to take effect.

Outbound port exclusions

Outbound TCP based traffic from applications is by default intercepted using the iptables rules programmed by FSM, and redirected to the Pipy proxy sidecar. In some cases, it might be desirable to not subject certain ports to be redirected and routed by the Pipy proxy sidecar based on service mesh policies. A common use case to exclude ports is to not route non-application logic based traffic via the Pipy proxy, such as control plane traffic. In such scenarios, excluding certain ports from being subject to service mesh traffic routing policies becomes necessary.

Outbound ports can be excluded at a global mesh scope or per pod scope.

1. Global outbound port exclusions

FSM provides the means to specify a global list of ports to exclude from outbound traffic interception applicable to all pods in the mesh, as follows:

1. During FSM install using the --set option:

   # To exclude the ports 6379 and 7070 from outbound sidecar interception
   fsm install --set=fsm.outboundPortExclusionList="{6379,7070}"
   

2. By setting the outboundPortExclusionList field in the fsm-mesh-config resource:

   ## Assumes FSM is installed in the fsm-system namespace
   kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"outboundPortExclusionList":[6379, 7070]}}}'  --type=merge
   

   When ports are set for exclusion post-install, make sure to restart the pods in monitored namespaces for this change to take effect.

Globally excluded ports are are stored in the fsm-mesh-config MeshConfig custom resource and are read at the time of sidecar injection by fsm-injector. These dynamically configurable ports are programmed by the init container along with the static rules used to intercept and redirect traffic via the Pipy proxy sidecar. Excluded ports will not be intercepted for traffic redirection to the Pipy proxy sidecar.

2. Pod scoped outbound port exclusions

Outbound port exclusions can be configured at pod scope by annotating the pod with a comma separated list of ports as flomesh.io/outbound-port-exclusion-list=<comma separated list of ports>:

# To exclude the ports 6379 and 7070 from outbound interception on the pod
kubectl annotate pod <pod> flomesh.io/outbound-port-exclusion-list=6379,7070

When ports are annotated post pod creation, make sure to restart the corresponding pods for this change to take effect.

Inbound port exclusions

Similar to outbound port exclusions described above, inbound traffic on pods can be excluded from being proxied to the sidecar based on the ports the traffic is directed to.

1. Global inbound port exclusions

FSM provides the means to specify a global list of ports to exclude from inbound traffic interception applicable to all pods in the mesh, as follows:

1. During FSM install using the --set option:

   # To exclude the ports 6379 and 7070 from inbound sidecar interception
   fsm install --set=fsm.inboundPortExclusionList="[6379,7070]"
   

2. By setting the inboundPortExclusionList field in the fsm-mesh-config resource:

   ## Assumes FSM is installed in the fsm-system namespace
   kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"inboundPortExclusionList":[6379, 7070]}}}'  --type=merge
   

   When ports are set for exclusion post-install, make sure to restart the pods in monitored namespaces for this change to take effect.

2. Pod scoped inbound port exclusions

Inbound port exclusions can be configured at pod scope by annotating the pod with a comma separated list of ports as flomesh.io/inbound-port-exclusion-list=<comma separated list of ports>:

# To exclude the ports 6379 and 7070 from inbound sidecar interception on the pod
kubectl annotate pod <pod> flomesh.io/inbound-port-exclusion-list=6379,7070

When ports are annotated post pod creation, make sure to restart the corresponding pods for this change to take effect.

3.2.2 - eBPF Redirection

FSM comes with eBPF functionality and provides users an options to use eBPF over default iptables.

The minimum kernel version is 5.4.

This guide shows how to start using this new functionality and enjoy the benefits eBPF. If you want to directly jump into quick start, refer to eBPF setup quickstart guide

For more details of comparison between iptables and eBPF, you can refer to Traffic Redirection.

Architecture

To provide eBPF features, Flomesh Service Mesh provides the fsm-cni CNI implementation and fsm-interceptor running on each node, where fsm-cni is compatible with mainstream CNI plugins.

When kubelet creates a pod on a node, it calls the CNI interface through the container runtime CRI to create the pod’s network namespace. After the pod’s network namespace is created, fsm-cni calls the interface of fsm-interceptor to load the BPF program and attach it to the hook point. In addition, fsm-interceptor also maintains pod information in eBPF Maps.

Implementation Principles

Next, we will introduce the implementation principles of the two features brought by the introduction of eBPF, but please note that many processing details will be ignored here.

Traffic interception

Outbound traffic

The figure below shows the interception of outbound traffic. Attach a BPF program to the socket operation connect, and in the program determine whether the current pod is managed by the service mesh, that is, whether it has a sidecar injected, and then modify the destination address to 127.0.0.1 and the destination port to the sidecar’s outbound port 15003. It is not enough to just modify it. The original destination address and port should also be saved in a map, using the socket’s cookie as the key.

After the connection with the sidecar is established, the original destination is saved in another map through a program attached to the mount point sock_ops, using local address + port and remote address + port as the key. When the sidecar accesses the target application later, it obtains the original destination through the getsockopt operation on the socket. Yes, a eBPF program is also attached to getsockopt, which retrieves the original destination address from the map and returns it.

Inbound traffic

For the interception of inbound traffic, the traffic originally intended for the application port is forwarded to the sidecar’s inbound port 15003. There are two cases:

• In the first case, the requester and the service are located on the same node. After the requester’s sidecar connect operation is intercepted, the destination port is changed to 15003.
• In the second case, the requester and the service are located on different nodes. When the handshake packet reaches the service’s network namespace, it is intercepted by the BPF program attached to the tc (traffic control) ingress, and the port is modified to 15003, achieving a functionality similar to DNAT.

Network communication acceleration

In Kubernetes networks, network packets unavoidably undergo multiple kernel network protocol stack processing. eBPF accelerates network communication by bypassing unnecessary kernel network protocol stack processing and directly exchanging data between two sockets that are peers.

The figure in the traffic interception section shows the sending and receiving trajectories of messages. When the program attached to sock_ops discovers that the connection is successfully established, it saves the socket in a map, using local address + port and remote address + port as the key. As the two sockets are peers, their local and remote information is opposite, so when a socket sends a message, it can directly address the peer socket from the map.

This solution also applies to communication between two pods on the same node.

Prerequisites

• Ubuntu 20.04
• Kernel 5.15.0-1034
• 2c4g VM * 3：master、node1、node2

Install CNI Plugin

Execute the following command on all nodes to download the CNI plugin.

sudo mkdir -p /opt/cni/bin
curl -sSL https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz | sudo tar -zxf - -C /opt/cni/bin

Master Node

Get the IP address of the master node. (Your machine IP might be different)

export MASTER_IP=10.0.2.6

Kubernetes cluster uses the k3s distribution, but when installing the cluster, you need to disable the flannel integrated by k3s and use independently installed flannel for validation. This is because k3s’s doesn’t follow Flannel directory structure /opt/cni/bin and store its CNI bin directory at /var/lib/rancher/k3s/data/xxx/bin where xxx is some randomly generated text.

curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable servicelb --flannel-backend=none --advertise-address $MASTER_IP --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config

Install Flannel. Note that the default Pod CIDR of Flannel is 10.244.0.0/16, and we will modify it to k3s’s default 10.42.0.0/16.

curl -s https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml | sed 's|10.244.0.0/16|10.42.0.0/16|g' | kubectl apply -f -

Get the access token of the API server for initializing worker nodes.

sudo cat /var/lib/rancher/k3s/server/node-token

Worker Node

Use the IP address of the master node and the token obtained earlier to initialize the node.

export INSTALL_K3S_VERSION=v1.23.8+k3s2
export NODE_TOKEN=K107c1890ae060d191d347504740566f9c506b95ea908ba4795a7a82ea2c816e5dc::server:2757787ec4f9975ab46b5beadda446b7
curl -sfL https://get.k3s.io | K3S_URL=https://${MASTER_IP}:6443 K3S_TOKEN=${NODE_TOKEN} sh -

Download FSM CLI

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.2.3
curl -L https://github.com/flomesh-io/fsm/releases/download/${release}/fsm-${release}-${system}-${arch}.tar.gz | tar -vxzf -
./${system}-${arch}/fsm version
sudo cp ./${system}-${arch}/fsm /usr/local/bin/

Install FSM

export fsm_namespace=fsm-system 
export fsm_mesh_name=fsm 

fsm install \
    --mesh-name "$fsm_mesh_name" \
    --fsm-namespace "$fsm_namespace" \
    --set=fsm.trafficInterceptionMode=ebpf \
    --set=fsm.fsmInterceptor.debug=true \
    --timeout=900s

Deploy Sample Application

#Sample services
kubectl create namespace ebpf
fsm namespace add ebpf

kubectl apply -n ebpf -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/interceptor/curl.yaml
kubectl apply -n ebpf -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/interceptor/pipy-ok.yaml

#Schedule Pods to Different Nodes
kubectl patch deployments curl -n ebpf -p '{"spec":{"template":{"spec":{"nodeName":"node1"}}}}'
kubectl patch deployments pipy-ok-v1 -n ebpf -p '{"spec":{"template":{"spec":{"nodeName":"node1"}}}}'
kubectl patch deployments pipy-ok-v2 -n ebpf -p '{"spec":{"template":{"spec":{"nodeName":"node2"}}}}'

sleep 5

#Wait for dependent Pods to start successfully
kubectl wait --for=condition=ready pod -n ebpf -l app=curl --timeout=180s
kubectl wait --for=condition=ready pod -n ebpf -l app=pipy-ok -l version=v1 --timeout=180s
kubectl wait --for=condition=ready pod -n ebpf -l app=pipy-ok -l version=v2 --timeout=180s

Testing

During testing, you can view the debug logs of BPF program execution by viewing the kernel tracing logs on the worker node using the following command. To avoid interference caused by sidecar communication with the control plane, first obtain the IP address of the control plane.

kubectl get svc -n fsm-system fsm-controller -o jsonpath='{.spec.clusterIP}'
10.43.241.189

Execute the following command on both worker nodes.

sudo cat /sys/kernel/debug/tracing/trace_pipe | grep bpf_trace_printk | grep -v '10.43.241.189'

Execute the following command on both worker nodes.

curl_client="$(kubectl get pod -n ebpf -l app=curl -o jsonpath='{.items[0].metadata.name}')"
kubectl exec ${curl_client} -n ebpf -c curl -- curl -s pipy-ok:8080

You should receive results similar to the following, and the kernel tracing logs should also output the debug logs of the BPF program accordingly (the content is quite long, so it will not be shown here).

Hi, I am pipy ok v1 !
Hi, I am pipy ok v2 !

3.3 - Traffic Splitting

The SMI Traffic Split API can be used to split outgoing traffic to multiple service backends. This can be used to orchestrate canary releases for multiple versions of the software.

What is supported

FSM implements the SMI traffic split v1alpha4 version.

It supports the following:

• Traffic splitting in both SMI and Permissive traffic policy modes
• HTTP and TCP traffic splitting
• Traffic splitting for canary or blue-green deployments

How it works

Outbound traffic destined to a Kubernetes service can be split to multiple service backends using the SMI Traffic Split API. Consider the following example where traffic to the bookstore.default.svc.cluster.local FQDN corresponding to the default/bookstore service is split to services default/bookstore-v1 and default/bookstore-v2, with a weight of 90 and 10 respectively.

apiVersion: split.smi-spec.io/v1alpha4
kind: TrafficSplit
metadata:
  name: bookstore-split
  namespace: default
spec:
  service: bookstore.default.svc.cluster.local
  backends:
  - service: bookstore-v1
    weight: 90
  - service: bookstore-v2
    weight: 10

For a TrafficSplit resource to be correctly configured, it is important to ensure the following conditions are met:

• metadata.namespace is a namespace added to the mesh
• metadata.namespace, spec.service, and spec.backends all belong to the same namespace
• spec.service specifies an FQDN of a Kubernetes service
• spec.service and spec.backends correspond to Kubernetes service objects
• The total weight of all backends must be greater than zero, and each backend must have a positive weight

When a TrafficSplit resource is created, FSM applies the configuration on client sidecars to split traffic directed to the root service (spec.service) to the backends (spec.backends) based the specified weights. For HTTP traffic, the Host/Authority header in the request must match the FQDNs of the root service specified in the TrafficSplit resource. In the above example, it implies that the Host/Authority header in the HTTP request originated by the client must match the Kubernetes service FQDNs of the default/bookstore service for traffic split to work.

Note: FSM does not configure Host/Authority header rewrites for the original HTTP requests, so it is necessary that the backend services referenced in a TrafficSplit resource accept requests with the original HTTP Host/Authority header.

It is important to note that a TrafficSplit resource only configures traffic splitting to a service, and does not give applications permission to communicate with each other. Thus, a valid TrafficTarget resource must be configured in conjunction with a TrafficSplit configuration to achieve traffic flow between applications as desired.

Refer to a demo on Canary rollouts using SMI Traffic Split to learn more.

3.4 - Circuit Breaking

Circuit breaking is a critical component of distributed systems and an important resiliency pattern. Circuit breaking allows applications to fail quickly and apply back pressure downstream as soon as possible, thereby providing the means to limit the impact of failures across the system. This guide describes how circuit breaking can be configured in FSM.

Configuring circuit breaking

FSM leverages its UpstreamTrafficSetting API to configure circuit breaking attributes for traffic directed to an upstream service. We use the term upstream service to refer to a service that receives connections and requests from clients and return responses. The specification enables configuring circuit breaking attributes for an upstream service at the connection and request level.

Each UpstreamTrafficSetting configuration targets an upstream host defined by the spec.host field. For a Kubernetes service my-svc in the namespace my-namespace, the UpstreamTrafficSetting resource must be created in the namespace my-namespace, and spec.host must be an FQDN of the form my-svc.my-namespace.svc.cluster.local. When specified as a match in an Egress policy, spec.host must correspond to the host specified in the Egress policy and the UpstreamTrafficSetting configuration must belong to the same namespace as the Egress resource.

Circuit breaking is applicable at both the TCP and HTTP level, and can be configured using the connectionSettings attribute in the UpstreamTrafficSetting resource. TCP traffic settings apply to both TCP and HTTP traffic, while HTTP settings only apply to HTTP traffic.

The following circuit breaking configurations are supported:

• Maximum connections: The maximum number of connections that a client is allowed to establish to all backends belonging to the upstream host specified via the spec.host field in the UpstreamTrafficSetting configuration. This setting can be configured using the tcp.maxConnections field and is applicable to both TCP and HTTP traffic. If not specified, the default is 4294967295 (2^32 - 1).

• Maximum pending requests: The maximum number of pending HTTP requests to the upstream host that are allowed to be queued. Requests are added to the list of pending requests whenever there aren’t enough upstream connections available to immediately dispatch the request. For HTTP/2 connections, if http.maxRequestsPerConnection is not configured, all requests will be multiplexed over the same connection so this circuit breaker will only be hit when no connection is already established. This setting can be configured using the http.maxPendingRequests field and is only applicable to HTTP traffic. If not specified, the default is 4294967295 (2^32 - 1).

• Maximum requests: The maximum number of parallel request that a client is allowed to make to the upstream host. This setting can be configured using the http.maxRequests field and is only applicable to HTTP traffic. If not specified, the default is 4294967295 (2^32 - 1).

• Maximum requests per connection: The maximum number of requests allowed per connection. This setting can be configured using the http.maxRequestsPerConnection field and is only applicable to HTTP traffic. If not specified, there is no limit.

• Maximum active retries: The maximum number of active retries that a client is allowed to make to the upstream host. This setting can be configured using the http.maxRetries field and is only applicable to HTTP traffic. If not specified, the default is 4294967295 (2^32 - 1).

To learn more about configuring circuit breaking, refer to the following demo guides:

• Circuit breaking for destinations within the mesh

3.5 - Retry

Retry is a resiliency pattern that enables an application to shield transient issues from customers. This is done by retrying requests that are failing from temporary faults such as a pod is starting up. This guide describes how to implement retry policy in FSM.

Configuring Retry

FSM uses its Retry policy API to allow retries on traffic from a specified source (ServiceAccount) to one or more destinations (Service). Retry is only applicable to HTTP traffic. FSM can implement retry for applications participating in the mesh.

The following retry configurations are supported:

• Per Try Timeout: The time allowed for a retry to take before it is considered a failed attempt. The default uses the global route timeout.

• Retry Backoff Base Interval: The base interval for exponential retry back-off. The backoff is randomly chosen from the range [0,(2**N-1)B], where N is the retry number and B is the base interval. The default is 25ms and the maximum interval is 10 times the base interval.

• Number of Retries: The maximum number of retries to attempt. The default is 1.

• Retry On: Specifies the policy for when a failed request will be retried. Multiple policies can be specified by using a , delimited list.

To learn more about configuring retry, refer to the Retry policy demo and [API documentation][1].

Examples

If requests from the bookbuyer service to bookstore-v1 service or bookstore-v2 service receive responses with a status code 5xx, then bookbuyer will retry the request 3 times. If an attempted retry takes longer than 3s it’s considered a failed attempt. Each retry has a delay period (backoff) before it is attempted calculated above. The backoff for all retries is capped at 10s.

kind: Retry
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: retry
spec:
  source:
    kind: ServiceAccount
    name: bookbuyer
    namespace: bookbuyer
  destinations:
  - kind: Service
    name: bookstore
    namespace: bookstore-v1
  - kind: Service
    name: bookstore
    namespace: bookstore-v2
  retryPolicy:
    retryOn: "5xx"
    perTryTimeout: 3s
    numRetries: 3
    retryBackoffBaseInterval: 1s

If requests from the bookbuyer service to bookstore-v2 service receive responses with a status code 5xx or retriable-4xx (409), then bookbuyer will retry the request 5 times. If an attempted retry takes longer than 4s it’s considered a failed attempt. Each retry has a delay period (backoff) before it is attempted calculated above. The backoff for all retries is capped at 20ms.

kind: Retry
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: retry
spec:
  source:
    kind: ServiceAccount
    name: bookbuyer
    namespace: bookbuyer
  destinations:
  - kind: Service
    name: bookstore
    namespace: bookstore-v2
  retryPolicy:
    retryOn: "5xx,retriable-4xx"
    perTryTimeout: 4s
    numRetries: 5
    retryBackoffBaseInterval: 2ms

3.6 - Rate Limiting

Rate limiting is an effective mechanism to control the throughput of traffic destined to a target host. It puts a cap on how often downstream clients can send network traffic within a certain timeframe.

Most commonly, when a large number of clients are sending traffic to a target host, if the target host becomes backed up, the downstream clients will overwhelm the upstream target host. In this scenario it is extremely difficult to configure a tight enough circuit breaking limit on each downstream host such that the system will operate normally during typical request patterns but still prevent cascading failure when the system starts to fail. In such scenarios, rate limiting traffic to the target host is effective.

FSM supports server-side rate limiting per target host, also referred to as local per-instance rate limiting.

Configuring local per-instance rate limiting

FSM leverages its UpstreamTrafficSetting API to configure rate limiting attributes for traffic directed to an upstream service. We use the term upstream service to refer to a service that receives connections and requests from clients and return responses. The specification enables configuring local rate limiting attributes for an upstream service at the connection and request level.

Each UpstreamTrafficSetting configuration targets an upstream host defined by the spec.host field. For a Kubernetes service my-svc in the namespace my-namespace, the UpstreamTrafficSetting resource must be created in the namespace my-namespace, and spec.host must be an FQDN of the form my-svc.my-namespace.svc.cluster.local.

Local rate limiting is applicable at both the TCP (L4) connection and HTTP request level, and can be configured using the rateLimit.local attribute in the UpstreamTrafficSetting resource. TCP settings apply to both TCP and HTTP traffic, while HTTP settings only apply to HTTP traffic. Both TCP and HTTP level rate limiting is enforced using a token bucket rate limiter.

Rate limiting TCP connections

TCP connections can be rate limited per unit of time. An optional burst limit can be specified to allow a burst of connections above the baseline rate to accommodate for connection bursts in a short interval of time. TCP rate limiting is applied as a token bucket rate limiter at the network filter chain of the upstream service’s inbound listener. Each incoming connection processed by the filter consumes a single token. If the token is available, the connection will be allowed. If no tokens are available, the connection will be immediately closed.

The following attributes nested under spec.rateLimit.local.tcp define the rate limiting attributes for TCP connections:

• connections: The number of connections allowed per unit of time before rate limiting occurs on all backends belonging to the upstream host specified via the spec.host field in the UpstreamTrafficSetting configuration. This setting is applicable to both TCP and HTTP traffic.

• unit: The period of time within which connections over the limit will be rate limited. Valid values are second, minute and hour.

• burst: The number of connections above the baseline rate that are allowed in a short period of time.

Refer to the TCP local rate limiting API for additional information regarding API usage.

Rate limiting HTTP requests

HTTP requests can be rate limited per unit of time. An optional burst limit can be specified to allow a burst of requests above the baseline rate to accommodate for request bursts in a short interval of time. HTTP rate limiting is applied as a token bucket rate limiter at the virtual host and/or HTTP route level at the upstream backend, depending on the rate limiting configuration. Each incoming request processed by the filter consumes a single token. If the token is available, the request will be allowed. If no tokens are available, the request will receive the configured rate limit status.

HTTP request rate limiting can be configured at the virtual host level by specifying the rate limiting attributes nested under the spec.rateLimit.local.http field. Alternatively, rate limiting can be configured per HTTP route allowed on the upstream backend by specifying the rate limiting attributes as a part of the spec.httpRoutes field. It is important to note that when configuring rate limiting per HTTP route, the route matches an HTTP path that has already been permitted by a service mesh policy, otherwise the rate limiting policy will be ignored.

The following rate limiting attributes can be configured for HTTP traffic:

• requests: The number of requests allowed per unit of time before rate limiting occurs on all backends belonging to the upstream host specified via the spec.host field in the UpstreamTrafficSetting configuration.

• unit: The period of time within which requests over the limit will be rate limited. Valid values are second, minute and hour.

• burst: The number of requests above the baseline rate that are allowed in a short period of time.

• responseStatusCode: The HTTP status code to use for responses to rate limited requests. Code must be in the 400-599 (inclusive) error range. If not specified, a default of 429 (Too Many Requests) is used.

• responseHeadersToAdd: The list of HTTP headers as key-value pairs that should be added to each response for requests that have been rate limited.

Demos

To learn more about configuring rate limting, refer to the following demo guides:

• Local rate limiting of TCP connections
• Local rate limiting of HTTP requests

3.7 - Ingress

3.7.1 - Ingress to Mesh

Using Ingress to manage external access to services within the cluster

Ingress refers to managing external access to services within the cluster, typically HTTP/HTTPS services. FSM’s ingress capability allows cluster administrators and application owners to route traffic from clients external to the service mesh to service mesh backends using a set of rules depending on the mechanism used to perform ingress.

IngressBackend API

FSM leverages its IngressBackend API to configure a backend service to accept ingress traffic from trusted sources. The specification enables configuring how specific backends must authorize ingress traffic depending on the protocol used, HTTP or HTTPS. When the backend protocol is http, the specified source kind must either be: 1. Service kind whose endpoints will be authorized to connect to the backend, or 2. IPRange kind that specifies the source IP CIDR range authorized to connect to the backend. When the backend protocol is https, the source specified must be an AuthenticatedPrincipal kind which defines the Subject Alternative Name (SAN) encoded in the client’s certificate that the backend will authenticate. A source with the kind Service or IPRange is optional for https backends, and if specified implies that the client must match the source in addition to its AuthenticatedPrincipal value. For https backends, client certificate validation is performed by default and can be disabled by setting skipClientCertValidation: true in the tls field for the backend. The port.number field for a backend service in the IngressBackend configuration must correspond to the targetPort of a Kubernetes service.

Note that when the Kind for a source in an IngressBackend configuration is set to Service, FSM controller will attempt to discover the endpoints of that service. For FSM to be able to discover the endpoints of a service, the namespace in which the service resides needs to be a monitored namespace. Enable the namespace to be monitored using:

kubectl label ns <namespace> flomesh.io/monitored-by=<mesh name>

Examples

The following IngressBackend configuration will allow access to the foo service on port 80 in the test namespace only if the source originating the traffic is an endpoint of the myapp service in the default namespace:

kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: basic
  namespace: test
spec:
  backends:
    - name: foo
      port:
        number: 80 # targetPort of the service
        protocol: http
  sources:
    - kind: Service
      namespace: default
      name: myapp

The following IngressBackend configuration will allow access to the foo service on port 80 in the test namespace only if the source originating the traffic has an IP address that belongs to the CIDR range 10.0.0.0/8:

kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: basic
  namespace: test
spec:
  backends:
    - name: foo
      port:
        number: 80 # targetPort of the service
        protocol: http
  sources:
    - kind: IPRange
      name: 10.0.0.0/8

The following IngressBackend configuration will allow access to the foo service on port 80 in the test namespace only if the source originating the traffic encrypts the traffic with TLS and has the Subject Alternative Name (SAN) client.default.svc.cluster.local encoded in its client certificate:

kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: basic
  namespace: test
spec:
  backends:
    - name: foo
      port:
        number: 80
        protocol: https # https implies TLS
      tls:
        skipClientCertValidation: false # mTLS (optional, default: false)
  sources:
    - kind: AuthenticatedPrincipal
      name: client.default.svc.cluster.local

Refer to the following sections to understand how the IngressBackend configuration looks like for http and https backends.

Choices to perform Ingress

FSM supports multiple options to expose mesh services externally using ingress which are described in the following sections. FSM has been tested with Contour and OSS Nginx, which work with the ingress controller installed outside the mesh and provisioned with a certificate to participate in the mesh.

Note: FSM integration with Nginx Plus has not been fully tested for picking up a self-signed mTLS certificate from a Kubernetes secret. However, an alternative way to incorporate Nginx Plus or any ingress is to install it in the mesh so that it is injected with an Pipy sidecar, which will allow it to participate in the mesh. Additional inbound ports such as 80 and 443 may need to be allowed to bypass the Pipy sidecar.

1. Using FSM ingress controllers and gateways

Using FSM ingress controllers and edge proxy is the preferred method for executing Ingress in an FSM managed services mesh. Using FSM, users get a high-performance ingress controller with rich policy specifications for a variety of scenarios, while maintaining lightweight profiles.

To use FSM as an ingress, enable it during mesh installation by passing option --set=fsm.fsmIngress.enabled=true:

fsm install \
    --set=fsm.fsmIngress.enabled=true

Or enable ingress feature after mesh installed:

fsm ingress enable --fsm-namespace <FSM NAMESPACE>

In addition to configuring the edge proxy for FSM using the appropriate API, the service mesh backend in FSM will only accept traffic from authorized edge proxy or gateways. FSM’s IngressBackend specification allows cluster administrators and application owners to explicitly specify how the service mesh backend should authorize ingress traffic. The following sections describe how to use the IngressBackend and HTTPProxy APIs in combination to allow HTTP and HTTPS ingress traffic to be routed to the mesh backend.

It is recommended that ingress traffic always be restricted to authorized clients. To do this, enable FSM to monitor the endpoints of the edge proxy located in the namespace where the ingress installation is located:

kubectl label ns <fsm namespace> flomesh.io/monitored-by=<mesh name>

If using FSM Ingress as Ingress controller, there is no need to execute command above.

HTTP Ingress using FSM

A minimal [HTTPProxy][2] configuration and FSM’s IngressBackend1 specification to route ingress traffic to the mesh service foo in the namespace test might look like the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fsm-ingress
  namespace: test
spec:
  ingressClassName: pipy
  rules:
  - host: foo-basic.bar.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: foo
            port:
              number: 80
---
kind: IngressBackend
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: basic
  namespace: test
spec:
  backends:
    - name: foo
      port:
        number: 80 # targetPort of the service
        protocol: http # http implies no TLS
  sources:
    - kind: Service
      namespace: fsm-system
      name: fsm-ingress

The above configuration allows external clients to access the foo service under the test namespace.

1. The Ingress configuration will route incoming HTTP traffic from external sources with the Host: header of foo-basic.bar.com to the service named foo on port 80 in the test namespace.
2. IngressBackend is configured to allow only endpoints named fsm-ingress service from the same namespace where FSM is installed (default is fsm-system) to access port 80 of the foo serivce under the test namespace.

Examples

Refer to the Ingress with FSM demo for examples on how to expose mesh services externally using FSM in FSM.

2. Bring your own Ingress Controller and Gateway

If using FSM with FSM for ingress is not feasible for your use case, FSM provides the facility to use your own ingress controller and edge gateway for routing external traffic to service mesh backends. Much like how ingress is configured above, in addition to configuring the ingress controller to route traffic to service mesh backends, an IngressBackend configuration is required to authorize clients responsible for proxying traffic originating externally.

3.7.2 - Service Loadbalancer

3.7.3 - FSM Ingress Controller

The Kubernetes Ingress API is designed with a separation of concerns, where the Ingress implementation provides an entry feature infrastructure managed by operations staff; it also allows application owners to control the routing of requests to the backend through rules.

Ingress is an API object for managing external access to services in a cluster, with typical access through HTTP. It provides load balancing, SSL termination, and name-based virtual hosting. For the Ingress resource to work, the cluster must have a running Ingress controller.

Ingress controller configures the HTTP load balancer by monitoring Ingress resources in the cluster.

3.7.3.1 - Installation

Installation

Prerequisites

• Kubernetes cluster version v1.19.0 or higher.
• FSM version >= v1.1.0.
• FSM CLI to install FSM and enable FSM Ingress.

There are two options to install FSM Ingress Controller. One is installing it along with FSM during FSM installation. It won’t be enabled by default so we need to enable it explicitly:

fsm install \
    --set=fsm.fsmIngress.enabled=true

Another is installing it separately if you already have FSM mesh installed.

Using the fsm command line tool to enable FSM Ingress Controller.

fsm ingress enable

Check the resource.

kubectl get pod,svc -n fsm-system -l app=fsm-ingress                                                                            
NAME                               READY   STATUS    RESTARTS   AGE
pod/fsm-ingress-574465b678-xj8l6   1/1     Running   0          14h

NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/fsm-ingress   LoadBalancer   10.43.243.124   10.0.2.4      80:30508/TCP   14h

Once all done, we can start to play with FSM Ingress Controller.

3.7.3.2 - Basics

Demo

• HTTP and HTTPS with FSM Ingress Controller

3.7.3.3 - Advanced TLS

FSM Ingress Controller - Advanced TLS

In the document of FSM Ingress Controller, we introduced FSM Ingress and some of its basic functinoality. In this part of series, we will continue on where we left and look into advanced TLS features and we can configure FSM Ingress to use them.

Normally, we see below four combinations of communication with upstream services

• Client -> HTTP Ingress -> HTTP Upstream
• Client -> HTTPS Ingress -> HTTP Upstream
• Client -> HTTP Ingress -> HTTPS Upstream
• Client -> HTTPS Ingress -> HTTPS Upstream

Two of the above combinations has been covered in basics introduction blog post and in this article we will introduce the remaining two combinations i.e. communicating with an upstream HTTPS service.

• HTTPS Upstream: The certificate of the backend service, the upstream, must be checked.
• Client Verification: Mainly when using HTTPS entrance, the certificate used by the client is checked.

Demo

• FSM Ingress Controller Advanced TLS features

3.7.3.4 - TLS Passthrough

FSM Ingress Controller - TLS Passthrough

This guide will demonstrate TLS passthrough feature of FSM Ingress.

What is TLS passthrough

TLS (Secure Socket Layer), also known as TLS (Transport Layer Security), protects the security communication between the client and the server through encryption.

TLS Passthrough is one of the two ways that a proxy server handles TLS requests (the other is TLS offload). In TLS passthrough mode, the proxy does not decrypt the TLS request from the client but instead forwards it to the upstream server for decryption, meaning the data remains encrypted while passing through the proxy, thus ensuring the security of important and sensitive data.

Advantages of TLS passthrough

• Since the data is not decrypted on the proxy but is forwarded to the upstream server in an encrypted manner, the data is protected from network attacks.
• Encrypted data arrives at the upstream server without decryption, ensuring the confidentiality of the data.
• This is also the simplest method of configuring TLS for the proxy.

Disadvantages of TLS passthrough

• Malicious code may be present in the traffic, which will directly reach the backend server.
• In the TLS passthrough process, switching servers is not possible.
• Layer-7 traffic processing cannot be performed.

Installation

The TLS passthrough feature can be enabled during installation of FSM.

fsm install --set=fsm.image.registry=addozhang --set=fsm.image.tag=latest-main --set=fsm.fsmIngress.enabled=true --set=fsm.fsmIngress.tls.enabled=true --set=fsm.fsmIngress.tls.sslPassthrough.enabled=true

Or you can enable it during FSM Ingress enabling when already have FSM installed.

fsm ingress enable --tls-enable --passthrough-enable

Demo

• How to use TLS passthrough feature of FSM Ingress

3.7.4 - FSM Gateway

The FSM Gateway serves as an implementation of the Kubernetes Gateway API, representing one of the various components within the FSM world.

Upon activation of the FSM Gateway, the FSM controller, assuming the position of gateway overseer, diligently monitors both Kubernetes native resources and Gateway API assets. Subsequently, it dynamically furnishes the pertinent configurations to Pipy, functioning as a proxy.

Should you have an interest in the FSM Gateway, the ensuing documentation might prove beneficial.

3.7.4.1 - Installation

To utilize the FSM Gateway, initial activation within the FSM is requisite. Analogous to the FSM Ingress, two distinct methodologies exist for its enablement.

Note: It is imperative to acknowledge that the minimum required version of Kubernetes to facilitate the FSM Gateway activation is v1.21.0.

Let’s start.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• FSM version >= v1.1.0.
• FSM CLI to install FSM and enable FSM Gateway.

Installation

One methodology for enabling FSM Gateway is enable it during FSM installation. Remember that it’s diabled by defaulty.

fsm install \
    --set=fsm.fsmGateway.enabled=true

Another approach is installing it individually if you already have FSM mesh installed.

fsm gateway enable

Once done, we can check the GatewayClass resource in cluster.

kubectl get GatewayClass
NAME              CONTROLLER                      ACCEPTED   AGE
fsm-gateway-cls   flomesh.io/gateway-controller   True       113s

Yes, the fsm-gateway-cls is just the one we are expecting. We can also get the controller name above.

Different from Ingress controller, there is no explicit Deployment or Pod unless create a Gateway manually.

Let’s try with below to create a simple FSM gateway.

Quickstart

To create a FSM gateway, we need to create Gateway resource. This manifest will setup a gateway which will listen on port 8000 and accept the xRoute resources from same namespace.

xRoute stands for HTTPRoute, HTTPRoute, TLSRoute, TCPRoute, UDPRoute and GRPCRoute.

kubectl apply -n fsm-system -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: simple-fsm-gateway
spec:
  gatewayClassName: fsm-gateway-cls
  listeners:
    - protocol: HTTP
      port: 8000
      name: http
      allowedRoutes:
        namespaces:
          from: Same
EOF

Then we can check the resoureces:

kubectl get po,svc -n fsm-system -l app=fsm-gateway
NAME                                          READY   STATUS    RESTARTS   AGE
pod/fsm-gateway-fsm-system-745ddc856b-v64ql   1/1     Running   0          12m

NAME                             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/fsm-gateway-fsm-system   LoadBalancer   10.43.20.139   10.0.2.4      8000:32328/TCP   12m

At this time, you will get result below if trying to access the gateway port:

curl -i 10.0.2.4:8000/
HTTP/1.1 404 Not Found
content-length: 13
connection: keep-alive

Not found

That’s why we have not configure any route. Let’s create a HTTRoute for the Service fsm-controller(The FSM controller has a Pipy repo running).

kubectl apply -n fsm-system -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: repo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  rules:
  - backendRefs:
    - name: fsm-controller
      port: 6060
EOF

Trigger the request again, it responds 200 this time.

curl -i 10.0.2.4:8000/
HTTP/1.1 200 OK
content-type: text/html
content-length: 0
connection: keep-alive

3.7.4.2 - HTTP Routing

In FSM Gateway, the HTTPRoute resource is used to configure route rules which will match request to backend servers. Currently, the Kubernetes Service is the only one accepted as backend resource.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

Deploy sample

First, let’s install the example in namespace httpbin with commands below.

kubectl create namespace httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/gateway/http-routing.yaml

Verification

Once done, we can get the gateway installed.

kubectl get pod,svc -n httpbin -l app=fsm-gateway                                                                                           default ⎈
NAME                                       READY   STATUS    RESTARTS   AGE
pod/fsm-gateway-httpbin-867768f76c-69s6x   1/1     Running   0          16m

NAME                          TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/fsm-gateway-httpbin   LoadBalancer   10.43.41.36   10.0.2.4      8000:31878/TCP   16m

Beyond the gateway resources, we also create the HTTPRoute resources.

kubectl get httproute -n httpbin
NAME             HOSTNAMES             AGE
http-route-foo   ["foo.example.com"]   18m
http-route-bar   ["bar.example.com"]   18m

Testing

To test the rules, we should get the address of gateway first.

export GATEWAY_IP=$(kubectl get svc -n httpbin -l app=fsm-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

We can trigger a request to gateway without hostname.

curl -i http://$GATEWAY_IP:8000/headers
HTTP/1.1 404 Not Found
server: pipy-repo
content-length: 0
connection: keep-alive

It responds with 404. Next, we can try with the hostnames configured in HTTPRoute resources.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "foo.example.com",
    "User-Agent": "curl/7.68.0"
  }
}

curl -H 'host:bar.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "bar.example.com",
    "User-Agent": "curl/7.68.0"
  }
}

This time, the server responds success message. There is hostname we are requesting in each response.

3.7.4.3 - HTTP URL Rewrite

The URL rewriting feature provides FSM Gateway users with a way to modify the request URL before the traffic enters the target service. This not only provides greater flexibility to adapt to changes in backend services, but also ensures smooth migration of applications and normalization of URLs.

The HTTPRoute resource utilizes HTTPURLRewriteFilter to rewrite the path in request to another one before it gets forwarded to upstream.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

We will follow the sample in HTTP Routing.

In backend server, there is a path /get which will responds more information than path /headers.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "foo.example.com",
    "User-Agent": "curl/7.68.0"
  },
  "origin": "10.42.0.87",
  "url": "http://foo.example.com/get"
}

Replace URL Full Path

Example bellow will replace the /get path to /headers path.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    filters:
    - type: URLRewrite
      urlRewrite: 
        path: 
          type: ReplaceFullPath
          replaceFullPath: /headers
    backendRefs:
    - name: httpbin
      port: 8080          
  - matches:
    - path:
        type: PathPrefix
        value: /        
    backendRefs:
    - name: httpbin
      port: 8080
EOF

After updated the HTTP rule, we will get the same response as /headers when requesting /get.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/get
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "foo.example.com",
    "User-Agent": "curl/7.68.0"
  }
}

Replace URL Prefix Path

In backend server, there is another two paths:

• /status/{statusCode} will respond with specified status code.
• /stream/{n} will respond the response of /get n times in stream.

curl -s -w "%{response_code}\n" -H 'host:foo.example.com' http://$GATEWAY_IP:8000/status/204
204
curl -s -H 'host:foo.example.com' http://$GATEWAY_IP:8000/stream/1
{"url": "http://foo.example.com/stream/1", "args": {}, "headers": {"Host": "foo.example.com", "User-Agent": "curl/7.68.0", "Accept": "*/*", "Connection": "keep-alive"}, "origin": "10.42.0.161", "id": 0}

If we hope to change the behavior of /status to /stream, the rule is required to update again.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /status    
    filters:
    - type: URLRewrite
      urlRewrite: 
        path: 
          type: ReplacePrefixMatch
          replacePrefixMatch: /stream
    backendRefs:
    - name: httpbin
      port: 8080          
  - matches:
    - path:
        type: PathPrefix
        value: /        
    backendRefs:
    - name: httpbin
      port: 8080
EOF

If we trigger the request to /status/204 path again, we will stream the request data 204 times.

curl -s -H 'host:foo.example.com' http://$GATEWAY_IP:8000/status/204
{"url": "http://foo.example.com/stream/204", "args": {}, "headers": {"Host": "foo.example.com", "User-Agent": "curl/7.68.0", "Accept": "*/*", "Connection": "keep-alive"}, "origin": "10.42.0.161", "id": 99}
...

Replace Host Name

Let’s follow the example rule below. It will replace host name from foo.example.com to baz.example.com for all traffic requesting /get.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    filters:
    - type: URLRewrite
      urlRewrite: 
        hostname: baz.example.com
    backendRefs:
    - name: httpbin
      port: 8080          
  - matches:
    - path:
        type: PathPrefix
        value: /        
    backendRefs:
    - name: httpbin
      port: 8080
EOF

Update rule and trigger request. We can see the client is requesting url http://foo.example.com/get, but the Host is replaced.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "baz.example.com",
    "User-Agent": "curl/7.68.0"
  },
  "origin": "10.42.0.87",
  "url": "http://baz.example.com/get"

3.7.4.4 - HTTP Redirect

Request redirection is a common network application function that allows the server to tell the client: “The resource you requested has been moved to another location, please go to the new location to obtain it.”

The HTTPRoute resource utilizes HTTPRequestRedirectFilter to redirect client to the specified new location.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

We will follow the sample in HTTP Routing.

In our backend server, there are two paths /headers and /get. The previous one responds all request headers as body, and the latter one responds more information of client than /headers.

To facilitate testing, it’s better to add records to local hosts.

echo $GATEWAY_IP foo.example.com bar.example.com >> /etc/hosts
-bash: /etc/hosts: Permission denied

curl foo.example.com/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "foo.example.com",
    "User-Agent": "curl/7.68.0"
  }
}
curl bar.example.com/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "bar.example.com",
    "User-Agent": "curl/7.68.0"
  },
  "origin": "10.42.0.87",
  "url": "http://bar.example.com/get"
}

Host Name Redirect

The HTTP status code 3XX are used to redirect client to another address. We can redirect all requests to foo.example.com to bar.example.com by responding 301 status and new hostname.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestRedirect
      requestRedirect:
        hostname: bar.example.com
        port: 8000
        statusCode: 301
    backendRefs:
    - name: httpbin
      port: 8080

Now, it will return the 301 code and bar.example.com:8000 when requesting foo.example.com.

curl -i http://foo.example.com:8000/get
HTTP/1.1 301 Moved Permanently
Location: http://bar.example.com:8000/get
content-length: 0
connection: keep-alive

By default, curl does not follow location redirecting unless enable it by assign opiton -L.

curl -L http://foo.example.com:8000/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "bar.example.com:8000",
    "User-Agent": "curl/7.68.0"
  },
  "origin": "10.42.0.161",
  "url": "http://bar.example.com:8000/get"
}

Prefix Path Redirect

With path redirection, we can implement what we did with URL Rewriting: redirect the request to /status/{n} to /stream/{n}.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /status
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /stream
        statusCode: 301
    backendRefs:
    - name: httpbin
      port: 8080
  - matches:
    backendRefs:
    - name: httpbin
      port: 8080

After update rull, we will get.

curl -i http://foo.example.com:8000/status/204
HTTP/1.1 301 Moved Permanently
Location: http://foo.example.com:8000/stream/204
content-length: 0
connection: keep-alive

Full Path Redirect

We can also change full path during redirecting, such as redirect all /status/xxx to /status/200.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /status
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplaceFullPath
          replaceFullPath: /status/200
        statusCode: 301
    backendRefs:
    - name: httpbin
      port: 8080
  - matches:
    backendRefs:
    - name: httpbin
      port: 8080      

Now, the status of requests to /status/xxx will be redirected to /status/200.

curl -i http://foo.example.com:8000/status/204
HTTP/1.1 301 Moved Permanently
Location: http://foo.example.com:8000/status/200
content-length: 0
connection: keep-alive

3.7.4.5 - HTTP Request Header Manipulate

The HTTP header manipulation feature allows you to fine-tune incoming and outgoing request and response headers.

In Gateway API, the HTTPRoute resource utilities two HTTPHeaderFilter filter for request and response header manipulation.

The both filters supports add, set and remove operation. The combination of them is also available.

This document will introduce the HTTP request header manipulation function of FSM Gateway. The introduction of HTTP response header manipulation is located in doc HTTP Response Header Manipulate.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

We will follow the sample in HTTP Routing.

In backend service, there is a path /headers which will respond all request headers.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "10.42.0.15:80",
    "User-Agent": "curl/8.1.2"
  }
}

Add HTTP Request header

With header adding feature, let’s try to add a new header to request by add HTTPHeaderFilter filter.

Modifying the HTTPRoute http-route-foo and add RequestHeaderModifier filter.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add: 
        - name: "header-2-add"
          value: "foo"
EOF

Now request the path /headers again and you will get the new header injected by gateway.

Thought HTTP header name is case insensitive but it will be converted to capital mode.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Header-2-Add": "foo",
    "Host": "10.42.0.15:80",
    "User-Agent": "curl/8.1.2"
  }
}

Set HTTP Request header

set operation is used to update the value of specified header. If the header not exist, it will do as add operation.

Let’s update the HTTPRoute resource again and set two headers with new value. One does not exist and another does.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        set: 
        - name: "header-2-set"
          value: "foo"
        - name: "user-agent"
          value: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"
EOF

In the response, we can get the two headers updated.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Header-2-Set": "foo",
    "Host": "10.42.0.15:80",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"
  }
}

Remove HTTP Request header

The last operation is remove, which can remove the header of client sending.

Let’s update the HTTPRoute resource to remove user-agent header directly to hide client type from backend service.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        remove:
        - "user-agent"
EOF

With resource udpated, the user agent is invisible on backend service side.

curl -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "10.42.0.15:80"
  }
}

3.7.4.6 - HTTP Response Header Manipulate

The HTTP header manipulation feature allows you to fine-tune incoming and outgoing request and response headers.

In Gateway API, the HTTPRoute resource utilities two HTTPHeaderFilter filter for request and response header manipulation.

The both filters supports add, set and remove operation. The combination of them is also available.

This document will introduce the HTTP response header manipulation function of FSM Gateway. The introduction of HTTP request header manipulation is located in doc HTTP Request Header Manipulate.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

We will follow the sample in HTTP Routing.

In backend service responds the generated headers as below.=

curl -I -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Tue, 21 Nov 2023 08:54:43 GMT
content-type: application/json
content-length: 106
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

Add HTTP Response header

With header adding feature, let’s try to add a new header to response by add HTTPHeaderFilter filter.

Modifying the HTTPRoute http-route-foo and add ResponseHeaderModifier filter.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add: 
        - name: "header-2-add"
          value: "foo"
EOF

Now request the path /headers again and you will get the new header in response injected by gateway.

curl -I -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Tue, 21 Nov 2023 08:56:58 GMT
content-type: application/json
content-length: 139
access-control-allow-origin: *
access-control-allow-credentials: true
header-2-add: foo
connection: keep-alive

Set HTTP Response header

set operation is used to update the value of specified header. If the header not exist, it will do as add operation.

Let’s update the HTTPRoute resource again and set two headers with new value. One does not exist and another does.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        set: 
        - name: "header-2-set"
          value: "foo"
        - name: "server"
          value: "fsm/gateway"
EOF

In the response, we can get the two headers updated.

curl -I -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
HTTP/1.1 200 OK
server: fsm/gateway
date: Tue, 21 Nov 2023 08:58:56 GMT
content-type: application/json
content-length: 139
access-control-allow-origin: *
access-control-allow-credentials: true
header-2-set: foo
connection: keep-alive

Remove HTTP Response header

The last operation is remove, which can remove the header of client sending.

Let’s update the HTTPRoute resource to remove server header directly to hide backend implementation from client.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
  - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
    filters:
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        remove:
        - "server"
EOF

With resource udpated, the backend server implementation is invisible on client side.

curl -I -H 'host:foo.example.com' http://$GATEWAY_IP:8000/headers
HTTP/1.1 200 OK
date: Tue, 21 Nov 2023 09:00:32 GMT
content-type: application/json
content-length: 139
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

3.7.4.7 - TCP Routing

This document will describe how to configure FSM Gateway to load balance TCP traffic.

During the L4 load balancing process, FSM Gateway determines which backend server to distribute traffic to based mainly on network layer and transport layer information, such as IP address and port number. This approach allows the FSM Gateway to make decisions quickly and forward traffic to the appropriate server, thereby improving overall network performance.

If you want to load balance HTTP traffic, please refer to the document HTTP Routing.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

Deploy sample

First, let’s install the example in namespace httpbin with commands below.

kubectl create namespace httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/gateway/tcp-routing.yaml

The command above will create Gateway and TCPRoute resources except for sample app ht tpbin.

In Gateway, there are two listener defined listening on ports 8000 and 8001.

listeners:
- protocol: TCP
  port: 8000
  name: foo
  allowedRoutes:
    namespaces:
      from: Same
- protocol: TCP
  port: 8001
  name: bar
  allowedRoutes:
    namespaces:
      from: Same 

The TCPRoute mapping to backend service httpbin is bound to the two ports defined above.

parentRefs:
- name: simple-fsm-gateway
  port: 8000
- name: simple-fsm-gateway
  port: 8001    
rules:
- backendRefs:
  - name: httpbin
    port: 8080

This means we should reach backend service via either of two ports.

Testing

Let’s record the IP address of Gateway first.

export GATEWAY_IP=$(kubectl get svc -n httpbin -l app=fsm-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Sending a request to port 8000 of gateway and it will forward the traffic to backend service.

curl http://$GATEWAY_IP:8000/headers
{
  "headers": {
    "Accept": "*/*",
    "Host": "20.24.88.85:8000",
    "User-Agent": "curl/8.1.2"
  }

With gatweay port 8081, it works fine too.

curl http://$GATEWAY_IP:8001/headers
{
  "headers": {
    "Accept": "*/*",
    "Host": "20.24.88.85:8001",
    "User-Agent": "curl/8.1.2"
  }
}

The path /headers responds all request header received. From the header Host, we can get the entrance.

3.7.4.8 - TLS Termination

TLS offloading is the process of terminating TLS connections at a load balancer or gateway, decrypting the traffic and passing it to the backend server, thereby relieving the backend server of the encryption and decryption burden.

This doc will show you how to use TSL termination for service.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

export GATEWAY_IP=$(kubectl get svc -n httpbin -l app=fsm-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Issue TLS certificate

If configure TLS, a certificate is required. Let’s issue a certificate first.

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout example.com.key -out example.com.crt \
  -subj "/CN=example.com"

With command above executed, you will get two files example.com.crt and example.com.key which we can create a secret with.

kubectl create namespace httpbin
kubectl create secret tls simple-gateway-cert --key=example.com.key --cert=example.com.crt -n httpbin

Deploy sample app

kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/gateway/tls-termination.yaml

Test

curl --cacert example.com.crt https://example.com/headers  --connect-to example.com:443:$GATEWAY_IP:8000
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "example.com",
    "User-Agent": "curl/7.68.0"
  }
}

3.7.4.9 - TLS Passthrough

TLS passthrough means that the gateway does not decrypt TLS traffic, but directly transmits the encrypted data to the back-end server, which decrypts and processes it.

This doc will guide how to use the TLS Passthrought feature.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

We will utilize https://httpbin.org for TLS passthrough testing, functioning similarly to the sample app deployed in other documentation sections.

Create Gateway

First of all, we need to create a gateway to accept incoming request. Different from TLS Termination, the mode is set to Passthrough for the listener.

Let’s create it in namespace httpbin which accepts route resources in same namespace.

kubectl create ns httpbin
kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: simple-fsm-gateway
spec:
  gatewayClassName: fsm-gateway-cls
  listeners:
  - protocol: TLS
    port: 8000
    name: foo
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces:
        from: Same
EOF

Let’s record the IP address of gateway.

export GATEWAY_IP=$(kubectl get svc -n httpbin -l app=fsm-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Create TCP Route

To route encrypted traffic to a backend service without decryption, the use of TLSRoute is necessary in this context.

In the rules.backendRefs configuration, we specify an external service using its host and port. For example, for https://httpbin.org, these would be set as name: httpbin.org and port: 443.

kubectl apply -n httpbin -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: tcp-route
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  rules:
  - backendRefs:
    - name: httpbin.org
      port: 443
EOF

Test

We issue requests to the URL https://httpbin.org, but in reality, these are routed through the gateway.

curl https://httpbin.org/headers --connect-to httpbin.org:443:$GATEWAY_IP:8000
{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.1.2",
    "X-Amzn-Trace-Id": "Root=1-655dd2be-583e963f5022e1004257d331"
  }
}

3.7.4.10 - gRPC Routing

The GRPCRoute is used to route gRPC request to backend service. It can match requests by hostname, gRPC service, gRPC method, or HTTP/2 header.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

Deploy sample

kubectl create namespace grpcbin
kubectl apply -n grpcbin -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/gateway/gprc-routing.yaml

In gRPC case, the listener configuration is similar with HTTP routing.

gRPC Route

We configure the match rule using service: hello.HelloService and method: SayHello to direct traffic to the target service.

rules:
- matches:
  - method:
      service: hello.HelloService
      method: SayHello
  backendRefs:
  - name: grpcbin
    port: 9000

Let’s test our configuration now.

Test

To test gRPC service, we will test with help of the tool grpcurl.

Let’s record the IP address of gateway first.

export GATEWAY_IP=$(kubectl get svc -n grpcbin -l app=fsm-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

Issue a request using the grpcurl command, specifying the service name and method. Doing so will yield the correct response.

grpcurl -plaintext -d '{"greeting":"Flomesh"}' $GATEWAY_IP:8000 hello.HelloService/SayHello
{
  "reply": "hello Flomesh"
}

3.7.4.11 - UDP Routing

The UDPRoute provides a method to route UDP traffic. When combined with a gateway listener, it can be used to forward traffic on a port specified by the listener to a set of backends defined in the UDPRoute.

Prerequisites

• Kubernetes cluster version v1.21.0 or higher.
• kubectl CLI
• FSM Gateway installed via guide doc.

Demonstration

Prerequisites

• Kubernetes cluster
• kubectl tool

Environment Setup

Deploying Sample Application

Use fortio server as a sample application, which provides a UDP service listening on port 8078 and echoes back the content sent by the client.

kubectl create namespace server
kubectl apply -n server -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: fortio
  labels:
    app: fortio
    service: fortio
spec:
  ports:
  - port: 8078
    name: udp-8078
  selector:
    app: fortio
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fortio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fortio
  template:
    metadata:
      labels:
        app: fortio
    spec:
      containers:
      - name: fortio
        image: fortio/fortio:latest_release
        imagePullPolicy: Always
        ports:
        - containerPort: 8078
          name: http
EOF