Operating guides

• 1: Install the FSM CLI
• 2: Install the FSM Control Plane
• 3: Upgrade the FSM Control Plane
• 4: Uninstall the FSM Control Plane and Components
• 5: Mesh configuration
• 6: Reconciler Guide
• 7: Extending FSM

1 - Install the FSM CLI

Prerequisites

• Kubernetes cluster running Kubernetes v1.19.0 or greater

Set up the FSM CLI

From the Binary Releases

Download platform specific compressed package from the Releases page.

Unpack the fsm binary and add it to $PATH to get started.

Linux and macOS

In a bash-based shell on Linux/macOS or Windows Subsystem for Linux, use curl to download the FSM release and then extract with tar as follows:

# Specify the FSM version that will be leveraged throughout these instructions
FSM_VERSION=v1.2.3

# Linux curl command only
curl -sL "https://github.com/flomesh-io/fsm/releases/download/$FSM_VERSION/fsm-$FSM_VERSION-linux-amd64.tar.gz" | tar -vxzf -

# macOS curl command only
curl -sL "https://github.com/flomesh-io/fsm/releases/download/$FSM_VERSION/fsm-$FSM_VERSION-darwin-amd64.tar.gz" | tar -vxzf -

The fsm client binary runs on your client machine and allows you to manage FSM in your Kubernetes cluster. Use the following commands to install the FSM fsm client binary in a bash-based shell on Linux or Windows Subsystem for Linux. These commands copy the fsm client binary to the standard user program location in your PATH.

sudo mv ./linux-amd64/fsm /usr/local/bin/fsm

For macOS use the following commands:

sudo mv ./darwin-amd64/fsm /usr/local/bin/fsm

You can verify the fsm client library has been correctly added to your path and its version number with the following command.

fsm version

From Source (Linux, MacOS)

Building FSM from source requires more steps but is the best way to test the latest changes and useful in a development environment.

You must have a working Go environment and Helm 3 installed.

git clone https://github.com/flomesh-io/fsm.git
cd fsm
make build-fsm

make build-fsm will fetch any required dependencies, compile fsm and place it in bin/fsm. Add bin/fsm to $PATH so you can easily use fsm.

Install FSM

FSM Configuration

By default, the control plane components are installed into a Kubernetes Namespace called fsm-system and the control plane is given a unique identifier attribute mesh-name defaulted to fsm.

During installation, the Namespace and mesh-name can be configured through flags when using the fsm CLI or by editing the values file when using the helm CLI.

The mesh-name is a unique identifier assigned to an fsm-controller instance during install to identify and manage a mesh instance.

The mesh-name should follow RFC 1123 DNS Label constraints. The mesh-name must:

• contain at most 63 characters
• contain only lowercase alphanumeric characters or ‘-’
• start with an alphanumeric character
• end with an alphanumeric character

Using the FSM CLI

Use the fsm CLI to install the FSM control plane on to a Kubernetes cluster.

Run fsm install.

# Install fsm control plane components
fsm install                                                                    
fsm-preinstall[fsm-preinstall-4vb8n] Done
fsm-bootstrap[fsm-bootstrap-cdbccf694-nwm74] Done
fsm-injector[fsm-injector-7c9f5f9cdf-tw99v] Done
fsm-controller[fsm-controller-6d5984fb9f-2nj7s] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

Run fsm install --help for more options.

2 - Install the FSM Control Plane

Prerequisites

• Kubernetes cluster running Kubernetes v1.19.0 or greater
• The FSM CLI or the helm 3 CLI or the OpenShift oc CLI.

Kubernetes support

FSM can be run on Kubernetes versions that are supported at the time of the FSM release. The current support matrix is:

Using the FSM CLI

Use the fsm CLI to install the FSM control plane on to a Kubernetes cluster.

FSM CLI and Chart Compatibility

Each version of the FSM CLI is designed to work only with the matching version of the FSM Helm chart. Many operations may still work when some version skew exists, but those scenarios are not tested and issues that arise when using different CLI and chart versions may not get fixed even if reported.

Running the CLI

Run fsm install to install the FSM control plane.

fsm install
fsm-preinstall[fsm-preinstall-xsmz4] Done
fsm-bootstrap[fsm-bootstrap-7f59b7bf7-rs55z] Done
fsm-injector[fsm-injector-787bc867db-54gl6] Done
fsm-controller[fsm-controller-58d758b7fb-2zrr8] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

Run fsm install --help for more options.

Note: Installing FSM via the CLI enforces deploying only one mesh in the cluster. FSM installs and manages the CRDs by adding a conversion webhook field to all the CRDs to support multiple API versions, which ties the CRDs to a specific instance of FSM. Hence, for FSM’s correct operation it is strongly recommended to have only one FSM mesh per cluster.

Using the Helm CLI

The FSM chart can be installed directly via the Helm CLI.

Editing the Values File

You can configure the FSM installation by overriding the values file.

1. Create a copy of the values file (make sure to use the version for the chart you wish to install).

2. Change any values you wish to customize. You can omit all other values.

   • To see which values correspond to the MeshConfig settings, see the FSM MeshConfig documentation

   • For example, to set the logLevel field in the MeshConfig to info, save the following as override.yaml:

     fsm:
       sidecarLogLevel: info
     

Helm install

Then run the following helm install command. The chart version can be found in the Helm chart you wish to install here.

helm install <mesh name> fsm --repo https://flomesh-io.github.io/fsm --version <chart version> --namespace <fsm namespace> --create-namespace --values override.yaml

Omit the --values flag if you prefer to use the default settings.

Run helm install --help for more options.

OpenShift

To install FSM on OpenShift:

1. Enable privileged init containers so that they can properly program iptables. The NET_ADMIN capability is not sufficient on OpenShift.

   fsm install --set="fsm.enablePrivilegedInitContainer=true"
   

   • If you have already installed FSM without enabling privileged init containers, set enablePrivilegedInitContainer to true in the FSM MeshConfig and restart any pods in the mesh.

2. Add the privileged security context constraint to each service account in the mesh.

   • Install the oc CLI.

   • Add the security context constraint to the service account

      oc adm policy add-scc-to-user privileged -z <service account name> -n <service account namespace>
     

Pod Security Policy

Deprecated: PSP support has been deprecated in FSM since v0.10.0

PSP support will be removed in FSM 1.0.0

If you are running FSM in a cluster with PSPs enabled, pass in --set fsm.pspEnabled=true to your fsm install or helm install CLI command.

Enable Reconciler in FSM

If you wish to enable a reconciler in FSM, pass in --set fsm.enableReconciler=true to your fsm install or helm install CLI command. More information on the reconciler can be found in the Reconciler Guide.

Inspect FSM Components

A few components will be installed by default. Inspect them by using the following kubectl command:

# Replace fsm-system with the namespace where FSM is installed
kubectl get pods,svc,secrets,meshconfigs,serviceaccount --namespace fsm-system

A few cluster wide (non Namespaced components) will also be installed. Inspect them using the following kubectl command:

kubectl get clusterrolebinding,clusterrole,mutatingwebhookconfiguration,validatingwebhookconfigurations -l app.kubernetes.io/name=flomesh.io

Under the hood, fsm is using Helm libraries to create a Helm release object in the control plane Namespace. The Helm release name is the mesh-name. The helm CLI can also be used to inspect Kubernetes manifests installed in more detail. Goto https://helm.sh for instructions to install Helm.

# Replace fsm-system with the namespace where FSM is installed
helm get manifest fsm --namespace fsm-system

Next Steps

Now that the FSM control plane is up and running, add services to the mesh.

3 - Upgrade the FSM Control Plane

This guide describes how to upgrade the FSM control plane.

How upgrades work

FSM’s control plane lifecycle is managed by Helm and can be upgraded with Helm’s upgrade functionality, which will patch or replace control plane components as needed based on changed values and resource templates.

Resource availability during upgrade

Since upgrades may include redeploying the fsm-controller with the new version, there may be some downtime of the controller. While the fsm-controller is unavailable, there will be a delay in processing new SMI resources, creating new pods to be injected with a proxy sidecar container will fail, and mTLS certificates will not be rotated.

Already existing SMI resources will be unaffected, this means that the data plane (which includes the Pipy sidecar configs) will also be unaffected by upgrading.

Data plane interruptions are expected if the upgrade includes CRD changes. Streamlining data plane upgrades is being tracked in issue #512.

Policy

Only certain upgrade paths are tested and supported.

Note: These plans are tentative and subject to change.

Breaking changes in this section refer to incompatible changes to the following user-facing components:

• fsm CLI commands, flags, and behavior
• SMI CRDs and controllers

This implies the following are NOT user-facing and incompatible changes are NOT considered “breaking” as long as the incompatibility is handled by user-facing components:

• Chart values.yaml
• fsm-mesh-config MeshConfig
• Internally-used labels and annotations (monitored-by, injection, metrics, etc.)

Upgrades are only supported between versions that do not include breaking changes, as described below.

For FSM versions 0.y.z:

• Breaking changes will not be introduced between 0.y.z and 0.y.z+1
• Breaking changes may be introduced between 0.y.z and 0.y+1.0

For FSM versions x.y.z where x >= 1:

• Breaking changes will not be introduced between x.y.z and x.y+1.0 or between x.y.z and x.y.z+1
• Breaking changes may be introduced between x.y.z and x+1.0.0

How to upgrade FSM

The recommended way to upgrade a mesh is with the fsm CLI. For advanced use cases, helm may be used.

CRD Upgrades

Because Helm does not manage CRDs beyond the initial installation, FSM leverages an init-container on the fsm-bootstrap pod to to update existing and add new CRDs during an upgrade. If the new release contains updates to existing CRDs or adds new CRDs, the init-fsm-bootstrap on the fsm-bootstrap pod will update the CRDs. The associated Custom Resources will remain as is, requiring no additional action prior to or immediately after the upgrade.

Please check the CRD Updates section of the release notes to see if any updates have been made to the CRDs used by FSM. If the version of the Custom Resources are within the versions the updated CRD supports, no immediate action is required. FSM implements a conversion webhook for all of its CRDs, ensuring support for older versions and providing the flexibilty to update Custom Resources at a later point in time.

Upgrading with the FSM CLI

Pre-requisites

• Kubernetes cluster with the FSM control plane installed
  • Ensure that the Kubernetes cluster has the minimum Kubernetes version required by the new FSM chart. This can be found in the Installation Pre-requisites
• fsm CLI installed
  • By default, the fsm CLI will upgrade to the same chart version that it installs. e.g. v0.9.2 of the fsm CLI will upgrade to v0.9.2 of the FSM Helm chart. Upgrading to any other version of the Helm chart than the version matching the CLI may work, but those scenarios are not tested and issues that arise may not get fixed even if reported.

The fsm mesh upgrade command performs a helm upgrade of the existing Helm release for a mesh.

Basic usage requires no additional arguments or flags:

fsm mesh upgrade
FSM successfully upgraded mesh fsm

This command will upgrade the mesh with the default mesh name in the default FSM namespace. Values from the previous release will NOT carry over to the new release by default, but may be passed individually with the --set flag on fsm mesh upgrade.

See fsm mesh upgrade --help for more details

Upgrading with Helm

Pre-requisites

• Kubernetes cluster with the FSM control plane installed
• The helm 3 CLI

FSM Configuration

When upgrading, any custom settings used to install or run FSM may be reverted to the default, this only includes any metrics deployments. Please ensure that you carefully follow the guide to prevent these values from being overwritten.

To preserve any changes you’ve made to the FSM configuration, use the helm --values flag. Create a copy of the values file (make sure to use the version for the upgraded chart) and change any values you wish to customize. You can omit all other values.

**Note: Any configuration changes that go into the MeshConfig will not be applied during upgrade and the values will remain as is prior to the upgrade. If you wish to update any value in the MeshConfig you can do so by patching the resource after an upgrade.

For example, if the logLevel field in the MeshConfig was set to info prior to upgrade, updating this in override.yaml will during an upgrade will not cause any change.

Warning: Do NOT change fsm.meshName or fsm.fsmNamespace

Helm Upgrade

Then run the following helm upgrade command.

helm upgrade <mesh name> fsm --repo https://flomesh-io.github.io/fsm --version <chart version> --namespace <fsm namespace> --values override.yaml

Omit the --values flag if you prefer to use the default settings.

Run helm upgrade --help for more options.

Upgrading Third Party Dependencies

Pipy

Pipy versions can be updated by modifying the value of the sidecarImage variable in fsm-mesh-config. For example, to update Pipy image to latest (this is for example only, the latest image is not recommended), the next command should be run.

export fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl patch meshconfig fsm-mesh-config -n $fsm_namespace -p '{"spec":{"sidecar":{"sidecarImage": "flomesh/pipy:latest"}}}' --type=merge

After the MeshConfig resource has been updated, all Pods and deployments that are part of the mesh must be restarted so that the updated version of the Pipy sidecar can be injected onto the Pod as part of the automated sidecar injection performed by FSM. This can be done with the kubectl rollout restart deploy command.

Prometheus, Grafana, and Jaeger

If enabled, FSM’s Prometheus, Grafana, and Jaeger services are deployed alongside other FSM control plane components. Though these third party dependencies cannot be updated through the meshconfig like Pipy, the versions can still be updated in the deployment directly. For instance, to update prometheus to v2.19.1, the user can run:

export fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM is installed
kubectl set image deployment/fsm-prometheus -n $fsm_namespace prometheus="prom/prometheus:v2.19.1"

To update to Grafana 8.1.0, the command would look like:

kubectl set image deployment/fsm-grafana -n $fsm_namespace grafana="grafana/grafana:8.1.0"

And for Jaeger, the user would run the following to update to 1.26.0:

kubectl set image deployment/jaeger -n $fsm_namespace jaeger="jaegertracing/all-in-one:1.26.0"

FSM Upgrade Troubleshooting Guide

FSM Mesh Upgrade Timing Out

Insufficient CPU

If the fsm mesh upgrade command is timing out, it could be due to insufficient CPU.

1. Check the pods to see if any of them aren’t fully up and running

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl get pods -n fsm-system

2. If there are any pods that are in Pending state, use kubectl describe to check the Events section

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl describe pod <pod-name> -n fsm-system

If you see the following error, then please increase the number of CPUs Docker can use.

`Warning  FailedScheduling  4s (x15 over 19m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.`

Error Validating CLI Parameters

If the fsm mesh upgrade command is still timing out, it could be due to a CLI/Image Version mismatch.

1. Check the pods to see if any of them aren’t fully up and running

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl get pods -n fsm-system

2. If there are any pods that are in Pending state, use kubectl describe to check the Events section for Error Validating CLI parameters

# Replace fsm-system with fsm-controller's namespace if using a non-default namespace
kubectl describe pod <pod-name> -n fsm-system

3. If you find the error, please check the pod’s logs for any errors

kubectl logs -n fsm-system <pod-name> | grep -i error

If you see the following error, then it’s due to a CLI/Image Version mismatch.

`"error":"Please specify the init container image using --init-container-image","reason":"FatalInvalidCLIParameters"`

Workaround is to set the container-registry and fsm-image-tag flag when running fsm mesh upgrade.

fsm mesh upgrade --container-registry $CTR_REGISTRY --fsm-image-tag $CTR_TAG --enable-egress=true

Other Issues

If you’re running into issues that are not resolved with the steps above, please open a GitHub issue.

4 - Uninstall the FSM Control Plane and Components

This guide describes how to uninstall FSM from a Kubernetes cluster. This guide assumes there is a single FSM control plane (mesh) running. If there are multiple meshes in a cluster, repeat the process described for each control plane in the cluster before uninstalling any cluster wide resources at the end of the guide. Taking into consideration both the control plane and dataplane, this guide aims to walk through uninstalling all remnants of FSM with minimal downtime.

Prerequisites

• Kubernetes cluster with FSM installed
• The kubectl CLI
• The FSM CLI or the Helm 3 CLI

Remove Pipy Sidecars from Application Pods and Pipy Secrets

The first step to uninstalling FSM is to remove the Pipy sidecar containers from application pods. The sidecar containers enforce traffic policies. Without them, traffic will flow to and from Pods according in accordance with default Kubernetes networking unless there are Kubernetes Network Policies applied.

FSM Pipy sidecars and related secrets will be removed in the following steps:

1. Disable automatic sidecar injection
2. Restart pods

Disable Automatic Sidecar Injection

FSM Automatic Sidecar Injection is most commonly enabled by adding namespaces to the mesh via the fsm CLI. Use the fsm CLI to see which namespaces have sidecar injection enabled. If there are multiple control planes installed, be sure to specify the --mesh-name flag.

View namespaces in a mesh:

fsm namespace list --mesh-name=<mesh-name>
NAMESPACE          MESH           SIDECAR-INJECTION
<namespace1>       <mesh-name>    enabled
<namespace2>       <mesh-name>    enabled

Remove each namespace from the mesh:

fsm namespace remove <namespace> --mesh-name=<mesh-name>
Namespace [<namespace>] successfully removed from mesh [<mesh-name>]

This will remove the flomesh.io/sidecar-injection: enabled annotation and flomesh.io/monitored-by: <mesh name> label from the namespace.

Alternatively, if sidecar injection is enabled via annotations on pods instead of per namespace, please modify the pod or deployment spec to remove the sidecar injection annotation.

Restart Pods

Restart all pods running with a sidecar:

# If pods are running as part of a Kubernetes deployment
# Can use this strategy for daemonset as well
kubectl rollout restart deployment <deployment-name> -n <namespace>

# If pod is running standalone (not part of a deployment or replica set)
kubectl delete pod <pod-name> -n namespace
k apply -f <pod-spec> # if pod is not restarted as part of replicaset

Now, there should be no FSM Pipy sidecar containers running as part of the applications that were once part of the mesh. Traffic is no longer managed by the FSM control plane with the mesh-name used above. During this process, your applications may experience some downtime as all the Pods are restarting.

Uninstall FSM Control Plane and Remove User Provided Resources

The FSM control plane and related components will be uninstalled in the following steps:

• Prerequisites
• Remove Pipy Sidecars from Application Pods and Pipy Secrets
  • Disable Automatic Sidecar Injection
  • Restart Pods
• Uninstall FSM Control Plane and Remove User Provided Resources
  • Uninstall the FSM control plane
  • Remove User Provided Resources
  • Delete FSM Namespace
  • Removal of FSM Cluster Wide Resources

Uninstall the FSM control plane

Use the fsm CLI to uninstall the FSM control plane from a Kubernetes cluster. The following step will remove:

1. FSM controller resources (deployment, service, mesh config, and RBAC)
2. Prometheus, Grafana, Jaeger, and Fluent Bit resources installed by FSM
3. Mutating webhook and validating webhook
4. The conversion webhook fields patched by FSM to the CRDs installed/required by FSM: CRDs for FSM will be unpatched. To delete cluster wide resources refer to Removal of FSM Cluster Wide Resources for more details.

Run fsm uninstall mesh:

# Uninstall fsm control plane components
fsm uninstall mesh --mesh-name=<mesh-name>
Uninstall FSM [mesh name: <mesh-name>] ? [y/n]: y
FSM [mesh name: <mesh-name>] uninstalled

Run fsm uninstall mesh --help for more options.

Alternatively, if you used Helm to install the control plane, run the following helm uninstall command:

helm uninstall <mesh name> --namespace <fsm namespace>

Run helm uninstall --help for more options.

Remove User Provided Resources

If any resources were provided or created for FSM at install time, they can be deleted at this point.

For example, if Hashicorp Vault was deployed for the sole purpose of managing certificates for FSM, all related resources can be deleted.

Delete FSM Namespace

When installing a mesh, the fsm CLI creates the namespace the control plane is installed into if it does not already exist. However, when uninstalling the same mesh, the namespace it lives in does not automatically get deleted by the fsm CLI. This behavior occurs because there may be resources a user created in the namespace that they may not want automatically deleted.

If the namespace was only used for FSM and there is nothing that needs to be kept around, the namespace can be deleted at the time of uninstall or later using the following command.

fsm uninstall mesh --delete-namespace

Warning: Only delete the namespace if resources in the namespace are no longer needed. For example, if fsm was installed in kube-system, deleting the namespace may delete important cluster resources and may have unintended consequences.

Removal of FSM Cluster Wide Resources

On installation FSM ensures that all the CRDs mentioned here exist in the cluster at install time. During installation, if they are not already installed, the fsm-bootstrap pod will install them before the rest of the control plane components are running. This is the same behavior when using the Helm charts to install FSM as well.

Uninstalling the mesh in both unmanaged and managed environments:

1. removes FSM control plane components, including control plane pods
2. removes/un-patches the conversion webhook fields from all the CRDs (which FSM adds to support multiple CR versions)

leaving behind certain FSM resources to prevent unintended consequences for the cluster after uninstalling FSM.The resources that are left behind will depend on whether FSM was uninstalled from a managed or unmanaged cluster environment.

When uninstalling FSM, both the fsm uninstall mesh command and Helm uninstallation will not delete any FSM or SMI CRD in any cluster environment (managed and unmanaged) for primarily two reasons:

1. CRDs are cluster-wide resources and may be used by other service meshes or resources running in the same cluster
2. deletion of a CRD will cause all custom resources corresponding to that CRD to also be deleted

To remove cluster wide resources that FSM installs (i.e. the meshconfig, secrets, FSM CRDs, SMI CRDs, and webhook configurations), the following command can be run during or after FSM’s uninstillation.

fsm uninstall mesh --delete-cluster-wide-resources

Warning: Deletion of a CRD will cause all custom resources corresponding to that CRD to also be deleted.

To troubleshoot FSM uninstallation, refer to the uninstall troubleshooting section

5 - Mesh configuration

FSM deploys a MeshConfig resource fsm-mesh-config as a part of its control plane (in the same namespace as that of the fsm-controller pod) which can be updated by the mesh owner/operator at any time. The purpose of this MeshConfig is to provide the mesh owner/operator the ability to update some of the mesh configurations based on their needs.

At the time of install, the FSM MeshConfig is deployed from a preset MeshConfig (preset-mesh-config) which can be found under charts/fsm/templates.

First, set an environment variable to refer to the namespace where fsm was installed.

export FSM_NAMESPACE=fsm-system # Replace fsm-system with the namespace where FSM is installed

To view your fsm-mesh-config in CLI use the kubectl get command.

kubectl get meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -o yaml

Note: Values in the MeshConfig fsm-mesh-config are persisted across upgrades.

Configure FSM MeshConfig

Kubectl Patch Command

Changes to fsm-mesh-config can be made using the kubectl patch command.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":true}}}'  --type=merge

Refer to the Config API reference for more information.

If an incorrect value is used, validations on the MeshConfig CRD will prevent the change with an error message explaining why the value is invalid.

For example, the below command shows what happens if we patch enableEgress to a non-boolean value.

kubectl patch meshconfig fsm-mesh-config -n "$FSM_NAMESPACE" -p '{"spec":{"traffic":{"enableEgress":"no"}}}'  --type=merge
# Validations on the CRD will deny this change
The MeshConfig "fsm-mesh-config" is invalid: spec.traffic.enableEgress: Invalid value: "string": spec.traffic.enableEgress in body must be of type boolean: "string"

Kubectl Patch Command for Each Key Type

Note: <fsm-namespace> refers to the namespace where the fsm control plane is installed. By default, the fsm namespace is fsm-system.

6 - Reconciler Guide

This guide describes how to enable the reconciler in FSM.

How the reconciler works

The goal of building a reconciler in FSM is to ensure resources required for the correct operation of FSM’s control plane are in their desired state at all times. Resources that are installed as a part of FSM install and have the labels flomesh.io/reconcile: true and app.kubernetes.io/name: flomesh.io will be reconciled by the reconciler.

Note: The reconciler will not operate as desired if the lables flomesh.io/reconcile: true and app.kubernetes.io/name: flomesh.io are modified or deleted on the reconcilable resources.

An update or delete event on the reconcilable resources will trigger the reconciler and it will reconcile the resource back to its desired state. Only metadata changes (excluding a name change) will be permitted on the reconcilable resources.

Resources reconciled

The resources that FSM reconciles are:

• CRDs : The CRDs installed/required by FSM CRDs for FSM will be reconciled. Since FSM manages the installation and upgrade of the CRDs it needs, FSM will also reconcile them to ensure that their spec, stored and served verions are always in the state that is required by FSM.
• MutatingWebhookConfiguration : A MutatingWebhookConfiguration is deployed as a part of FSM’s control plane to enable automatic sidecar injection. As this is a very critical component for pods joining the mesh, FSM reconciles this resource.
• ValidatingWebhookConfiguration : A ValidatingWebhookConfiguration is deployed as a part of FSM’s control plane to validate various mesh configurations. This resources validates configurations being applied to the mesh, hence FSM will reconcile this resource.

How to install FSM with the reconciler

To install FSM with the reconciler, use the below command:

fsm install --set fsm.enableReconciler=true
fsm-preinstall[fsm-preinstall-zqmxm] Done
fsm-bootstrap[fsm-bootstrap-7f59b7bf7-vf96p] Done
fsm-injector[fsm-injector-787bc867db-m5wxk] Done
fsm-controller[fsm-controller-58d758b7fb-46v4k] Done
FSM installed successfully in namespace [fsm-system] with mesh name [fsm]

7 - Extending FSM

Extending FSM with Plugin Interface

In the latest 1.3.0 version of Flomesh service mesh FSM, we have introduced a significant feature: Plugin. This feature aims to provide developers with a way to extend the functionality of the service mesh without changing the FSM itself.

Nowadays, service mesh seems to be developing in two directions. One is like Istio, which provides a lot of ready-to-use functions and is very rich in features. The other like Linkerd, Flomesh FSM, and others that uphold the principle of simplicity and provide a minimum functional set that meets the user’s needs. There is no superiority or inferiority between the two: the former is rich in features but inevitably has the additional overhead of proxy, not only in resource consumption but also in the cost of learning and maintenance; the latter is easy to learn and use, consumes fewer resources, but the provided functions might not be enough for the immediate need of user desired functionality.

It is not difficult to imagine that the ideal solution is the low cost of the minimum functional set + the flexibility of scalability. The core of the service mesh is in the data plane, and the flexibility of scalability requires a high demand for the physique of the sidecar proxy. This is also why the Flomesh service mesh chose programmable proxy Pipy as the sidecar proxy.

Pipy is a programmable network proxy for cloud, edge, and IoT. It is flexible, fast, small, programmable, and open-source. The modular design of Pipy provides a large number of reusable filters that can be assembled into pipelines to process network data. Pipy provides a set of api and small usable filters to achieve business objectives while hiding the underlying details. Additionally, Pipy scripts (programming code that implements functional logic) can be dynamically delivered to Pipy instances over the network, enabling the proxy to be extended with new features without the need for compilation or restart.

Flomesh FSM extension solution

FSM provides three new CRDs for extensibility:

• Plugin: The plugin contains the code logic for the new functionality. The default functions provided by FSM are also available as plugins, but not in the form of a Plugin resource. These plugins can be adjusted through the Helm values file when installing FSM. For more information, refer to the built-in plugin list in the Helm values.yaml file.
• PluginChain: The plugin chain is the execution of plugins in sequence. The system provides four plugin chains: inbound-tcp, inbound-http, outbound-tcp, outbound-http. They correspond to the OSI layer-4 and layer-7 processing stages of inbound and outbound traffic, respectively.
• PluginConfig: The plugin configuration provides the configuration required for the plugin logic to run, which will be sent to the FSM sidecar proxy in JSON format.

For detailed information on plugin CRDs, refer to the Plugin API document.

Built-in variables

Below is a list of built-in PipyJS variables which can be imported into your custom plugins via PipyJS import keyword.

Demo

For a simple demonstration of how to extend FSM via Plugins, refer to below demo:

• Adding Identity and Access Management functionality