Overview of FSM

• 1: About FSM
• 2: FSM Components
• 3: FSM Resource Requests and Limits
• 4: Kubernetes Gateway API Compatibility
• 5: Service Mesh Interface (SMI) Support

1 - About FSM

The Flomesh Service Mesh (FSM) inherits a very small portion of archived OSM code.

• FSM utilizes Flomesh Pipy proxy. This enables FSM to achieve lightweight control and data planes, optimizing CPU and memory resources effectively.
• Implemented traffic interception using eBPF-based technology instead of iptables-based traffic interception. To keep conformance with existing users, iptables is the default, while new users during installation can select ebpf option.
• FSM offers comprehensive north-south traffic management capabilities, including Ingress and Gateway APIs.
• Additionally, it facilitates seamless interconnectivity among multiple clusters and incorporates service discovery functionality.

Flomesh Pipy is a programmable network proxy that provides a high-performance, low-latency, and secure way to route traffic between services.

FSM is dedicated to providing a holistic, high-performance, and user-friendly suite of traffic management and service governance capabilities for microservices operating on the Kubernetes platform. By harnessing the combined strengths of FSM and Pipy, we present a dynamic and versatile service mesh solution that empowers Kubernetes-based environments.

Why FSM?

In practice, we have encountered users from a variety of industries with similar requirements for service mesh. These industry users and scenarios include:

• Energy and power companies. They want to build simple server rooms at each substation or gas station to deploy computing and storage capacity for the processing of data generated by devices within the coverage area of that location. They want to push traditional data center applications to these simple rooms and take full advantage of data center application management and operation capabilities
• Telematics service providers. They want to build their simple computing environments in non-data center environments for data collection and service delivery to cars and vehicle owners. These environments may be near highway locations, parking lots, or high-traffic areas
• Retailers. They want to build a minimal computing environment in each store, and in addition to supporting traditional capabilities such as inventory, sales, and payment collection, they also hope to introduce new capabilities of data collection, processing, and transmission
• Medical institutions. They want to provide network capabilities at every hospital, or a simple point of care, so that in addition to providing digital service capability for patients, they can also complete data collection and data linkage with higher management departments at the same time
• Teaching institutions, hospitals, and campuses alike. These campuses are characterized by a relatively regular and dense flow of people. They want to deploy computing resources near more crowd gathering points for delivering digital services, as well as collecting and processing data in real-time

These are typical edge computing scenarios, and they have similar needs:

• Users want to bring the traditional data center computing model, especially microservices, and the related application delivery, management operation, and maintenance capabilities to the edge side
• In terms of the working environment, users have to deal with factors such as power supply, limited computing power, and unstable network. Therefore, computing platforms are required to be more robust and can be deployed quickly or recover a computing environment completely in extreme situations
• The number of locations (we call POP=Point of Presence) that usually need to be deployed is large and constantly evolving and expanding. The cost of a POP point, the price of maintenance, and the price of expansion are all important cost considerations.
• Common, or low-end, PC servers are more often used in these scenarios to replace cloud-standard servers; low-power technology-based computing power such as ARM is further replacing low-end PC servers. On these hardware platforms, which are not comparable to cloud-standard servers, users still want to have enough computing power to handle the growth in functionality and data volume. The conflicting demands of computing moving closer to where the data is generated, the growth in data volume and functional requirements at the edge, and the limited computing resources at the edge require edge-side computing platforms to have better computing power efficiency ratios, i.e., running more applications and supporting larger data volumes with as little power and as few servers as possible
• The fragility and a large number of POP points require better application support for multi-cluster, cross-POP failover. For example, if a POP point fails, the neighboring POP points can quickly share or even temporarily take over the computing tasks.

Compared with the cloud data center computing scenario, the three core and main differences and difficulties of edge computing are:

• Edge computing requires support for heterogeneous hardware architectures. We see non-x86 computing power being widely used at the edge, often with the advantage of low power consumption and low cost
• Edge computing POP points are fragile. This fragility is reflected in the fact that they may not have an extremely reliable power supply, or the power supply is not as powerful as a data center; they may operate in a worse environment than the constant temperature and ventilation of a data center; their networks may be narrowband and unstable
• Edge computing is naturally distributed computing. In almost all edge computing scenarios, there are multiple POP points, and the number of POP points is continuously increasing. the POP points can disaster-proof each other and migrate to adjacent POP points in case of failure, which is a fundamental capability of edge computing

The evolution of Kubernetes to the edge side solves the difficulties of edge computing to a certain extent, especially against fragility; while the development of service mesh to the edge side focuses on network issues in edge computing, against network fragility, as well as providing basic network support for distributed, such as fault migration. In practice, container platforms, as today’s de facto quasi-standard means of application delivery, are rapidly evolving to the edge side, with a large number of releases targeting edge features, such as k3s; but service mesh, as an important network extension for container platforms, are not quickly keeping up with this trend. It is currently difficult for users to find service mesh for edge computing scenarios, so we started the [FSM][1] an open source project with several important considerations and goals, namely

• Support and compatibility with the SMI specification, so that it can meet users’ needs for standardization of service mesh management
• Full support for the ARM ecosystem, which is the “first-class citizen” or even the preferred computing platform for edge computing, and the Service Mesh should be fully adapted to meet this trend. [FSM][1] follows the ARM First strategy, which means that all features are developed, tested, and delivered on the ARM platform first
• High performance and low resources. The service mesh as infrastructure should use fewer resources (CPU/MEM) while delivering higher performance (TPS/Latency) at the edge.

Features

• Light-weight, high-performant, cloud-native, extensible
• Out-of-the-box supports x86, ARM architectures
• Easily and transparently configure traffic shifting for deployments
• Secure service-to-service communication by enabling mutual TLS
• Define and execute fine-grained access control policies for services
• Observability and insights into application metrics for debugging and monitoring services
• Integrate with external certificate management services/solutions with a pluggable interface
• Onboard applications onto the mesh by enabling automatic sidecar injection of Pipy proxy
• Supports Multi-cluster Kubernetes the Kubernetes way by implementing MCS-API
• Supports Ingress/Egress and Kubernetes Gateway API
• Supports protocols like MQTT

Architecture

To break the tight coupling on Pipy and open doors for 3rd parties to develop or make use of their data plane or sidecar proxies, we have refactored the FSM v1.1.x codebase to make it generic and provide extension points. We strongly believe in and support open source and our proposal for this refactoring have been submitted to upstream for their review, discussion, and/or utilization.

2 - FSM Components

Inspect FSM Components

Some FSM components will be installed by default in the chosen namespace, which defaults to fsm-system. Inspect them by using the following kubectl command:

# Replace fsm-system with the namespace where FSM is installed
kubectl get pods,svc,secrets,meshconfigs,serviceaccount --namespace fsm-system

Some cluster-wide (non-namespaced) FSM components will also be installed. Inspect them using the following kubectl command:

kubectl get clusterrolebinding,clusterrole,mutatingwebhookconfiguration

Under the hood, fsm is using Helm libraries to create a Helm release object in the control plane Namespace. The Helm release name is the mesh-name. The helm CLI can also be used to inspect Kubernetes manifests installed in more detail. See the Helm docs for how to install Helm.

helm get manifest fsm --namespace <fsm-namespace>

Components

Let’s take a look at each component:

(1) FSM Controller - Control Plane

The Control Plane plays a key part in operating the service mesh. All proxies are installed as sidecars and establish an mTLS gRPC connection to the Control Plane. The proxies continuously receive configuration updates. This component implements the interfaces required by the specific reverse proxy chosen. FSM implements Pipy Repo.

Besides providing configuration to sidecar proxied, it also distributes configuration to other components, such as FSM Ingress, FSM Gateway and Egress Gateway.

(2) Pipy Sidecar - Sidecar Proxy

FSM uses Pipy as a sidecar proxy

Pipy is a programmable proxy for the cloud, edge and IoT. It’s written in C++, which makes it extremely lightweight and fast. It’s also fully programmable by using PipyJS, a tailored version from the standard JavaScript language.

The sidecar is injected into pod by injector during pod starting. All traffic of pod is intercepted to sidecar proxy. Then sidecar proxy will redirect traffic to local application in same pod or other applications base on the configuration distributed from the Control Plane.

This means that all sidecar proxies handle the traffic of Data Plane and connect to Control Plane for configuration.

(3) FSM Ingress - Ingress Controller

FSM Ingress is an implementation of Kubernegtes Ingress which will handle the traffic from outside the Kubernetes cluster.

It focuses on L7 traffic management and it uses Pipy to handle traffic.

(4) FSM Gateway - Gateway API

Kubernetes Gateway API is another API of Ingress traffic provided by Kubernetes community. It provides much flexibility and extendibility than Ingress API.

FSM Gateway is just the implementation of Gateway API. It also uses Pipy to handle traffic.

(5) Egress Gateway - Egress Traffic Control

Egress Gateway is another standalone component of FSM, but is optional. The traffic sending outside the cluster can be redirected to Egress Gateway first. So the Egress Gateway runs as unified outlet of traffic sent outside cluster.

The underlying traffic processing is completed by Pipy.

(6) Connectors - Discovery Integration

By default, service mesh supports Kubernetes service discovery, but Kubernetes service discovery is not the default implementation for users, or it is not the only one.

Connectors is the componets to support other discovery services, such as HashiCorp Consul, Netflix Eureka, Nacos and so on. Currently, it supports Consul and Eureka only.

3 - FSM Resource Requests and Limits

Note: These are the default values and can be configured in values.yaml

4 - Kubernetes Gateway API Compatibility

Kubernetes Gateway API Compatibility

This document describes which Gateway API resources FSM supports and the extent of that support.

Summary

Terminology

We use the following words to describe support status:

• Supported. The resource or field is fully supported and conformant to the Gateway API specification.
• Partially supported. The resource or field is supported partially or with limitations. It will become fully supported in future releases.
• Not supported. The resource or field is not yet supported. It will become partially or fully supported in future releases.

Resources

Below we list the resources and the support status of their corresponding fields.

For a description of each field, visit the Gateway API documentation.

GatewayClass

Status: Partially supported.

FSM supports only GatewayClass resource whose ControllerName is flomesh.io/gateway-controller. If multiple valid GatewayClasses are created, the oldest is active and take effect.

Fields:

• spec
  • controllerName - supported.
  • parametersRef - not supported.
  • description - supported.
• status
  • conditions - partially supported. Support Accepted type and added ConditionType Active.

Gateway

Status: Partially supported.

FSM supports only a single Gateway resource per namespace. The Gateway resource must reference FSM’s corresponding effective GatewayClass, whose controller name is flomesh.io/gateway-controller. In case of multiple Gateway resources created in the same namespace, FSM will choose the oldest ONE by creation timestamp. If the timestamps are equal, FSM will choose the resource that appears first in alphabetical order by “{name}”. We might support multiple Gateway resources. Due to the limitation of Kubernetes Service of type LoadBalancer, the UDP gateway cannot coexist with gateways of other protocols. If you want to use UDP gateway, you need to create a new one.

Fields:

• spec
  • gatewayClassName - supported.
  • listeners
    • name - supported.
    • hostname - supported.
    • port - supported, must be LTE 60000, all priviliged ports will be mapped to 60000 + port.
    • protocol - supported. Allowed values: HTTP, HTTPS, TLS, TCP, UDP.
    • tls
      • mode - supported. Allowed value: Terminate, Passthrough.
      • certificateRefs - partially supported. The TLS certificate and key must be stored in a Secret. Multiple references are supported. You must deploy the Secrets before the Gateway resource. Secret rotation (watching for updates) is supported.
      • options - not supported.
    • allowedRoutes - not supported.
  • addresses - not supported.

• status
  • addresses - supported.
  • conditions - supported, Accepted type for active Gateway.
  • listeners
    • name - supported.
    • supportedKinds - supported.
      • attachedRoutes - not supported.
      • conditions - partially supported.

HTTPRoute

Status: Partially supported.

Fields:

• spec
  • parentRefs - partially supported. port must always be set.
  • hostnames - supported.
    • matches
      • path - supported, Prefix, Exact and Regex.
      • headers - supported, Exact and Regex.
      • queryParams - supported, Exact and Regex.
      • method - supported.
    • filters
      • type - supported.
      • requestRedirect, requestHeaderModifier, responseHeaderModifier, requestMirror, urlRewrite supported
    • extensionRef - not supported.
    • backendRefs - supported.
• status
  • parents
    • parentRef - supported.
    • controllerName - supported.
    • conditions - partially supported. Supported (Condition/Status/Reason):
      • Accepted/True/Accepted
      • Accepted/False/NoMatchingListenerHostname
      • ResolvedRefs/True/ResolvedRefs

TLSRoute

Status: Partially supported.

Fields:

• spec
  • parentRefs - partially supported. port must always be set.
  • hostnames - supported.
  • backendRefs - supported.
• status
  • parents
    • parentRef - supported.
    • controllerName - supported.
    • conditions - partially supported. Supported (Condition/Status/Reason):
      • Accepted/True/Accepted
      • Accepted/False/NoMatchingListenerHostname
      • ResolvedRefs/True/ResolvedRefs

GRPCRoute

Status: Partially supported.

Fields:

• spec
  • parentRefs - partially supported. port must always be set.
  • hostnames - supported.
  • matches
    • headers
      • type - supported, Exact and Regex.
      • name - supported.
      • value - supported.
    • method:
      • type - supported, Exact and Regex.
      • service - supported.
      • method - supported.
  • filters
    • type - supported.
    • requestHeaderModifier, responseHeaderModifier, requestMirror supported
    • extensionRef - not supported.
• status
  • parents
    • parentRef - supported.
    • controllerName - supported.
    • conditions - partially supported. Supported (Condition/Status/Reason):
      • Accepted/True/Accepted
      • Accepted/False/NoMatchingListenerHostname
      • ResolvedRefs/True/ResolvedRefs

TCPRoute

Status: Partially supported.

Fields:

• spec
  • parentRefs - partially supported. port must always be set.
  • backendRefs - supported.
• status
  • parents
    • parentRef - supported.
    • controllerName - supported.
    • conditions - partially supported. Supported (Condition/Status/Reason):
      • Accepted/True/Accepted
      • ResolvedRefs/True/ResolvedRefs

UDPRoute

Status: Partially supported.

Fields:

• spec
  • parentRefs - partially supported. port must always be set.
  • backendRefs - supported.
• status
  • parents
    • parentRef - supported.
    • controllerName - supported.
    • conditions - partially supported. Supported (Condition/Status/Reason):
      • Accepted/True/Accepted
      • ResolvedRefs/True/ResolvedRefs

ReferenceGrant

Status: Not supported.

Custom Policies

Status: Partially supported.

Custom policies will be FSM-specific CRDs that will allow supporting features like timeouts, load-balancing methods, authentication, etc. - important data-plane features that are not part of the Gateway API spec.

While those CRDs are not part of the Gateway API, the mechanism of attaching them to Gateway API resources is part of the Gateway API. See the Policy Attachment doc.

Listener Protocol and Supported Route Types

5 - Service Mesh Interface (SMI) Support

Overview

FSM implements Service Mesh Interface (SMI) resources. This allows FSM users to have flexible implementations of common service mesh scenarios.

Supported versions

To find out what versions of SMI resources are supported in your mesh, you can run fsm mesh list. Here is an excerpt from sample output:

MESH NAME   MESH NAMESPACE   SMI SUPPORTED
fsm         fsm-system       HTTPRouteGroup:v1alpha4,TCPRoute:v1alpha4,TrafficSplit:v1alpha4,TrafficTarget:v1alpha3,TrafficMetrics:v1alpha1

The following are the currently supported SMI resources and their versions:

SMI Specification support