Application onboarding

• 1: Prerequisites
• 2: Namespace addition
• 3: Sidecar Injection
• 4: Application Protocol Selection

fsm namespace remove <namespace>

1 - Prerequisites

Security Contexts

• Do not run applications with the user ID (UID) value of 1500. This is reserved for the Pipy proxy sidecar container injected into pods by FSM’s sidecar injector.
• If security context runAsNonRoot is set to true at the pod level, a runAsUser value must be provided either for the pod or for each container. For example:

    securityContext:
      runAsNonRoot: true
      runAsUser: 1200
  

  If the UID is omitted, application containers may attempt to run as root user by default, causing conflict with the pod’s security context.
• Additional capabilities are not required.

Note: the FSM init container is programmed to run as root and add capability NET_ADMIN as it requires these security contexts to finish scheduling. These values are not changed by application security contexts.

Ports

Do not use the following ports as they are used by the Pipy sidecar.

2 - Namespace addition

Overview

When setting up an FSM control plane (also referred to as a “mesh”), one can also enroll a set of Kubernetes namespaces to the mesh. Enrolling a namespace to FSM allows FSM to monitor the resources within that Namespace whether they be applications deployed in Pods, Services, or even traffic policies represented as SMI resources.

Only one mesh can monitor a namespace, so this is something to watch out for when there are multiple instances of FSM within the same Kubernetes cluster. When applying policies to applications, FSM will only assess resources in either monitored namespaces so it is important to enroll namespaces where your applications are deployed to the correct instance of FSM with the correct mesh name. Enrolling a namespace also optionally allows for metrics to be collected for resources in the given namespace and for Pods in the namespace to be automatically injected with sidecar proxy containers. These are all features that help FSM provide functionality for traffic management and observability. Scoping this functionality at the namespace level allows teams to organize which segments of their cluster should be part of which mesh.

Namespace monitoring, automatic sidecar injection, and metrics collection is controlled by adding certain labels and annotations to a Kubernetes namespace. This can be done manually or using the fsm CLI although using the fsm CLI is the recommended approach. The presence of the label flomesh.io/monitored-by=<mesh-name> allows an FSM control plane with the given mesh-name to monitor all resources within that namespace. The annotation flomesh.io/sidecar-injection=enabled enables FSM to automatically inject sidecar proxy containers in all Pods created within that namespace. The metrics annotation flomesh.io/metrics=enabled allows FSM to collect metrics on resources within a Namespace.

See how to use the FSM CLI to manage namespace monitoring below.

Adding a Namespace to the FSM Control Plane

Add a namespace for monitoring and sidecar injection to the mesh with the following command:

fsm namespace add <namespace>

Explicitly disable sidecar injection while adding the namespace using --disable-sidecar-injection flag as shown here.

Remove a Namespace from the FSM control plane

Remove a namespace from being monitored by the mesh and disable sidecar injection with the following command:

fsm namespace remove <namespace>

This command will remove the FSM specific labels and annotations on the namespace thus removing it from the mesh.

Enable Metrics for a Namespace

fsm metrics enable --namespace <namespace>

Ignore a Namespace

There may be namespaces in a cluster that should never be part of a mesh. To explicity exclude a namespace from FSM:

fsm namespace ignore <namespace>

List Namespaces Part of a Mesh

To list namespaces within a specific mesh:

fsm namespace list --mesh-name=<mesh-name>

Troubleshooting Guide

Policy Issues

If you’re not seeing changes in SMI policies being applied to resources in a namespace, ensure the namespace is enrolled in the correct mesh:

fsm namespace list --mesh-name=<mesh-name>

NAMESPACE         MESH   SIDECAR-INJECTION
<namespace>       fsm    enabled

If the namespace does not show up, check the labels on the namespace using kubectl:

kubectl get namespace <namespace> --show-labels

NAME          STATUS   AGE   LABELS
<namespace>   Active   36s   flomesh.io/monitored-by=<mesh-name>

If the label value is not the expected mesh-name, remove the namespace from the mesh and add it back using the correct mesh-name.

fsm namespace remove <namespace> --mesh-name=<current-mesh-name>
fsm namespace add <namespace> --mesh-name=<expected-mesh-name>

If the monitored-by label is not present, it was either not added to the mesh or there was an error when adding it to the mesh. Add the namespace to the mesh either with the fsm CLI or using kubectl:

fsm namespace add <namespace> --mesh-name=<mesh-name>

kubectl label namespace <namespace> flomesh.io/monitored-by=<mesh-name>

Issues with Automatic Sidecar Injection

If you’re not seeing your Pods being automatically injected with sidecar containers, ensure that sidecar injection is enabled:

fsm namespace list --mesh-name=<mesh-name>

NAMESPACE         MESH   SIDECAR-INJECTION
<namespace>       fsm    enabled

If the namespace does not show up, check the annotations on the namespace using kubectl:

kubectl get namespace <namespace> -o=jsonpath='{.metadata.annotations.flomesh\.io\/sidecar-injection}'

If the output is anything other than enabled, either add namespace using the fsm CLI or add the annotation with kubectl:

fsm namespace add <namespace> --mesh-name=<mesh-name> --disable-sidecar-injection=false

kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=enabled --overwrite

Issues with Metrics Collection

If you’re not seeing metrics for resources in a particular namespace, ensure metrics are enabled:

kubectl get namespace <namespace> -o=jsonpath='{.metadata.annotations.flomesh\.io\/metrics}'

If the output is anything other than enabled, enable the namespace usng the fsm CLI or add the annotation with kubectl:

fsm metrics enable --namespace <namespace>

kubectl annotate namespace <namespace> flomesh.io/metrics=enabled --overwrite

Other Issues

If you’re running into issues that have not been resolved with the debugging techniques above, please open a GitHub issue on the repository.

3 - Sidecar Injection

Services participating in the service mesh communicate via sidecar proxies installed on pods backing the services. The following sections describe the sidecar injection workflow in FSM.

Automatic Sidecar Injection

Automatic sidecar injection is currently the only way to inject sidecars into the service mesh. Sidecars can be automatically injected into applicable Kubernetes pods using a mutating webhook admission controller provided by FSM.

Automatic sidecar injection can be configured per namespace as a part of enrolling a namespace into the mesh, or later using the Kubernetes API. Automatic sidecar injection can be enabled either on a per namespace or per pod basis by annotating the namespace or pod resource with the sidecar injection annotation. Individual pods and namespaces can be explicitly configured to either enable or disable automatic sidecar injection, giving users the flexibility to control sidecar injection on pods and namespaces.

Enabling Automatic Sidecar Injection

Prerequisites:

• The namespace to which the pods belong must be a monitored namespace that is added to the mesh using the fsm namespace add command.
• The namespace to which the pods belong must not be set to be ignored using the fsm namespace ignore command.
• The namespace to which the pods belong must not have a label with key name and value corresponding to the FSM control plane namespace. For example, a namespace with a label name: fsm-system where fsm-system is the control plane namespace cannot have sidecar injection enabled for pods in this namespace.
• The pod must not have hostNetwork: true in the pod spec. Pods with hostNetwork: true are not injected with a sidecar since doing so can result in routing failures in the host network.

Automatic Sidecar injection can be enabled in the following ways:

• While enrolling a namespace into the mesh using fsm cli: fsm namespace add <namespace>: Automatic sidecar injection is enabled by default with this command.

• Using kubectl to annotate individual namespaces and pods to enable sidecar injection:

  # Enable sidecar injection on a namespace
  $ kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=enabled
  

  # Enable sidecar injection on a pod
  $ kubectl annotate pod <pod> flomesh.io/sidecar-injection=enabled
  

• Setting the sidecar injection annotation to enabled in the Kubernetes resource spec for a namespace or pod:

  metadata:
    name: test
    annotations:
      'flomesh.io/sidecar-injection': 'enabled'
  

  Pods will be injected with a sidecar ONLY if the following conditions are met:

  1. The namespace to which the pod belongs is a monitored namespace.
  2. The pod is explicitly enabled for the sidecar injection, OR the namespace to which the pod belongs is enabled for the sidecar injection and the pod is not explicitly disabled for sidecar injection.

Explicitly Disabling Automatic Sidecar Injection on Namespaces

Namespaces can be disabled for automatic sidecar injection in the following ways:

• While enrolling a namespace into the mesh using fsm cli: fsm namespace add <namespace> --disable-sidecar-injection: If the namespace was previously enabled for sidecar injection, it will be disabled after running this command.

• Using kubectl to annotate individual namespaces to disable sidecar injection:

  # Disable sidecar injection on a namespace
  $ kubectl annotate namespace <namespace> flomesh.io/sidecar-injection=disabled
  

Explicitly Disabling Automatic Sidecar Injection on Pods

Individual pods can be explicitly disabled for sidecar injection. This is useful when a namespace is enabled for sidecar injection but specific pods should not be injected with sidecars.

• Using kubectl to annotate individual pods to disable sidecar injection:

  # Disable sidecar injection on a pod
  $ kubectl annotate pod <pod> flomesh.io/sidecar-injection=disabled
  

• Setting the sidecar injection annotation to disabled in the Kubernetes resource spec for the pod:

  metadata:
    name: test
    annotations:
      'flomesh.io/sidecar-injection': 'disabled'
  

Automatic sidecar injection is implicitly disabled for a namespace when it is removed from the mesh using the fsm namespace remove command.

4 - Application Protocol Selection

FSM is capable of routing different application protocols such as HTTP, TCP, and gRPC differently. The following guide describes how to configure service ports to specify the application protocol to use for traffic filtering and routing.

Configuring the application protocol

Kubernetes services expose one or more ports. A port exposed by an application running the service can serve a specific application protocol such as HTTP, TCP, gRPC etc. Since FSM filters and routes traffic for different application protocols differently, a configuration on the Kubernetes service object is necessary to convey to FSM how traffic directed to a service port must be routed.

In order to determine the application protocol served by a service’s port, FSM expects the appProtocol field on the service’s port to be set.

FSM supports the following application protocols for service ports:

1. http: For HTTP based filtering and routing of traffic
2. tcp: For TCP based filtering and routing of traffic
3. tcp-server-first: For TCP based filtering and routing of traffic where the server initiates communication with a client, such as mySQL, PostgreSQL, and others
4. gRPC: For HTTP2 based filtering and routing of gRPC traffic

The application protocol configuration described is applicable to both SMI and Permissive traffic policy modes.

Examples

Consider the following SMI traffic access and traffic specs policies:

• A TCPRoute resource named tcp-route that specifies the port TCP traffic should be allowed on.
• An HTTPRouteGroup resource named http-route that specifies the HTTP routes for which HTTP traffic should be allowed.
• A TrafficTarget resource named test that allows pods in the service account sa-2 to access pods in the service account sa-1 for the specified TCP and HTTP rules.

kind: TCPRoute
metadata:
  name: tcp-route
spec:
  matches:
    ports:
    - 8080
---
kind: HTTPRouteGroup
metadata:
  name: http-route
spec:
  matches:
  - name: version
    pathRegex: "/version"
    methods:
    - GET
---
kind: TrafficTarget
metadata:
  name: test
  namespace: default
spec:
  destination:
    kind: ServiceAccount
    name: sa-1 # There are 2 services under this service account:  service-1 and service-2
    namespace: default
  rules:
  - kind: TCPRoute
    name: tcp-route
  - kind: HTTPRouteGroup
    name: http-route
  sources:
  - kind: ServiceAccount
    name: sa-2
    namespace: default

Kubernetes service resources should explicitly specify the application protocol being served by the service’s ports using the appProtocol field.

A service service-1 backed by a pod in service account sa-1 serving http application traffic should be defined as follows:

kind: Service
metadata:
  name: service-1
  namespace: default
spec:
  ports:
  - port: 8080
    name: some-port
    appProtocol: http

A service service-2 backed by a pod in service account sa-1 serving raw tcp application traffic shold be defined as follows:

kind: Service
metadata:
  name: service-2
  namespace: default
spec:
  ports:
  - port: 8080
    name: some-port
    appProtocol: tcp