This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Getting Started

The following pages demonstrate FSM’s basic features with a sample microservice topology, from installation to configuring traffic policies to cleanup.

1 - Setup FSM

Install the FSM control plane using the FSM CLI

Prerequisites

This demo of FSM v1.2.3 requires:

Note: This document assumes you have already installed credentials for a Kubernetes cluster in ~/.kube/config and kubectl cluster-info executes successfully.

Download and install the FSM command-line tool

The fsm command-line tool contains everything needed to install and configure Open Service Mesh. The binary is available on the FSM GitHub releases page.

GNU/Linux

Download the 64-bit GNU/Linux or macOS binary of FSM v1.2.3:

system=$(uname -s | tr '[:upper:]' '[:lower:]')
arch=$(uname -m | sed -E 's/x86_/amd/' | sed -E 's/aarch/arm/')
release=v1.2.3
curl -L https://github.com/flomesh-io/fsm/releases/download/${release}/fsm-${release}-${system}-${arch}.tar.gz | tar -vxzf -
./${system}-amd64/fsm version

macOS

Download the 64-bit macOS binaries for FSM v1.2.3

system=$(uname -s | tr "[:upper:]" "[:lower:]")
arch=$(uname -m)
release=v1.2.3
curl -L https://github.com/flomesh-io/fsm/releases/download/$release/fsm-$release-$system-$arch.tar.gz | tar -vxzf -
./$system-$arch/fsm version

The fsm CLI can be compiled from source according to this guide.

Installing FSM on Kubernetes

With the fsm binary downloaded, unzipped, and placed into $PATH, we are ready to install Open Service Mesh on a Kubernetes cluster:

The command below shows how to install FSM on your Kubernetes cluster. This command enables Prometheus, Grafana and Jaeger integrations. The fsm.enablePermissiveTrafficPolicy chart parameter in the values.yaml file instructs FSM to ignore any policies and let traffic flow freely between the pods. With Permissive Traffic Policy mode enabled, new pods will be injected with Pipy, but traffic will flow through the proxy and will not be blocked by access control policies.

Note: Permissive Traffic Policy mode is an important feature for brownfield deployments, where it may take some time to craft SMI policies. While operators design the SMI policies, existing services will continue to operate as they have been before FSM was installed.

export fsm_namespace=fsm-system # Replace fsm-system with the namespace where FSM will be installed
export fsm_mesh_name=fsm # Replace fsm with the desired FSM mesh name

fsm install \
    --mesh-name "$fsm_mesh_name" \
    --fsm-namespace "$fsm_namespace" \
    --set=fsm.enablePermissiveTrafficPolicy=true \
    --set=fsm.deployPrometheus=true \
    --set=fsm.deployGrafana=true \
    --set=fsm.deployJaeger=true

Read more on FSM’s integrations with Prometheus, Grafana, and Jaeger in the observability documentation.

Next Steps

Now that the FSM control plane is up and running, add applications to the mesh.

2 - Deploy Sample Applications

Deploy the sample bookstore applications

In this section we will deploy 5 different Pods, and we will apply policies to control the traffic between them.

  • bookbuyer is an HTTP client making requests to bookstore. This traffic is permitted.
  • bookthief is an HTTP client and much like bookbuyer also makes HTTP requests to bookstore. This traffic should be blocked.
  • bookstore is a server, which responds to HTTP requests. It is also a client making requests to the bookwarehouse service. This traffic is permitted.
  • bookwarehouse is a server and should respond only to bookstore. Both bookbuyer and bookthief should be blocked.
  • mysql is a MySQL database only reachable by bookwarehouse.

We are going to define and deploy traffic access policies using SMI, which will bring us to this final desired

state of allowed and blocked traffic between pods:

from / to:bookbuyerbookthiefbookstorebookwarehousemysql
bookbuyern/a❌✔❌❌
bookthief❌n/a❌❌❌
bookstore❌❌n/a✔❌
bookwarehouse❌❌❌n/a✔
mysql❌❌❌❌n/a

To show how to split traffic using SMI Traffic Split, we will deploy an additional application:

  • bookstore-v2 - this is the same container as the first bookstore we deployed, but for this demo we will assume that it is a new version of the app we need to upgrade to.

The bookbuyer, bookthief, bookstore, and bookwarehouse Pods will be in separate Kubernetes Namespaces with

the same names. mysql will be in the bookwarehouse namespace. Each new Pod in the service mesh will be injected with an Pipy sidecar container.

Create the Namespaces

kubectl create namespace bookstore
kubectl create namespace bookbuyer
kubectl create namespace bookthief
kubectl create namespace bookwarehouse

Add the new namespaces to the FSM control plane

fsm namespace add bookstore bookbuyer bookthief bookwarehouse

Now each one of the four namespaces is labelled with flomesh.io/monitored-by: fsm and also

annotated with flomesh.io/sidecar-injection: enabled. The FSM Controller, noticing the label and annotation

on these namespaces, will start injecting all new pods with Pipy sidecars.

Create Pods, Services, ServiceAccounts

Create the bookbuyer service account and deployment:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookbuyer.yaml

Create the bookthief service account and deployment:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookthief.yaml

Create the bookstore service account, service, and deployment:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookstore.yaml

Create the bookwarehouse service account, service, and deployment:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookwarehouse.yaml

Create the mysql service account, service, and stateful set:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/mysql.yaml

Checkpoint: What Got Installed?

A Kubernetes Deployment and Pods for each of bookbuyer, bookthief, bookstore and bookwarehouse, and a StatefulSet for mysql. Also, Kubernetes Services and Endpoints for bookstore, bookwarehouse, and mysql.

To view these resources on your cluster, run the following commands:

kubectl get pods,deployments,serviceaccounts -n bookbuyer
kubectl get pods,deployments,serviceaccounts -n bookthief

kubectl get pods,deployments,serviceaccounts,services,endpoints -n bookstore
kubectl get pods,deployments,serviceaccounts,services,endpoints -n bookwarehouse

In addition, a Kubernetes Service Account was also created for each application. The Service Account serves as the application’s identity which will be used later in the demo to create service-to-service access control policies.

View the Application UIs

Set up client port forwarding with the following steps to access the applications in the Kubernetes cluster. It is best to start a new terminal session for running the port forwarding scripts to maintain the port forwarding session, while using the original terminal to continue to issue commands. The port-forward-all.sh scripts will look for a .env file for environment variables needed to run the script. The .env creates the necessary variables that target the previously created namespaces. We will use the reference .env.example file and then run the port forwarding scripts.

In a new terminal session, run the following commands to enable port forwarding into the Kubernetes cluster from the root of the project directory (your local clone of upstream FSM).


git clone https://github.com/flomesh-io/fsm.git -b main
cd fsm
cp .env.example .env
bash <<EOF
./scripts/port-forward-bookbuyer-ui.sh &
./scripts/port-forward-bookstore-ui.sh app=bookstore &
./scripts/port-forward-bookthief-ui.sh &
wait
EOF

Note: To override the default ports, prefix the BOOKBUYER_LOCAL_PORT, BOOKSTORE_LOCAL_PORT, and/or BOOKTHIEF_LOCAL_PORT variable assignments to the port-forward scripts. For example:

export BOOKBUYER_LOCAL_PORT=7070 BOOKTHIEF_LOCAL_PORT=7073 BOOKSTORE_LOCAL_PORT=7074
bash <<EOF
./scripts/port-forward-bookbuyer-ui.sh &
./scripts/port-forward-bookstore-ui.sh app=bookstore &
./scripts/port-forward-bookthief-ui.sh &
wait
EOF

In a browser, open up the following urls:

Position the windows so that you can see all of them at the same time. The header at the top of the webpage indicates the application and version.

Next Steps

Now that the sample applications are running, configure traffic policies between the applications.

3 - Configure Traffic Policies

Configure traffic to flow between the bookstore applications

Traffic Policy Modes

Once the applications are up and running, they can interact with each other using permissive traffic policy mode or SMI traffic policy mode. In permissive traffic policy mode, traffic between application services is automatically configured by fsm-controller, and access control policies defined by SMI Traffic Targets are not enforced. In the SMI policy mode, all traffic is denied by default unless explicitly allowed using a combination of SMI access and routing policies.

Traffic Encryption

All traffic is encrypted via mTLS regardless of whether you’re using access control policies or have enabled permissive traffic policy mode.

How to Check Traffic Policy Mode

Check whether permissive traffic policy mode is enabled or not by retrieving the value for the enablePermissiveTrafficPolicyMode key in the fsm-mesh-config MeshConfig resource.

# Replace fsm-system with fsm-controller's namespace if using a non default namespace
kubectl get meshconfig fsm-mesh-config -n fsm-system -o jsonpath='{.spec.traffic.enablePermissiveTrafficPolicyMode}{"\n"}'
# Output:
# false: permissive traffic policy mode is disabled, SMI policy mode is enabled
# true: permissive traffic policy mode is enabled, SMI policy mode is disabled

The following sections demonstrate using FSM with permissive traffic policy mode and SMI Traffic Policy Mode.

Permissive Traffic Policy Mode

In permissive traffic policy mode, application connectivity within the mesh is automatically configured by fsm-controller. It can be enabled in the following ways.

  1. During install using fsm CLI:
fsm install --set=fsm.enablePermissiveTrafficPolicy=true
  1. Post install by patching the fsm-mesh-config custom resource in the control plane’s namespace (fsm-system by default)
kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}'  --type=merge

Verify FSM is in permissive traffic policy mode

Before proceeding, verify the traffic policy mode and ensure the enablePermissiveTrafficPolicyMode key is set to true in the fsm-mesh-config MeshConfig resource. Refer to the section above to enable permissive traffic policy mode.

In step Deploy the Bookstore Application, we have already deployed the applications needed to verify traffic flow in permissive traffic policy mode. The bookstore service we previously deployed is encoded with an identity of bookstore-v1 for demo purpose, as can be seen in the Deployment’s manifest. The identity reflects which counter increments in the bookbuyer and bookthief UI, and the identity displayed in the bookstore UI.

The counter in the bookbuyer, bookthief UI for the books bought and stolen respectively from bookstore v1 should now be incrementing:

The counter in the bookstore UI for the books sold should also be incrementing:

The bookbuyer and bookthief applications are able to buy and steal books respectively from the newly deployed bookstore application because permissive traffic policy mode is enabled, thereby allowing connectivity between applications without the need for SMI traffic access policies.

This can be demonstrated further by disabling permissive traffic policy mode and verifying that the counter for books bought from bookstore is not incrementing anymore:

kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}'  --type=merge

Note: When you disable permissive traffic policy mode, SMI traffic access mode is implicitly enabled. If counters for the books are incrementing then it could be because some SMI Traffic Access policies have been applied previously to allow such traffic.

SMI Traffic Policy Mode

SMI traffic policies can be used for the following:

  1. SMI access control policies to authorize traffic access between service identities
  2. SMI traffic specs policies to define routing rules to associate with access control policies
  3. SMI traffic split policies to direct client traffic to multiple backends based on weights

The following sections describe how to leverage each of these policies to enforce fine grained control over traffic flowing within the service mesh. Before proceeding, verify the traffic policy mode and ensure the enablePermissiveTrafficPolicyMode key is set to false in the fsm-mesh-config MeshConfig resource.

SMI traffic policy mode can be enabled by disabling permissive traffic policy mode:

kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}'  --type=merge

Deploy SMI Access Control Policies

At this point, applications do not have access to each other because no access control policies have been applied. Confirm this by verifying that none of the counters in the bookbuyer, bookthief, bookstore, and bookstore-v2 UI are incrementing.

Apply the SMI Traffic Target and SMI Traffic Specs resources to define access control and routing policies for the applications to communicate:

Deploy SMI TrafficTarget and HTTPRouteGroup policy:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/access/traffic-access-v1.yaml

The counters should now be incrementing for the bookbuyer, and bookstore applications:

Note that the counter is not incrementing for the bookthief application:

That is because the SMI Traffic Target SMI HTTPRouteGroup resources deployed only allow bookbuyer to communicate with bookstore.

Allowing the Bookthief Application to access the Mesh

Currently the Bookthief application has not been authorized to participate in the service mesh communication. We will now update the TrafficTarget to allow bookthief to communicate with bookstore.

Current TrafficTarget spec without bookthief listed in spec.sources:

kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
  name: bookstore-v1
  namespace: bookstore
spec:
  destination:
    kind: ServiceAccount
    name: bookstore
    namespace: bookstore
  rules:
  - kind: HTTPRouteGroup
    name: bookstore-service-routes
    matches:
    - buy-a-book
    - books-bought
  sources:
  - kind: ServiceAccount
    name: bookbuyer
    namespace: bookbuyer

Updated TrafficTarget spec with bookthief in spec.sources:

kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
 name: bookstore-v1
 namespace: bookstore
spec:
 destination:
   kind: ServiceAccount
   name: bookstore
   namespace: bookstore
 rules:
 - kind: HTTPRouteGroup
   name: bookstore-service-routes
   matches:
   - buy-a-book
   - books-bought
 sources:
 - kind: ServiceAccount
   name: bookbuyer
   namespace: bookbuyer
 - kind: ServiceAccount
   name: bookthief
   namespace: bookthief

Apply the updated TrafficTarget:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/access/traffic-access-v1-allow-bookthief.yaml

The counter in the bookthief window will start incrementing.

Apply the original Traffic Target object without the bookthief listed as an allowed source:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/access/traffic-access-v1.yaml

The counter in the bookthief window will stop incrementing.

Next Steps

Learn how to balance traffic between services by configuring traffic split.

4 - Configure Traffic Split

Balance traffic between services with the SMI Traffic Split API

We will now demonstrate how to balance traffic between two Kubernetes services, commonly known as a traffic split. We will be splitting the traffic directed to the root bookstore service between the backends bookstore-v1 service and bookstore-v2 service. The bookstore-v1 and bookstore-v2 services are also known as leaf services. Learn more on how to configure services for Traffic Splitting in the traffic Splitting how-to guide

Deploy bookstore v2 application

To demonstrate usage of SMI traffic access and split policies, we will now deploy version v2 of the bookstore application (bookstore-v2) - remember that if you are using openshift, you must add the security context constraint to the bookstore-v2 service account.

# Contains the bookstore-v2 Kubernetes Service, Service Account, Deployment and SMI Traffic Target resource to allow
# `bookbuyer` to communicate with `bookstore-v2` pods
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookstore-v2.yaml

Wait for the bookstore-v2 pod to be running in the bookstore namespace. Next, exit and restart the port-forward scripts in order to access v2 of bookstore:

bash <<EOF
./scripts/port-forward-bookbuyer-ui.sh &
./scripts/port-forward-bookstore-ui.sh app=bookstore,version=v1 &
./scripts/port-forward-bookstore-ui-v2.sh &
./scripts/port-forward-bookthief-ui.sh &
wait
EOF

The bookstore-v2 counter should be incrementing because traffic to the root bookstore service is configured with a label selector which selects endpoints that include both the bookstore-v1 and bookstore-v2 backend pods.

Create SMI Traffic Split

Deploy the SMI traffic split policy to direct 100 percent of the traffic sent to the root bookstore service to the bookstore-v1 service backend. This is necessary to ensure traffic directed to the bookstore service is only directed to version v1 of the bookstore app, which includes the pods backing the bookstore-v1 service. The TrafficSplit configuration will be subsequently updated to direct a percentage of traffic to version v2 of the bookstore service using the bookstore-v2 leaf service.

For this reason, it is important to ensure client applications always communicate with the root service if a traffic split is desired. Otherwise the client application will need to be updated to communicate with the root service when a traffic split is desired.

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/split/traffic-split-v1.yaml

Note: The root service is a Kubernetes service whose selector needs to match the pods backing the leaf services. In this demo, the root service bookstore has the selector app:bookstore, which matches both the labels app:bookstore,version:v1 and app:bookstore,version=v2 on the bookstore (v1) and bookstore-v2 deployments respectively. The root service can be referred to in the SMI Traffic Split resource as the name of the service with or without the .<namespace>.svc.cluster.local suffix.

The count for the books sold from the bookstore-v2 browser window should stop incrementing. This is because the current traffic split policy is weighted 100 for bookstore-v1 which exludes pods backing the bookstore-v2 service. You can verify the traffic split policy by running the following and viewing the Backends properties:

kubectl describe trafficsplit bookstore-split -n bookstore

Split Traffic to Bookstore v2

Update the SMI Traffic Split policy to direct 50 percent of the traffic sent to the root bookstore service to the bookstore service and 50 perfect to bookstore-v2 service by adding the bookstore-v2 backend to the spec and modifying the weight fields.

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/split/traffic-split-50-50.yaml

Wait for the changes to propagate and observe the counters increment for bookstore and bookstore-v2 in your browser windows. Both

counters should be incrementing:

Split All Traffic to Bookstore v2

Update the bookstore-split TrafficSplit to configure all traffic to go to bookstore-v2:

kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/split/traffic-split-v2.yaml

Wait for the changes to propagate and observe the counters increment for bookstore-v2 and freeze for bookstore in your

browser windows:

Now, all traffic directed to the bookstore service is flowing to bookstore-v2.

Next Steps

5 - Configure Observability with Prometheus and Grafana

Use FSM’s observability integrations with Prometheus and Grafana to inspect the traffic between the bookstore applications

The following article shows you how to install FSM with automatic provisioning of the Prometheus and Grafana stack for observability and monitoring. For an example using a bring your own (BYO) Prometheus and Grafana stack on your cluster with FSM, see the Integrate FSM with Prometheus and Grafana demo.

The configuration created in this article should not be used in production environments. For production-grade deployments, see Prometheus Operator and Deploy Grafana in Kubernetes.

Install FSM with Prometheus and Grafana

On fsm install, a Prometheus and/or Grafana instance can be automatically provisioned with the default FSM configuration.

 fsm install --set=fsm.deployPrometheus=true \
             --set=fsm.deployGrafana=true

More information on observability can be found in the Observability Guide.

Prometheus

When configured with the --set=fsm.deployPrometheus=true flag, FSM installation will deploy a Prometheus instance to scrape the sidecar and FSM control plane’s metrics endpoints. The scraping configuration file defines the default Prometheus behavior and the set of metrics collected by FSM.

Grafana

FSM can be configured to deploy a Grafana instance using the --set=fsm.deployGrafana=true flag in fsm install. FSM provides pre-configured dashboards that are documented in the FSM Grafana dashboards section of the Observability Guide.

Enable Metrics Scraping

Metrics can be enabled at the namespace scope using the fsm metrics command. By default, FSM does not configure metrics scraping for pods in the mesh.

fsm metrics enable --namespace test
fsm metrics enable --namespace "test1, test2"

Note: The namespace that you are enabling for metrics scraping must already be a part of the mesh.

Inspect Dashboards

The FSM Grafana dashboards can be viewed with the following command:

fsm dashboard

Navigate to http://localhost:3000 to access the Grafana dashboards. The default user name is admin and the default password is admin. On the Grafana homepage click on the Home icon, you will see a folder containing dashboards for both FSM Control Plane and FSM Data Plane.

Next Steps

Cleanup sample applications and uninstall FSM.

6 - Uninstall FSM

Uninstall FSM and the bookstore applications

The articles in the getting started section outline installing FSM and sample applications. To uninstall all of these resources, delete the sample application and related SMI resources and uninstall the FSM control plane and cluster-wide FSM resources.

Delete the sample applications

To clean up the sample applications and related SMI resources, delete their namespaces. For example:

kubectl delete ns bookbuyer bookthief bookstore bookwarehouse

Uninstall FSM control plane

To uninstall FSM control plane, use fsm unistall mesh.

fsm uninstall mesh

Uninstall FSM cluster-wide resources

To uninstall FSM cluster-wide resources, use fsm uninstall mesh --delete-cluster-wide-resources.

fsm uninstall mesh --delete-cluster-wide-resources

It’s recommended to delete all resources with command fsm uninstall mesh -a -f --delete-namespace during learning.

For more details about uninstalling FSM, see the uninstallation guide.