Egress Policy

Accessing external services using Egress policies

This guide demonstrates a client within the service mesh accessing destinations external to the mesh using FSM’s Egress policy API.

Prerequisites

Kubernetes cluster version v1.19.0 or higher.
Interact with the API server using kubectl.
FSM CLI installed.
FSM Ingress Controller installed followed by installation document

Demo

Enable egress policy if not enabled, at the same time we should confirm egress passthrough is disabled:

# Replace fsm-system with the namespace where FSM is installed
kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"featureFlags":{"enableEgressPolicy":true},"traffic":{"enableEgress":false}}}' --type=merge

Deploy the curl client into the curl namespace after enrolling its namespace to the mesh.

# Create the curl namespace
kubectl create namespace curl

# Add the namespace to the mesh
fsm namespace add curl

# Deploy curl client in the curl namespace
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/samples/curl/curl.yaml -n curl

Confirm the curl client pod is up and running.

kubectl get pods -n curl
NAME                    READY   STATUS    RESTARTS   AGE
curl-54ccc6954c-9rlvp   2/2     Running   0          20s

HTTP Egress

Confirm the curl client is unable make the HTTP request http://httpbin.org:80/get to the httpbin.org website on port 80.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/get
command terminated with exit code 7

Apply an Egress policy to allow the curl client’s ServiceAccount to access the httpbin.org website on port 80 serving the http protocol.

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin-80
  namespace: curl
spec:
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
  hosts:
  - httpbin.org
  ports:
  - number: 80
    protocol: http
EOF

Confirm the curl client is able to make successful HTTP requests to http://httpbin.org:80/get.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/get
HTTP/1.1 200 OK
date: Mon, 04 Jul 2022 07:48:24 GMT
content-type: application/json
content-length: 313
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

Confirm the curl client can no longer make successful HTTP requests to http://httpbin.org:80/get when the above policy is removed.

kubectl delete egress httpbin-80 -n curl

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/get
command terminated with exit code 7

HTTPS Egress

Since HTTPS traffic is encrypted with TLS, FSM routes HTTPS based traffic by proxying it to its original destination as a TCP stream. The Server Name Indication (SNI) indicated by the HTTPS client application in the TLS handshake is matched against hosts specified in the Egress policy.

Confirm the curl client is unable make the HTTPS request https://httpbin.org:443/get to the httpbin.org website on port 443.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://httpbin.org:443/get
command terminated with exit code 7

Apply an Egress policy to allow the curl client’s ServiceAccount to access the httpbin.org website on port 443 serving the https protocol.

kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin-443
  namespace: curl
spec:
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
  hosts:
  - httpbin.org
  ports:
  - number: 443
    protocol: https
EOF

Confirm the curl client is able to make successful HTTPS requests to https://httpbin.org:443/get.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://httpbin.org:443/get
HTTP/2 200
date: Thu, 13 May 2021 22:09:36 GMT
content-type: application/json
content-length: 260
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

Confirm the curl client can no longer make successful HTTPS requests to https://httpbin.org:443/get when the above policy is removed.

kubectl delete egress httpbin-443 -n curl

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://httpbin.org:443/get
command terminated with exit code 7

TCP Egress

TCP based Egress traffic is matched against the destination port and IP address ranges specified in an egress policy. If an IP address range is not specified, traffic will be matched only based on the destination port.

Confirm the curl client is unable make the HTTPS request https://flomesh.io:443 to the flomesh.io website on port 443. Since HTTPS uses TCP as the underlying transport protocol, TCP based routing should implicitly enable access to any HTTP(s) host on the specified port.
```
kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://flomesh.io:443
command terminated with exit code 7
```
Apply an Egress policy to allow the curl client’s ServiceAccount to access the any destination on port 443 serving the tcp protocol.
```
kubectl apply -f - <<EOF
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: tcp-443
  namespace: curl
spec:
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
  ports:
  - number: 443
    protocol: tcp
EOF
```
Note: For server-first protocols such as MySQL, PostgreSQL, etc., where the server initiates the first bytes of data between the client and server, the protocol must be set to tcp-server-first to indicate to FSM to not perform protocol detection on the port. Protocol detection relies on inspecting the initial bytes of a connection, which is incompatible with server-first protocols. When the port’s protocol is set to tcp-server-first, protocol detection is skipped for that port number. It is also important to note that server-first port numbers must not be used for other application ports that require protocol detection to performed, which means the port numbers used for server-first protocols must not be used with other protocols such as HTTP and TCP that require protocol detection to be performed.

Confirm the curl client is able to make successful HTTPS requests to https://flomesh.io:443.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://flomesh.io:443
HTTP/2 200
content-type: text/html

Confirm the curl client can no longer make successful HTTPS requests to https://flomesh.io:443 when the above policy is removed.

kubectl delete egress tcp-443 -n curl

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI https://flomesh.io:443
command terminated with exit code 7

HTTP Egress with SMI route matches

HTTP Egress policies can specify SMI HTTPRouteGroup matches for fine grained traffic control based on HTTP methods, headers and paths.

Confirm the curl client is unable make HTTP requests to http://httpbin.org:80/get and http://httpbin.org:80/status/200 to the httpbin.org website on port 80.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/get
command terminated with exit code 7

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/status/200
command terminated with exit code 7

Apply an SMI HTTPRouteGroup resource to allow access to the HTTP path /get and an Egress policy to access the httpbin.org on website port 80 that matches on the SMI HTTPRouteGroup.

kubectl apply -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: egress-http-route
  namespace: curl
spec:
  matches:
  - name: get
    pathRegex: /get
---
kind: Egress
apiVersion: policy.flomesh.io/v1alpha1
metadata:
  name: httpbin-80
  namespace: curl
spec:
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
  hosts:
  - httpbin.org
  ports:
  - number: 80
    protocol: http
  matches:
  - apiGroup: specs.smi-spec.io/v1alpha4
    kind: HTTPRouteGroup
    name: egress-http-route
EOF

Confirm the curl client is able to make successful HTTP requests to http://httpbin.org:80/get.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/get
HTTP/1.1 200 OK
date: Thu, 13 May 2021 21:49:35 GMT
content-type: application/json
content-length: 335
access-control-allow-origin: *
access-control-allow-credentials: true

Confirm the curl client is unable to make successful HTTP requests to http://httpbin.org:80/status/200.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/status/200
HTTP/1.1 403 Forbidden
content-length: 13
connection: keep-alive

Update the matching SMI HTTPRouteGroup resource to allow requests to HTTP paths matching the regex /status.*.

kubectl apply -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: egress-http-route
  namespace: curl
spec:
  matches:
  - name: get
    pathRegex: /get
  - name: status
    pathRegex: /status.*
EOF

Confirm the curl client can now make successful HTTP requests to http://httpbin.org:80/status/200.

kubectl exec $(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}') -n curl -c curl -- curl -sI http://httpbin.org:80/status/200
HTTP/1.1 200 OK
date: Fri, 14 May 2021 17:10:48 GMT
content-type: text/html; charset=utf-8
content-length: 0
access-control-allow-origin: *
access-control-allow-credentials: true

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified June 18, 2024: Update API version in gateway manifests to align with gwapi 1.1 Signed-off-by: Addo.Zhang <duwasai@gmail.com> (c83f664)