Logs

FSM control plane components log diagnostic messages to stdout to aid in managing a mesh.

In the logs, users can expect to see the following kinds of information alongside messages:

• Kubernetes resource metadata, like names and namespaces
• mTLS certificate common names

FSM will not log sensitive information, such as:

• Kubernetes Secret data
• entire Kubernetes resources

Verbosity

Log verbosity controls when certain log messages are written, for example to include more messages for debugging or to include fewer messages that only point to critical errors.

FSM defines the following log levels in order of increasing verbosity:

Each of the above log levels can be configured in the MeshConfig at spec.observability.fsmLogLevel or on install with the fsm.controllerLogLevel chart value.

Fluent Bit

When enabled, Fluent Bit can collect these logs, process them and send them to an output of the user’s choice such as Elasticsearch, Azure Log Analytics, BigQuery, etc.

Fluent Bit is an open source log processor and forwarder which allows you to collect data/logs and send them to multiple destinations. It can be used with FSM to forward FSM controller logs to a variety of outputs/log consumers by using its output plugins.

FSM provides log forwarding by optionally deploying a Fluent Bit sidecar to the FSM controller using the --set=fsm.enableFluentbit=true flag during installation. The user can then define where FSM logs should be forwarded using any of the available Fluent Bit output plugins.

Configuring Log Forwarding with Fluent Bit

By default, the Fluent Bit sidecar is configured to simply send logs to the Fluent Bit container’s stdout. If you have installed FSM with Fluent Bit enabled, you may access these logs using kubectl logs -n <fsm-namespace> <fsm-controller-name> -c fluentbit-logger. This command will also help you find how your logs are formatted in case you need to change your parsers and filters.

Note: <fsm-namespace> refers to the namespace where the fsm control plane is installed.

To quickly bring up Fluent Bit with default values, use the --set=fsm.enableFluentbit option:

fsm install --set=fsm.enableFluentbit=true

By default, logs will be filtered to emit info level logs. You may change the log level to “debug”, “warn”, “fatal”, “panic”, “disabled” or “trace” during installation using --set fsm.controllerLogLevel=<desired log level> . To get all logs, set the log level to trace.

Once you have tried out this basic setup, we recommend configuring log forwarding to your preferred output for more informative results.

To customize log forwarding to your output, follow these steps and then reinstall FSM with Fluent Bit enabled.

1. Find the output plugin you would like to forward your logs to in Fluent Bit documentation. Replace the [OUTPUT] section in fluentbit-configmap.yaml with appropriate values.

2. The default configuration uses CRI log format parsing. If you are using a kubernetes distribution that causes your logs to be formatted differently, you may need to add a new parser to the [PARSER] section and change the parser name in the [INPUT] section to one of the parsers defined here.

3. Explore available Fluent Bit Filters and add as many [FILTER] sections as desired.

   • The [INPUT] section tags ingested logs with kube.* so make sure to include Match kube.* key/value pair in each of your custom filters.
   • The default configuration uses a modify filter to add a controller_pod_name key/value pair to help you query logs in your output by refining results on pod name (see example usage below).

4. For these changes to take effect, run:

   make build-fsm
   

5. Once you have updated the Fluent Bit ConfigMap template, you can deploy Fluent Bit during FSM installation using:

   fsm install --set=fsm.enableFluentbit=true [--set fsm.controllerLogLevel=<desired log level>]
   

   You should now be able to interact with error logs in the output of your choice as they get generated.

Example: Using Fluent Bit to send logs to Azure Monitor

Fluent Bit has an Azure output plugin that can be used to send logs to an Azure Log Analytics workspace as follows:

1. Create a Log Analytics workspace

2. Navigate to your new workspace in Azure Portal. Find your Workspace ID and Primary key in your workspace under Agents management. In values.yaml, under fluentBit, update the outputPlugin to azure and keys workspaceId and primaryKey with the corresponding values from Azure Portal (without quotes). Alternatively, you may replace entire output section in fluentbit-configmap.yaml as you would for any other output plugin.

3. Run through steps 2-5 above.

4. Once you run FSM with Fluent Bit enabled, logs will populate under the Logs > Custom Logs section in your Log Analytics workspace. There, you may run the following query to view most recent logs first:

   fluentbit_CL
   | order by TimeGenerated desc
   

5. Refine your log results on a specific deployment of the FSM controller pod:

   | where controller_pod_name_s == "<desired fsm controller pod name>"
   

Once logs have been sent to Log Analytics, they can also be consumed by Application Insights as follows:

1. Create a Workspace-based Application Insights instance.

2. Navigate to your instance in Azure Portal. Go to the Logs section. Run this query to ensure that logs are being picked up from Log Analytics:

   workspace("<your-log-analytics-workspace-name>").fluentbit_CL
   

You can now interact with your logs in either of these instances.

Note: Fluent Bit is not currently supported on OpenShift.

Configuring Outbound Proxy Support for Fluent Bit

You may require outbound proxy support if your egress traffic is configured to go through a proxy server. There are two ways to enable this.

If you have already built FSM with the MeshConfig changes above, you can simply enable proxy support using the FSM CLI, replacing your values in the command below:

fsm install --set=fsm.enableFluentbit=true,fsm.fluentBit.enableProxySupport=true,fsm.fluentBit.httpProxy=<http-proxy-host:port>,fsm.fluentBit.httpsProxy=<https-proxy-host:port>

Alternatively, you may change the values in the Helm chart by updating the following in values.yaml:

1. Change enableProxySupport to true

2. Update the httpProxy and httpsProxy values to "http://<host>:<port>". If your proxy server requires basic authentication, you may include its username and password as: http://<username>:<password>@<host>:<port>

3. For these changes to take effect, run:

   make build-fsm
   

4. Install FSM with Fluent Bit enabled:

   fsm install --set=fsm.enableFluentbit=true
   

NOTE: Ensure that the Fluent Bit image tag is 1.6.4 or greater as it is required for this feature.