openshift logs and monitoring for Kubernetes

Log collection and monitoring are essential for troubleshooting and optimizing Kubernetes applications running on OpenShift. OpenShift provides several tools for collecting and monitoring logs, including the OpenShift logging subsystem.

The OpenShift logging subsystem collects logs from various sources, including pods, nodes, and services. It stores these logs in a centralized location, such as Elasticsearch.

To monitor logs in OpenShift, you can use the OpenShift web console or the `oc` command-line tool. The web console provides a graphical interface for viewing logs, while the `oc` tool allows you to query and filter logs programmatically.

Monitoring logs in OpenShift can help you identify issues and optimize performance, but it requires careful configuration and management to ensure accuracy and efficiency.

Log collection is a crucial part of OpenShift logging, and it's set up similarly to other Kubernetes clusters. The Datadog Agent's log collection is mounted in the /var/log/pods directory, where the Agent pod can monitor logs from pods and containers on its host.

The Datadog Operator requires additional SELinux options to give the Agent permissions to read log files. This is a key difference from other Kubernetes clusters.

The OpenShift Container Platform Event Router is a pod that watches Kubernetes events and logs them for collection by OpenShift Container Platform Logging. You must manually deploy the Event Router.

The logging collector uses journald for system logs and /var/log/containers/*.log for container logs by default. It's a daemon set that deploys pods to each OpenShift Container Platform node, collecting logs from various sources and forwarding them as configured.

Elasticsearch is used by default to store log data, but you can use the Log Forwarder API to forward logs to an external store. You can also configure Elasticsearch to make copies of shards, called replicas, for data redundancy and resilience to failure.

Log collection is a crucial step in monitoring and analyzing system logs. The Datadog Agent's log collection is set up in OpenShift largely the same as other Kubernetes clusters.

The Datadog Agent pod uses the /var/log/pods directory to monitor the logs of the pods and containers on its respective host. This is mounted by the Datadog Operator and Helm Chart.

To give the Agent permissions to read these log files, you need to apply additional SELinux options. This is a specific requirement when using the Datadog Operator.

For more general information on Kubernetes log collection, you can refer to the Kubernetes Log Collection page. For configuration examples, you can check the Distributions page.

Here is a list of the log sources that the Datadog Agent can collect:

The OpenShift Container Platform Event Router is a pod that watches Kubernetes events and logs them for collection by OpenShift Container Platform Logging. You must manually deploy the Event Router.

The Event Router collects events from all projects and writes them to STDOUT. Fluentd collects those events and forwards them into the OpenShift Container Platform Elasticsearch instance.

Elasticsearch indexes the events to the infra index. The logging subsystem Elasticsearch instance is optimized and tested for short term storage, approximately seven days.

The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes. A highly-available Elasticsearch environment requires at least three Elasticsearch nodes, each on a different host.

You can use a ClusterLogging custom resource (CR) to increase the number of Elasticsearch nodes, as needed. Role-based access control (RBAC) applied on the Elasticsearch indices enables the controlled access of the logs to the developers.

Here's a summary of the supported log sources for Fluentd and Vector:

The Event Router is a pod that watches OpenShift Container Platform events and collects them for the logging subsystem.

By default, the logging subsystem sends logs to the default internal Elasticsearch log store, defined in the ClusterLogging custom resource.

You can use the log forwarding features to send logs to specific endpoints within or outside your cluster.

Event routing is a crucial step in log forwarding and routing, and it's handled by the Event Router pod in OpenShift Container Platform. This pod watches events so they can be collected by the logging subsystem.

The Event Router collects events from all projects, making it a comprehensive solution for event logging. It writes these events to STDOUT.

Fluentd then collects these events and forwards them into the OpenShift Container Platform Elasticsearch instance. Elasticsearch indexes the events to the infra index, making them easily searchable.

You must manually deploy the Event Router to start collecting and storing Kubernetes events.

Output to Streams is a crucial part of log forwarding and routing. In OpenShift Container Platform, you can output messages to streams, specifically stdout and stderr.

To output a message to stdout, you can use the CLI command `oc logs` followed by the name of your frontend pod. For example, if your pod name is `ostoy-frontend-679cb85695-5cn7x`, you would run `oc logs ostoy-frontend-679cb85695-5cn7x` and see your messages.

You can also output a message to stdout from the UI. Click on the Home menu item and then click in the message box for "Log Message (stdout)" and write any message you want to output to the stdout stream. Try "All is well!" and click "Send Message".

To output a message to stderr, you can use the same `oc logs` command as before, but this time specify the stderr stream. For example, `oc logs ostoy-frontend-679cb85695-5cn7x` will show you both stdout and stderr messages.

Alternatively, from the UI, click in the message box for "Log Message (stderr)" and write any message you want to output to the stderr stream. Try "Oh no! Error!" and click "Send Message".

You can see both stdout and stderr messages in the output. The Event Router collects events from all projects and writes them to STDOUT, which is then collected by Fluentd and forwarded into the OpenShift Container Platform Elasticsearch instance.