Logging

Logging is important for observability of the platform. Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform. Log levels can be set for individual modules and configuration is identical in each Stacklet, but custom logging configuration files can be supplied as well.

Motivation

Aggregate and persist - The logging architecture is designed to aggregate logs from all parts of the platform to simplify correlating events from different Stacklets. For this, logs from different sources are configured to share the same structure, and are collected and made viewable in a central location. The collected logs are also be persisted centrally, so if a component crashes its logs are still there to investigate the cause of the crash.

Easy to read on the fly - At the same time logs are kept accessible in an plain format on the containers, supporting easy on the fly inspection of running Stacklets. The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs. This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.

Consistent configuration - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs. Logging for each product is configured in the Stacklet resource. For advanced log configurations, supplying custom product specific log configuration files is also supported.

Architecture

Below you can see the overall architecture using ZooKeeper as an example. Stackable uses Vector for log aggregation and any of the supported sinks can be used to persist the logs.

Log source

You configure your logging settings in the Stacklet definition (ZookeeperCluster in this example), seen in the top left in the diagram (see the configuration section below). The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration. The ConfigMap is then mounted into the containers. The ZooKeeper Pod has three containers: The prepare sidecar container, the zookeeper container and the vector sidecar container. All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector aggregator.

Aggregator and sinks

The aggregator is configured to use one or multiple sinks (for example OpenSearch, Elasticsearch), it sends all logs to all sinks. If a sink is unavailable, the aggregator buffers the log messages. It is also the single point where the sinks are configured, so the sinks are decoupled from the Stacklet definitions and only need to be configured in this single location for the whole platform.

Configuration

Identical configuration structure in each product

Logging configuration usually requires configuration what to log, where to log it and in which format. Usually you want to persist logs but also view them on the fly in a console, and possibly both of these in different formats. Stackable provides configuration for this out of the box. You only need to enable log aggregation and configure what you want to log. As part of the logging configuration, individual modules can be configured with different log levels:

spec:
  clusterConfig:
    vectorAggregatorConfigMapName: vector-aggregator-discovery  (1)
  myRole:
    config:                        (2)
      logging:
        enableVectorAgent: true
        containers:
          main-container:
            console:
              level: INFO
    myRoleGroup:
      config:                      (2)
        logging:                   (3)
          enableVectorAgent: true  (4)
          containers:              (5)
            main-container:
              console:             (6)
                level: DEBUG
              file:
                level: INFO
              loggers:             (7)
                ROOT:
                  level: INFO
                my.module:
                  level: DEBUG
                some.chatty.module:
                  level: NONE
            sidecar-container:
              console:
                level: ERROR

1	The discovery ConfigMap of the Vector aggregator to publish the logs to. This is set at cluster level.
2	The role or role group config containing the logging configuration.
3	The logging configuration fragment, can be set at role or role group level.
4	Enable the Vector agent to aggregate logs.
5	Logging is defined for each container.
6	Console and file appenders can have different log level thresholds.
7	Setting log levels for individual modules is also possible.

Log levels per module - In the loggers section the log levels for each module are configured. Note the ROOT module. This is a special module and refers to the default log level for the container. The log levels are the commonly used ones (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) as well as the NONE level to turn of logging. The default log level is INFO.

Log levels for console and file - The configuration supports setting another level threshold for the console and the file output from which the aggregation is done. A typical use case is that the log level for the console is set to a more detailed level to allow for detailed inspection of a container, but less information in the aggregated log store.

Per container configuration - A Pod actually consists of multiple containers and you might want different log levels for each of them. Also the log levels per module are container specific.

Following the Stackable roles and role groups concept, this configuration fragment can be set either at role or role group level.

Configuring the Aggregator

Follow the installation instructions for the aggregator. Configure a Vector source at adress 0.0.0.0:6000 and configure sinks and additional settings according to your needs.

Configuring the aggregator location

Every product also has a vectorAggregatorConfigMapName property in its spec that needs to be set to the name of the ConfigMap that contains the address of the aggregator. The field is called ADDRESS and the value could be vector-aggregator:6000 if you’re running the aggregator behind a service named vector-aggregator.

Custom overrides

As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file. This is then product specific.

logging:
  enableVectorAgent: false       (1)
  containers:
    my-container:
      custom:
        configMap: my-configmap  (2)

1	The vector logging agent is not deployed.
2	A custom logging configuration is loaded from a ConfigMap called `my-configmap`.