将 Docker 日志发送到 Quickwit

要将 Docker 容器日志发送到 Quickwit，您只需要设置一个带有文件日志接收器的 OpenTelemetry Collector。在本教程中，我们将使用 docker compose 来启动收集器和 Quickwit。

您只需要一分钟即可获得 Quickwit 日志 UI！

OTEL collector 配置

以下 collector 配置将收集位于 /var/lib/docker/containers/*/*-json.log 的 Docker 日志（根据您的系统，日志文件可能位于其他位置），添加一些属性并通过 gRPC 发送到 Quickwit 的 http://quickwit:7281。

otel-collector-config.yaml

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*-json.log
    operators:
     - id: parser-docker
       timestamp:
         layout: '%Y-%m-%dT%H:%M:%S.%LZ'
         parse_from: attributes.time
       type: json_parser
     - field: attributes.time
       type: remove
     - id: extract_metadata_from_docker_tag
       parse_from: attributes.attrs.tag
       regex: ^(?P<name>[^\|]+)\|(?P<image_name>[^\|]+)\|(?P<id>[^$]+)$
       type: regex_parser
       if: 'attributes?.attrs?.tag != nil'
     - from: attributes.name
       to: resource["docker.container.name"]
       type: move
       if: 'attributes?.name != nil'
     - from: attributes.image_name
       to: resource["docker.image.name"]
       type: move
       if: 'attributes?.image_name != nil'
     - from: attributes.id
       to: resource["docker.container.id"]
       type: move
       if: 'attributes?.id != nil'
     - from: attributes.log
       to: body
       type: move

processors:
  batch:
    timeout: 5s

exporters:
  otlp/qw:
    endpoint: quickwit:7281
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/qw]

启动 OTEL collector 和 Quickwit 实例

让我们使用以下配置的 docker compose：

docker-compose.yaml

version: "3"

x-default-logging: &logging
 driver: "json-file"
 options:
   max-size: "5m"
   max-file: "2"
   tag: "{{.Name}}|{{.ImageName}}|{{.ID}}"

services:
  quickwit:
    image: quickwit/quickwit:${QW_VERSION:-0.8.1}
    volumes:
      - ./qwdata:/quickwit/qwdata
    ports:
      - 7280:7280
    environment:
      - NO_COLOR=true
    command: ["run"]
    logging: *logging

  otel-collector:
    user: "0" # Needed to access the directory /var/lib/docker/containers/
    image: otel/opentelemetry-collector-contrib:${OTEL_VERSION:-0.87.0}
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: ["--config=/etc/otel-collector-config.yaml"] 
    logging: *logging

您会注意到自定义的 logging，OTEL collector 将使用这些额外的信息来丰富日志。

运行并搜索

下载配置文件并启动容器：

mkdir qwdata
docker compose up

几秒钟后，您将在 Quickwit UI http://localhost:7280 中看到日志。

它看起来应该是这样的：

{
  "attributes": {
    "log.file.name": "34ad1a84c71de1d29ad75f99b56d01205e2976440f2398734037151ba2bcde1a-json.log",
    "stream": "stdout"
  },
  "body": {
    "message": "2023-10-23T16:39:57.892  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.ListObjects => 200\n"
  },
  "observed_timestamp_nanos": 1698079197979435000,
  "service_name": "unknown_service",
  "severity_number": 0,
  "timestamp_nanos": 1698079197892726000,
  "trace_flags": 0
}

故障排除

有可能您在 UI 中看不到任何日志。在这种情况下，请检查 docker compose 的日志。问题通常来自 OTEL 收集器的错误配置。