Fluentd Kubernetes DaemonSet 镜像是专为在Kubernetes集群中以DaemonSet方式部署Fluentd而设计的Docker镜像。Fluentd是一个开源的数据收集器,能够统一日志收集和处理流程,为Kubernetes集群提供集中式日志管理解决方案。
该镜像旨在简化Fluentd在Kubernetes环境中的部署和配置,支持多种日志输出目的地,并针对容器化环境进行了优化。
自v1.17.0起,容器镜像构建流程已从hub.docker.com的自动构建迁移到GitHub Actions。这是因为hub.docker.com对自动构建数量有限制,而GitHub Actions则没有构建流水线数量的限制。
注意:v1.16.5或更早版本的DaemonSet镜像存在一些限制:
papertrail、syslog镜像(x86_64/arm64)不再发布logentries、loggly、logzio、s3的arm64镜像不再发布(仅支持x86_64) 如果需要使用上述未发布的镜像,请自行构建。Dockerfile仍在本仓库中维护。
Azureblob
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-azureblob-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-azureblob-1Elasticsearch8
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-elasticsearch8-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-elasticsearch8-1Elasticsearch7
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-elasticsearch7-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-elasticsearch7-1docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearchOpensearch
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-opensearch-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-opensearch-1Loggly
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-loggly-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-loggly-1Logentries
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-logentries-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-logentries-1Cloudwatch
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-cloudwatch-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-cloudwatch-1S3
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-s3-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-s3-1Syslog
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-syslog-1Forward
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-forward-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-forward-1Gcs
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-gcs-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-gcs-1Graylog
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-graylog-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-graylog-1Papertrail
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-papertrail-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-papertrail-1Logzio
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-logzio-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-logzio-1Kafka
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-kafka-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-kafka-1Kafka2
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-kafka2-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-kafka2-1Kinesis
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-kinesis-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-kinesis-1Datadog
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-datadog-1.1docker pull fluent/fluentd-kubernetes-daemonset:v1.19-debian-datadog-1各输出目的地的x86_64镜像标签格式为:v{版本}-debian-{目的地}-amd64-{修订号},例如:
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-elasticsearch8-amd64-1.1各输出目的地的arm64镜像标签格式为:v{版本}-debian-{目的地}-arm64-{修订号},例如:
docker pull fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-elasticsearch8-arm64-1.1该镜像支持通过环境变量进行基本配置:
| 环境变量 | 描述 | 默认值 |
|---|---|---|
FLUENT_UID | 运行Fluentd的用户ID | 1000 |
FLUENT_CONTAINER_TAIL_PATH | 容器日志文件路径 | /var/log/containers/*.log |
FLUENT_CONTAINER_TAIL_EXCLUDE_PATH | 排除的日志文件路径 | 无 |
FLUENTD_SYSTEMD_CONF | 是否启用systemd日志收集 | enable |
FLUENTD_PROMETHEUS_CONF | 是否启用Prometheus指标 | enable |
FLUENT_POS_EXTRA_DIR | 额外的pos文件目录 | 无 |
LD_PRELOAD | 预加载的库 | 无 |
在Kubernetes默认配置下,Fluentd需要root权限才能读取/var/log目录下的日志并写入pos文件。可以通过设置环境变量FLUENT_UID=0来以root用户运行:
yamlenv: - name: FLUENT_UID value: "0"
虽然镜像提供了默认配置,但用户可以通过Kubernetes ConfigMap来自定义配置。每个镜像包含以下配置文件:
fluent.conf: 目的地设置(Elasticsearch、Kafka等)kubernetes.conf: Kubernetes特定设置,包括日志文件tail输入和元数据过滤tail_container_parse.conf: 容器日志解析器配置prometheus.conf: Prometheus监控插件配置systemd.conf: systemd日志收集配置可以通过ConfigMap覆盖这些配置文件,例如:
yamlapiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: kube-system data: fluent.conf: | # 自定义输出配置 <match **> @type elasticsearch host elasticsearch-master port 9200 logstash_format true logstash_prefix fluentd include_tag_key true tag_key @log_name flush_interval 5s </match> tail_container_parse.conf: | <parse> @type cri </parse>
然后在DaemonSet中挂载此ConfigMap:
yamlvolumeMounts: - name: config-volume mountPath: /fluentd/etc/fluent.conf subPath: fluent.conf - name: config-volume mountPath: /fluentd/etc/tail_container_parse.conf subPath: tail_container_parse.conf volumes: - name: config-volume configMap: name: fluentd-config
默认情况下,镜像使用json解析器处理Docker格式的日志。对于使用containerd或cri-o的集群,需要使用cri解析器:
yamlenv: - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE value: "cri"
或者通过ConfigMap覆盖tail_container_parse.conf:
<parse> @type cri </parse>
可以通过FLUENT_CONTAINER_TAIL_EXCLUDE_PATH环境变量排除特定日志:
yamlenv: - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH value: '["/var/log/containers/fluentd-*", "/var/log/containers/kube-*"]'
自v1.17.0-1.3/v1.16.5-1.3版本起,jemalloc内存分配器默认禁用。如果不使用systemd插件,可以通过以下方式启用:
yamlenv: - name: LD_PRELOAD value: "/usr/lib/libjemalloc.so.2"
如果不需要收集systemd日志,可以通过环境变量禁用:
yamlenv: - name: FLUENTD_SYSTEMD_CONF value: "disable"
如果不需要Prometheus监控,可以通过环境变量禁用:
yamlenv: - name: FLUENTD_PROMETHEUS_CONF value: "disable"
yamlapiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-elasticsearch image: fluent/fluentd-kubernetes-daemonset:v1.19-debian-elasticsearch7-1 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi env: - name: FLUENT_UID value: "0" - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-master" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENTD_SYSTEMD_CONF value: "disable" volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
yamlapiVersion: v1 kind: ServiceAccount metadata: name: fluentd-elasticsearch namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluentd-elasticsearch rules: - apiGroups: [""] resources: - namespaces - pods verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: fluentd-elasticsearch subjects: - kind: ServiceAccount name: fluentd-elasticsearch namespace: kube-system roleRef: kind: ClusterRole name: fluentd-elasticsearch apiGroup: rbac.authorization.k8s.io
bashoc project kube-system oc create -f [***] oc adm policy add-scc-to-user privileged -z fluentd oc patch ds fluentd -p "spec: template: spec: containers: - name: fluentd securityContext: privileged: true" oc delete pod -l k8s-app=fluentd-logging
当在同一集群中部署多个Fluentd实例(例如需要将日志发送到多个目的地),需要设置FLUENT_POS_EXTRA_DIR环境变量来指定额外的pos文件目录,避免pos文件冲突:
yamlenv: - name: FLUENT_POS_EXTRA_DIR value: "/var/log/fluentd-pos-extra"
由于Debian 10上的Zookeeper gem存在兼容性问题,Kafka镜像不包含Zookeeper gem。
本仓库不提供Windows Kubernetes DaemonSet支持,但有社区用户提供了相关实现:
推荐使用debian-kafka2或debian-kafka2-arm64镜像,这些镜像使用out_kafka2插件,而debian-kafka或debian-kafka-arm64使用已弃用的out_kafka_buffered插件。
部分镜像由社区贡献者维护,如遇特定镜像问题,请联系相应维护者:
来自真实用户的反馈,见证轩辕镜像的优质服务
免费版仅支持 Docker Hub 加速,不承诺可用性和速度;专业版支持更多镜像源,保证可用性和稳定速度,提供优先客服响应。
免费版仅支持 docker.io;专业版支持 docker.io、gcr.io、ghcr.io、registry.k8s.io、nvcr.io、quay.io、mcr.microsoft.com、docker.elastic.co 等。
当返回 402 Payment Required 错误时,表示流量已耗尽,需要充值流量包以恢复服务。
通常由 Docker 版本过低导致,需要升级到 20.x 或更高版本以支持 V2 协议。
先检查 Docker 版本,版本过低则升级;版本正常则验证镜像信息是否正确。
使用 docker tag 命令为镜像打上新标签,去掉域名前缀,使镜像名称更简洁。
探索更多轩辕镜像的使用方法,找到最适合您系统的配置方式
通过 Docker 登录认证访问私有仓库
在 Linux 系统配置镜像加速服务
在 Docker Desktop 配置镜像加速
Docker Compose 项目配置加速
Kubernetes 集群配置 Containerd
在宝塔面板一键配置镜像加速
Synology 群晖 NAS 配置加速
飞牛 fnOS 系统配置镜像加速
极空间 NAS 系统配置加速服务
爱快 iKuai 路由系统配置加速
绿联 NAS 系统配置镜像加速
QNAP 威联通 NAS 配置加速
Podman 容器引擎配置加速
HPC 科学计算容器配置加速
ghcr、Quay、nvcr 等镜像仓库
无需登录使用专属域名加速
需要其他帮助?请查看我们的 常见问题 或 官方QQ群: 13763429