https://github.com/fluent/fluentd-kubernetes-daemonset

实践一：监听业务应用日志，收集并输出

目标：收集容器内的nginx应用的access.log日志，原始日志的格式为：

$ tail -f access.log
...
53.49.146.149 1561620585.973 0.005 502 [27/Jun/2019:15:29:45 +0800] 178.73.215.171 33337 GET https

思路：

• 配置fluent.conf

  • 使用@tail插件通过监听access.log文件
  • 启动fluentd服务

• 手动追加内容至access.log文件

• 观察本地输出内容是否符合预期

fluent.conf

<source>
	@type tail
	path /var/log/nginx/access.log
	pos_file /var/log/nginx/nginx_access.posg
	tag nginx_access
	format none
	@log_level trace
</source>
<match  nginx_access>
	@type stdout
</match>

启动服务，追加文件内容：

# https://github.com/fluent/fluentd-kubernetes-daemonset
$ docker run -u root --rm -ti --entrypoint='' fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-amd64 bash
/ # fluentd -c fluent.conf
/ # echo '53.49.146.149 1561620585.973 0.005 502 [27/Jun/2019:15:29:45 +0800] 178.73.215.171 33337 GET https' \>\>/var/log/nginx/access.log


# 输出结果：
2022-11-19 03:31:42.956800427 +0000 nginx_access: {"message":"53.49.146.149 1561620585.973 0.005 502 [27/Jun/2019:15:29:45 +0800] 178.73.215.171 33337 GET https"}

实践二：实现业务应用日志的收集及字段解析

目标：收集容器内的nginx应用的access.log日志，并解析日志字段为JSON格式，原始日志的格式为：

$ tail -f access.log
...
53.49.146.149 1561620585.973 0.005 502 [27/Jun/2019:15:29:45 +0800] 178.73.215.171 33337 GET https

收集并处理成：

{
    "serverIp": "53.49.146.149",
    "timestamp": "1561620585.973",
    "respondTime": "0.005",
    "httpCode": "502",
    "eventTime": "27/Jun/2019:15:29:45 +0800",
    "clientIp": "178.73.215.171",
    "clientPort": "33337",
    "method": "GET",
    "protocol": "https"
}

fluent.conf

<source>
	@type tail
	path /var/log/nginx/access.log
	pos_file /var/log/nginx/nginx_access.posg
	tag nginx_access
	format none
	@log_level trace
</source>
<filter  nginx_access>
    @type parser
	key_name message
	format  /(?<serverIp>[^ ]*) (?<timestamp>[^ ]*) (?<respondTime>[^ ]*) (?<httpCode>[^ ]*) \[(?<eventTime>[^\]]*)\] (?<clientIp>[^ ]*) (?<clientPort>[^ ]*) (?<method>[^ ]*) (?<protocol>[^ ]*)/
</filter>
<match  nginx_access>
  @type stdout
</match>

输出结果：

2022-11-19 03:34:43.005963334 +0000 nginx_access: {"serverIp":"53.49.146.149","timestamp":"1561620585.973","respondTime":"0.005","httpCode":"502","eventTime":"27/Jun/2019:15:29:45 +0800","clientIp":"178.73.215.171","clientPort":"33337","method":"GET","protocol":"https"}

使用该网站进行正则校验： http://fluentular.herokuapp.com

实践三：使用ruby实现日志字段的转换及自定义处理

<source>
	@type tail
	path /var/log/nginx/access.log
	pos_file /var/log/nginx/nginx_access.posg
	tag nginx_access
	format none
	@log_level trace
</source>
<filter  nginx_access>
    @type parser
	key_name message
	format  /(?<serverIp>[^ ]*) (?<timestamp>[^ ]*) (?<respondTime>[^ ]*) (?<httpCode>[^ ]*) \[(?<eventTime>[^\]]*)\] (?<clientIp>[^ ]*) (?<clientPort>[^ ]*) (?<method>[^ ]*) (?<protocol>[^ ]*)/
</filter>
<filter  nginx_access>   
   @type record_transformer
   enable_ruby
   <record>
	  host_name "#{Socket.gethostname}"
      my_key  "my_val"
      tls ${record["protocol"].index("https") ? "true" : "false"}
    </record>
</filter>
<match  nginx_access>
  @type stdout
</match>

输出结果：

2022-11-19 03:37:08.365890122 +0000 nginx_access: {"serverIp":"53.49.146.149","timestamp":"1561620585.973","respondTime":"0.005","httpCode":"502","eventTime":"27/Jun/2019:15:29:45 +0800","clientIp":"178.73.215.171","clientPort":"33337","method":"GET","protocol":"https","host_name":"23c22d63a1a7","my_key":"my_val","tls":"true"}

Input -\> filter 1 -\> ... -\> filter N -\> Buffer -\> Output