Filebeat 安装使用

Filebeat 是一个日志文件托运工具，在你的服务器上安装客户端后，filebeat 会监控日志目录或者指定的日志文件，追踪读取这些文件（追踪文件的变化，不停的读），并且转发这些信息到 elasticsearch 或者 logstarsh 中存放。本文介绍 Filebeat 安装部署。

安装

• 最新版本下载
• 之前版本下载

# Ubuntu/Debian
sudo apt-get update && sudo apt-get install filebeat

# CentOS/RHEL
sudo yum install filebeat

# 服务管理
sudo systemctl enable filebeat
sudo systemctl start filebeat

配置

• https://github.com/elastic/beats/blob/main/filebeat/filebeat.reference.yml#L1961
• https://blog.csdn.net/qq_44930876/article/details/127534009

input

• 参考

filebeat.inputs:
- type: log
  paths:
    - "/logs/*"
  fields:
    apache: true
  tags: ["json"]
  fields_under_root: true

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /logs/*.log
  tags: ["tomcat"]
  #指定多行匹配的类型，可选值为pattern（常用，只要下面匹配的参数出现就换行）,count（次数，用于下面匹配的参数出现多少次就换行）
  multiline.type: pattern
  #指定匹配的模式，这里的'^\d{2}代表的是以两个数字开头的，例如11或者12'
  multiline.pattern: '^\d{2}'
  #下面两个参数参考官方架构图即可；https://www.elastic.co/guide/en/beats/filebeat/7.17/multiline-examples.html
  multiline.negate: true
  multiline.match: after

multiline

• 参考

parsers:
- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /logs/*.log
  tags: ["tomcat"]
  parsers:
    - multiline:
        #指定多行匹配的类型，可选值为pattern（常用，只要下面匹配的参数出现就换行）,count（次数，用于下面匹配的参数出现多少次就换行）
        type: pattern
        #指定匹配的模式，这里的'^\d{2}代表的是以两个数字开头的，例如11或者12'
        pattern: '^\d{2}'
        #下面两个参数参考官方架构图即可；https://www.elastic.co/guide/en/beats/filebeat/7.17/multiline-examples.html
        negate: true
        match: after

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.1.1:9200","http://192.168.1.2:9200","http://192.168.1.3:9200"]
  index: "tomcat-error-%{+yyyy.MM.dd}"

output

• 参考

output.elasticsearch:
  hosts: ["node-1:9200"]
  indices:
    - index: "cisco-beat-%{+yyyy.MM}"
      when.contains:
        event.module: "cisco"
  protocol: "https"
  ssl.certificate: "/etc/filebeat/filebeat/filebeat.crt"
  ssl.key: "/etc/filebeat/filebeat/filebeat.key"
  ssl.certificate_authorities:
    - /etc/filebeat/ca/ca.crt
  username: "elastic"
  password: "***********"

使用

• demo1

filebeat.inputs:
- type: log
  #是否启用当前的输入类型，默认值为true
  enabled: true
  #指定数据路径
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  #给当前的输入类型打上标签
  tags: ["linux","DBA运维"]
  #自定义字段
  fields:
    school: "abc"
    class: "linux"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["linux","abc"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  #将自定义字段的key-value放到顶级字段，
  #默认值为false，会将数据放在一个叫fields的字段下面；
  fields_under_root: true

output.console:
  pretty: true

• demo2

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["linux","DBA运维"]
  fields:
    school: "abc"
    class: "linux"

- type: log
  enabled: true
  paths:
    - /logs/*.log
  tags: ["linux","abc"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.1.1:9200","http://192.168.1.2:9200","http://192.168.1.3:9200"]
  indices:
    - index: "linux-elk-%{+yyyy.MM.dd}"
      #匹配指定字段包含的内容
      when.contains:
        tags: "容器运维"
    - index: "linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "abc"

filebeat.inputs:
- type: filestream
  id: mysql-general-log
  enabled: true
  paths:
    - /var/log/mysql/general.log
  # 处理多行日志（如跨行 SQL 语句）
  multiline:
    pattern: '^[0-9]{6} [0-9]{2}:[0-9]{2}:[0-9]{2}'
    negate: true
    match: after
  fields:
    source: mysql
    log_type: general
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  indices:
    - index: "mysql-general-%{+yyyy.MM.dd}"
  username: "elastic"
  password: "your-password"

output.logstash:
  hosts: ["logstash:5044"]

mysql

• 参考

$ filebeat modules enable mysql
# 然后修改文件
$ cat /etc/filebeat/modules.d/mysql.yml
- module: mysql
  error:
    enabled: true
    var.paths: ["/path/to/log/mysql/error.log*"]
  slowlog:
    enabled: true
    var.paths: ["/path/to/log/mysql/mysql-slow.log*"]

排错

重启系统导致无法启动问题

问题描述：

类似 Filebeat Fails After Power Failure，在断电或者版本升级之后都可能概率性触发，root couse 是在异常情况下 registry 文件没有 EOF 标识符。这种情况下，需要删除 registry 文件才可以启动成功。

[root@xiexianbin_cn ~]# systemctl status filebeat.service
● filebeat.service - filebeat
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Fri 2018-04-18 19:51:43 CST; 1min 5s ago
     Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
  Process: 15760 ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat (code=exited, status=1/FAILURE)
 Main PID: 15760 (code=exited, status=1/FAILURE)

Jun 22 19:51:43 xiexianbin_cn systemd[1]: Unit filebeat.service entered failed state.
Jun 22 19:51:43 xiexianbin_cn systemd[1]: filebeat.service failed.
Jun 22 19:51:43 xiexianbin_cn systemd[1]: filebeat.service holdoff time over, schedulin...t.
Jun 22 19:51:43 xiexianbin_cn systemd[1]: start request repeated too quickly for filebe...ce
Jun 22 19:51:43 xiexianbin_cn systemd[1]: Failed to start filebeat.
Jun 22 19:51:43 xiexianbin_cn systemd[1]: Unit filebeat.service entered failed state.
Jun 22 19:51:43 xiexianbin_cn systemd[1]: filebeat.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@xiexianbin_cn ~]#

解决方法：

rm -rf /var/lib/filebeat/registry
systemctl reset-failed filebeat
systemctl start filebeat

filebeat 启动失败问题

错误日志：

Exiting: Could not start registrar: Error loading state: Error decoding states: EOF

解决办法：

cd /var/lib/filebeat/
rm -rf registry
systemctl reset-failed filebeat
systemctl start filebeat