https://www.iyunv.com/thread-144079-1-1.html

需求:

通过自定义rsyslog的输出格式,通过json编码方式将日志信息发送给logstash进行处理。

Rsyslog配置:

1、将centos6.5的rsyslog升级到最新版

[iyunv@centos-yum ~]# vi /etc/yum.repo.d/rsyslog.repo

[rsyslog_v8]

name=Adiscon CentOS-$releasever – local packages for $basearch

baseurl=http://rpms.adiscon.com/v8-stable/epel-$releasever/$basearch

enabled=1

gpgcheck=0

gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon

protect=1

[iyunv@centos-yum ~]#yum update

[iyunv@centos-yum ~]#yum install rsyslog

2、修改rsyslog的配置文件

[iyunv@centos-yum ~]# vi /etc/rsyslog.conf
*增加以下内容
template(name=”json_lines” type=”list” option.json=”on”) {
constant(value=”{“)
constant(value=”\”timestamp\”:\””)
property(name=”timereported” dateFormat=”rfc3339″)
constant(value=”\”,\”message\”:\””)
property(name=”msg”)
constant(value=”\”,\”host\”:\””)
property(name=”hostname”)
constant(value=”\”,\”severity\”:\””)
property(name=”syslogseverity-text”)
constant(value=”\”,\”facility\”:\””)
property(name=”syslogfacility-text”)
constant(value=”\”,\”app-name\”:\””)
property(name=”programname”)
constant(value=”\”,\”procid\”:\””)
property(name=”procid”)
constant(value=”\”}\n”)
}

action(
type=”omfwd”
Target=”10.20.20.67″
Port=”8515″
Protocol=”tcp”
template=”json_lines”
)

rsyslog的json格式输出需要首先定义日志格式模板template,template通过以下格式定义:

template(parameters) { list-descriptions }

每个tmplate都必须有一个name参数,用于定义template名称,该template名称必须唯一;template还需要有一个参数type,用于定义template类型,rsyslog的template支持以下类型:

list

subtree

string

plugin

本文只讨论在list的类型下输出json格式,其它类型不做讨论。

template的<options>部分为可选配置,template的options大小写敏感,目前有以下类型:

option.sql – 将输出格式化为适合mysql的格式。

option.stdsql – 将输出格式化为适合兼容标准sql的格式。

option.json – 将输出格式化为json格式。

本次我们采用option.json格式输出。

2.2 template中List定义

在list类型的template中,通过一组constant和property描述定义该list,constant描述的是固定的文本信息,property定义了引用的rsyslog属性,例如:

template(name=”tpl1″ type=”list”) {
constant(value=”Syslog MSG is: ‘”)
property(name=”msg”)
constant(value=”‘, “)
property(name=”timereported”
dateFormat=”rfc3339″ caseConversion=”lower”)
constant(value=”\n”)
}

property可以有很多参数选择,本例中我们property中的name参数。用于调用rsyslog的properties。

2.3 rsyslog的property

rsyslog中的data item被称为“properties”,需要通过调用对应的property来访问数据。

本例中基与name调用需要发送的日志信息,例如:

property(name=”hostname”)

详细的property可以参考文档最后的property参考。

2.4 定义对应的rsyslog输出action

rsyslog支持许多种输出方式,本文调用omfwd模块将日志输出到对应的logstash服务器,如下

action(
type=”omfwd”
Target=”10.20.20.67″
Port=”8515″
Protocol=”tcp”
template=”json_lines”
)

3、logstash对应配置文件

logstash对应的input定义如下:

input {
tcp {
port => “8515”
codec => “json”
type => “syslog-json”
}
}

可以观察到成功收到对应的syslog-json格式日志

{
“timestamp” => “2015-11-26T10:10:59.257734+08:00”,
“message” => ” pam_unix(sshd:session): session opened for user root by (uid=0)”,
“host” => “centos-yum”,
“severity” => “info”,
“facility” => “authpriv”,
“syslog-tag” => “sshd[15251]:”,
“app-name” => “sshd”,
“procid” => “15251”,
“@version” => “1”,
“@timestamp” => “2015-11-26T02:10:58.013Z”,
“type” => “syslog-json”
}

附:rsyslog的property参考

rsyslog中的data item被称为“properties”,需要通过调用对应的property来访问数据。

Messgage Properties:

msg

the MSG part of the message (aka “the message” ;))

rawmsg

the message excactly as it was received from the socket. Should be useful for debugging. It is also useful if a message should be forwarded totally unaltered.

rawmsg-after-pri

Almost the same as rawmsg, but the syslog PRI is removed. If no PRI was present, rawmsg-after-pri is identical to rawmsg. Note that the syslog PRI is header field that contains information on syslog facility and severity. It is enclosed in greater-than and less-than characters, e.g. “<191>”. This field is often not written to log files, but usually needs to be present for the receiver to properly classify the message. There are some rare cases where one wants the raw message, but not the PRI. You can use this property to obtain that. In general, you should know that you need this format, otherwise stay away from the property.

hostname

hostname from the message

source

alias for HOSTNAME

fromhost

hostname of the system the message was received from (in a relay chain, this is the system immediately in front of us and not necessarily the original sender). This is a DNS-resolved name, except if that is not possible or DNS resolution has been disabled.

fromhost-ip

The same as fromhost, but always as an IP address. Local inputs (like imklog) use 127.0.0.1 in this property.

syslogtag

TAG from the message

programname

the “static” part of the tag, as defined by BSD syslogd. For example, when TAG is “named[12345]”, programname is “named”.

priPRI part of the message – undecoded (single value)pri-textthe PRI part of the message in a textual form with the numerical PRI appended in brackes (e.g. “local0.err<133>”)iutthe monitorware InfoUnitType – used when talking to a MonitorWare backend (also for Adiscon LogAnalyzer)syslogfacilitythe facility from the message – in numerical formsyslogfacility-textthe facility from the message – in text formsyslogseverityseverity from the message – in numerical formsyslogseverity-textseverity from the message – in text formsyslogpriorityan alias for syslogseverity – included for historical reasons (be careful: it still is the severity, not PRI!)syslogpriority-textan alias for syslogseverity-texttimegeneratedtimestamp when the message was RECEIVED. Always in high resolutiontimereportedtimestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds)timestampalias for timereportedprotocol-versionThe contents of the PROTCOL-VERSION field from IETF draft draft-ietf-syslog-protcolstructured-dataThe contents of the STRUCTURED-DATA field from IETF draft draft-ietf-syslog-protocolapp-nameThe contents of the APP-NAME field from IETF draft draft-ietf-syslog-protocolprocidThe contents of the PROCID field from IETF draft draft-ietf-syslog-protocolmsgidThe contents of the MSGID field from IETF draft draft-ietf-syslog-protocolinputnameThe name of the input module that generated the message (e.g. “imuxsock”, “imudp”). Note that not all modules necessarily provide this property. If not provided, it is an empty string. Also note that the input module may provide any value of its liking. Most importantly, it is not necessarily the module input name. Internal sources can also provide inputnames. Currently, “rsyslogd” is defined as inputname for messages internally generated by rsyslogd, for example startup and shutdown and error messages. This property is considered useful when trying to filter messages based on where they originated – e.g. locally generated messages (“rsyslogd”, “imuxsock”, “imklog”) should go to a different place than messages generated somewhere.

System Properties:

System Proprties由rsyslog的core engine提供数据。

$bom

The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in templates for RFC5424 support, when the character set is know to be Unicode.

$now

The current date stamp in the format YYYY-MM-DD

$year

The current year (4-digit)

$month

The current month (2-digit)

$day

The current day of the month (2-digit)

$hour

The current hour in military (24 hour) time (2-digit)

$hhour

The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1.

$qhour

The current quarter hour we are in. Much like $HHOUR, but values range from 0 to 3 (for the four quater hours that are in each hour)

$minute

The current minute (2-digit)

$myhostname

The name of the current host as it knows itself (probably useful for filtering in a generic way)

 

 

 

 

 

 

 


About

由于环境基于CentOS 6.7 x64,rsyslog本身就是OS的组件,由于本文使用rsyslog作为统一日志采集器,需要与kafka进行数据交付,而原始支持kafka组件是在v8.7.0以后版本才支持,而系统自带的rsyslog是版本是v5的,因此需要对rsyslog版本进行升级。

有网的方法比较简单,通过yum命令即可,官网参考官网 http://www.rsyslog.com/rhelcentos-rpms/ 即可安装指定版本。

安装说明

由于鉴于内网环境无Internet无法使用yum进行安装,因此选择离线rpm方式升级rsyslog(rpm包下载地址:http://rpms.adiscon.com/ )。

此处有坑:按习惯,一般使用官网提供的最新的稳定版/v8-stable/epel-7,然而由于CentOS 6.7 glibc版本为2.12而epel-7依赖的libgt-0.3.11-1.el7.x86_64、libfastjson-0.99.2-1.el7.x86_64、libestr-0.1.10-1.el7.x86_64几个组件需要 libc.so.6: version `GLIBC_2.14,CentOS 6.7升级后会提示bash版本有问题,因此此处选用/v8-stable/epel-6版本(ps:此坑是个人问题,centos6.x本来就应该使用epel-6,因此此坑可忽略)

需要下载的rpm

可在http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/ 获取

  • json-c-0.11-4.el6.x86_64.rpm
  • libfastjson-0.99.2-1.el6.x86_64.rpm
  • libgt-0.3.11-1.el6.x86_64.rpm
  • libestr-0.1.10-1.el6.x86_64.rpm
  • liblogging-1.0.5-1.el6.x86_64.rpm
  • librdkafka1-0.8.5-0.x86_64.rpm
  • rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
  • rsyslog-8.19.0-1.el6.x86_64.rpm

rpm安装

执行安装前 先给没个文件加上运行属性 chmod +x

rpm -ivh  json-c-0.11-4.el6.x86_64.rpm
rpm -ivh libfastjson-0.99.2-1.el6.x86_64.rpm
rpm -ivh libgt-0.3.11-1.el6.x86_64.rpm
rpm -ivh libestr-0.1.10-1.el6.x86_64.rpm
rpm -ivh liblogging-1.0.5-1.el6.x86_64.rpm
rpm -ivh librdkafka1-0.8.5-0.x86_64.rpm

#这边使用 -U 参数是为了直接升级本地rsyslog
rpm -Uvh rsyslog-8.19.0-1.el6.x86_64.rpm

#rsyslog-kafka组件安装依赖 rsyslog-8.19.0librdkafka,该命令最后执行
rpm -ivh rsyslog-kafka-8.19.0-1.el6.x86_64.rpm

验证安装

  • 安装完后执行 rsyslogd -v, 验证rsyslog是否升级成功。
# rsyslogd -v
rsyslogd 8.19.0, compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              No
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        Number of Bits in RainerScript integers: 64
 
See http://www.rsyslog.com for more information.
  • 查看/lib64/rsyslog/(32位系统为 /lib/rsyslog) 中 是否存在 omkafka.so,验证rsyslog-kafka是否安装成功。

配置说明

rsyslog官网是个大坑,能路过尽量路过。

接下来说明下配置,首先声明下本文使用rsyslog的目的是作为应用的log collector,应用使用log4x(log4net:log4net.Appender.RemoteSyslogAppender, log4j:org.apache.log4j.net.SyslogAppender),通过UDP协议将日志传输到rsyslog。

rsyslo的默认的配置文件 /etc/rsyslog.config

# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced
# things, RainerScript configuration is suggested.

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously done by rklogd)
#module(load"immark")  # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  /var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
 
 
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###                                      

接下来说明配置修改:

配置 input

  • 开启接收UDP消息
module(load="imudp") # 加载imudp模块
input(type="imudp" port="514") # 配置UDP监听的端口号

配置 output

  • 开启数据转发kafka
 module(load="omkafka") # 配置加载omkafka模块
 # 转发到kafka消息格式
template(name="rsyslogToKafka" type="list" option.json="on"){
        constant(value="{")
        #constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
        constant(value="\"message\":\"") property(name="msg" format="json")
        constant(value="\",\"host\":\"") property(name="hostname")
        constant(value="\",\"severity\":\"") property(name="syslogseverity")
        constant(value="\",\"severity-text\":\"") property(name="syslogseverity-text")
        constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
        constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
        constant(value="\",\"type\":\"application\"")
        constant(value=",\"recieve-from\":\"") property(name="fromhost-ip")
        constant(value="\",\"recieve-at\":\"") property(name="timegenerated" dateFormat="rfc3339")
        constant(value="\",\"report-at\":\"") property(name="timereported" dateFormat="rfc3339")
        constant(value="\"}")
}

# 配置kafka相关信息及模板
action( type="omkafka"
        topic="test"
        broker="11.4.74.26:9092"
        template="rsyslogToKafka"
)
  • 另外一个坑:关于EscapeControlCharacters转换的问题

rsyslog接收的时候会对一些回车换行等敏感字符进行转换 比如 \r\n 会被转换为#015#012,而通过log4x提交的日志如果包含Exception在kibana显示就不能正常的换行,因此关闭了rsyslog中敏感字的转换

 $EscapeControlCharactersOnReceive off

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注