Extracting fields from different syslog messages using props.conf transforms.conf

Thuan — Mon, 28 Sep 2020 19:48:40 GMT

The syslog messages we receive from the firewall have multiple formats. A limited sample is listed below

Apr 30 15:26:20 147.81.86.22 %ASA-4-313004: Denied ICMP type=0, from laddr 10.11.20.59 on interface outside to 147.81.104.109: no matching session
Apr 29 13:39:31 147.81.86.22 %ASA-6-605004: Login denied from 72.81.224.40/63508 to inside:72.81.86.22/ssh for user "dstrollo"
Apr 28 13:58:47 147.81.37.241 %ASA-3-710003: TCP access denied by ACL from 45.64.188.70/48986 to Internet:72.83.128.9/23

Our requirement is to be able to extract all the fields in these messages for our analysts.

My question is whether the configuration below will work. If it doesn't it may mess up the existing index. Any advice will be much appreciated.

In props.conf, I set up a basic search to extract the syslog message in each record. The results are listed below

Denied ICMP type=0, from laddr 10.11.20.59 on interface outside to 147.81.104.109: no matching session
Login denied from 72.81.224.40/63508 to inside:72.81.86.22/ssh for user "dstrollo"
TCP access denied by ACL from 45.64.188.70/48986 to Internet:72.83.128.9/23

The props.conf is listed below

[syslog]

-------

NO_BINARY_CHECK = 1
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = false
TIME_PREFIX = ^

%Y = year, %m = month, %d = day, %H = hour, %M = minute, %S = seconds, %z = time zone offset

example = 2015-03-25T16:22:01-04:00

the - ( in -4:00 => %z ) is part of the timezone specification (UTC-04:00 versus UTC+04:00)

TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
EXTRACT-syslog = \s(?[^\s]+)\s(?[^:]+:)\s(?.+)

Then I use different transforms to process each message type separately at search time

In props.conf

[syslog]
....
EXTRACT-syslog = "as above"
REPORT-SyslogMsg = SyslogMsg1, SyslogMsg2, SyslogMsg3

In trasnforms.conf

[SyslogMsg1]
REGEX = "extract fields in Msg1"

SyslogMsg2
REGEX = "extract fields in Msg2"

SyslogMsg3
REGEX = "extract fields in Msg3"

Thank you.

Re: Extracting fields from different syslog messages using props.conf transforms.conf

woodcock — Sat, 02 May 2015 13:34:31 GMT

It will work and even if it doesn't, because you are using the "REPORT-" directive (Search-time) instead of the "TRANSFORMS-" (or "EXTRACT-") directive (Index-time), it will not do any permanent modifications so there is no risk. Is there really missing backslashes for this transform or did you just mess up the markdown for it:

I would make sure each of your 3 REGEX strings contains the string literal text for each individual message variation to avoid false extractions (e.g. "ASA-4-313004", "ASA-6-605004", and "ASA-3-710003").

P.S. You do not need the "EXTRACT-syslog" more than once.

Re: Extracting fields from different syslog messages using props.conf transforms.conf

Thuan — Fri, 08 May 2015 19:17:05 GMT

Thank you !,

topic Re: Extracting fields from different syslog messages using props.conf transforms.conf in Getting Data In

Extracting fields from different syslog messages using props.conf transforms.conf

-------

%Y = year, %m = month, %d = day, %H = hour, %M = minute, %S = seconds, %z = time zone offset

example = 2015-03-25T16:22:01-04:00

the - ( in -4:00 => %z ) is part of the timezone specification (UTC-04:00 versus UTC+04:00)

Re: Extracting fields from different syslog messages using props.conf transforms.conf

Re: Extracting fields from different syslog messages using props.conf transforms.conf