Parsing at search time variable white spaces

FrancoiseMathy — Tue, 29 Sep 2020 11:08:42 GMT

Hello,

I am new in Splunk parsing and I am facing some problems with this.
I am trying to parse, at Search Time, a source of logs (containing two sourcetypes, AIX and Linux). In the AIX sourcetype, I have three different type of events (one for "PROC_Execute", another for "CRON" and the last one for "S_PASSWD_READ". The beginning of the event is the same for the three types.

I have a UF -> IDX environment -> SH.

I have tried many many things with transforms.conf and/or props.conf without any success. Everytime I search for the sourcetype events, nothing is parsed in SH Gui.

My logs look like this, for AIX (sourcetype=aix) (I will do linux afterwards.)

PROC_Execute root OK Mon Jul 25 10:53:26 2016 uncompress 14614680 21364880 root

euid: 0 egid: 0 epriv: ffffffff:ffffffff name /usr/bin/uncompress -f /audit/tempfile.21364880
PROC_Execute i51534a OK Mon Jul 25 10:53:26 2016 sh 15335586 7798934 root

euid: 0 egid: 0 epriv: ffffffff:ffffffff name sh -c LANG=C /usr/bin/vmstat
S_PASSWD_READ root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

audit object read event detected /etc/security/passwd
S_PASSWD_READ root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

audit object read event detected /etc/security/passwd
CRON_Start root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

event = start cron job cmd = > /lpar2rrd/product/logs/error.log time = Mon Jul 25 10:54:00 2016

As you can see, multiple white spaces is used as delimiter and more than that, the amount of white spaces between the "sh"/"uncompress" and the "process_number" can vary.

I tried the following in the SH gui and it works fine. The events are parsed with correct values :
"index= host= | rex field=_raw "^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$""

In transforms.conf and/or props.congf, I tried using (?x) for free-spacing mode as the numbers of free spaces between "trail_aix_short_command" and "trail_aix_parent_process_id" can vary.
I tried to put in props.conf, only the first extract "EXTRACT-aix_command = ^(?P[^ ]+)" (created with the fiedls extractor) without success. -> no parsing at search time in SR GUI.

I tried, in transforms.conf, the following
"(?x)(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.)"
(with and without ") and
"(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.)"
(with and without ")
without success.

I tried also with (.*) for each element in transforms.conf without success.

=> I need some help.

What I would like to do is parse until the "user name" in a common extraction, and then use three other extractions, one for "CRON", one for "PASSWD" and one for "PROC_EXECUTE" => can you help?

Re: Parsing at search time variable white spaces

sundareshr — Fri, 23 Sep 2016 17:11:11 GMT

You said a specific rex worked in the GUI. Did you try that same (as-is) regex in the props/transforms?

Re: Parsing at search time variable white spaces

FrancoiseMathy — Tue, 29 Sep 2020 11:10:32 GMT

Hello,

Yes I haved tried what you suggested :
In GUI, I put "index=trailsyslog host = | rex field=_raw "^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$"
=> this is ok. The events are parsed and different fields (trail_aix_status, trail_aix; time, ...) are present in the GUI.

Then,
In transforms, I put
[aix_common_fields]
REGEX = ^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$

In props, I put
[trailsyslog-aix]
REPORT-aix_common_fields = aix_common_fields
.
FYI, trailsyslog-aix is my sourcetype

=> I did a "splunk reload deploy-server"
=> I did the search "index=trailsyslog host = => nothing is parsed. The fields are not present.

topic Re: Parsing at search time variable white spaces in Splunk Search

Parsing at search time variable white spaces

Re: Parsing at search time variable white spaces

Re: Parsing at search time variable white spaces