Splunk Search

Parsing at search time variable white spaces

New Member


I am new in Splunk parsing and I am facing some problems with this.
I am trying to parse, at Search Time, a source of logs (containing two sourcetypes, AIX and Linux). In the AIX sourcetype, I have three different type of events (one for "PROC_Execute", another for "CRON" and the last one for "S_PASSWD_READ". The beginning of the event is the same for the three types.

I have a UF -> IDX environment -> SH.

I have tried many many things with transforms.conf and/or props.conf without any success. Everytime I search for the sourcetype events, nothing is parsed in SH Gui.

My logs look like this, for AIX (sourcetype=aix) (I will do linux afterwards.)

PROC_Execute root OK Mon Jul 25 10:53:26 2016 uncompress 14614680 21364880 root

euid: 0 egid: 0 epriv: ffffffff:ffffffff name /usr/bin/uncompress -f /audit/tempfile.21364880
PROC_Execute i51534a OK Mon Jul 25 10:53:26 2016 sh 15335586 7798934 root

euid: 0 egid: 0 epriv: ffffffff:ffffffff name sh -c LANG=C /usr/bin/vmstat
S_PASSWD_READ root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

audit object read event detected /etc/security/passwd
S_PASSWD_READ root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

audit object read event detected /etc/security/passwd
CRON_Start root OK Mon Jul 25 10:54:00 2016 cron 21233890 6684896 root

event = start cron job cmd = > /lpar2rrd/product/logs/error.log time = Mon Jul 25 10:54:00 2016

As you can see, multiple white spaces is used as delimiter and more than that, the amount of white spaces between the "sh"/"uncompress" and the "process_number" can vary.

I tried the following in the SH gui and it works fine. The events are parsed with correct values :
"index= host= | rex field=_raw "^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$""

In transforms.conf and/or props.congf, I tried using (?x) for free-spacing mode as the numbers of free spaces between "trail_aix_short_command" and "trail_aix_parent_process_id" can vary.
I tried to put in props.conf, only the first extract "EXTRACT-aix_command = ^(?P[^ ]+)" (created with the fiedls extractor) without success. -> no parsing at search time in SR GUI.

I tried, in transforms.conf, the following
"(?x)(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.)"
(with and without ") and
"(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.
(with and without ")
without success.

I tried also with (.*) for each element in transforms.conf without success.

=> I need some help.

What I would like to do is parse until the "user name" in a common extraction, and then use three other extractions, one for "CRON", one for "PASSWD" and one for "PROC_EXECUTE" => can you help?

Tags (1)
0 Karma


You said a specific rex worked in the GUI. Did you try that same (as-is) regex in the props/transforms?

0 Karma

New Member


Yes I haved tried what you suggested :
In GUI, I put "index=trailsyslog host = | rex field=_raw "^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$"
=> this is ok. The events are parsed and different fields (trail_aix_status, trail_aix; time, ...) are present in the GUI.

In transforms, I put
REGEX = ^(?P\w+)\s+(?P\w+)\s+(?P\w+)\s+(?P\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)\s+(?P[^ ]+)\s+(?P\d+)\s+(?P\d+)\s+(?P\w+)\s+(?P.*)$

In props, I put
REPORT-aix_common_fields = aix_common_fields
FYI, trailsyslog-aix is my sourcetype

=> I did a "splunk reload deploy-server"
=> I did the search "index=trailsyslog host = => nothing is parsed. The fields are not present.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...