Splunk Search

Can you help me improve the performance of a custom search?

Hi,

We have Linux Auditd data coming into Splunk with sourcetype=linux:audit. In Auditd logs, Record Types define events based on what information is being presented, e.g. a type of "CWD" is triggered to record current working directory, type "SYSCALL" records a system call to kernel etc. Full list here in case anyone's interested. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-aud...

To find out when any user runs a potentially risky command e.g. rm -rf XYZ or chmod 777 XYZ, I use the following search which joins three types(CWD, EXECVE and SYSCALL) based on msg field which holds the id unique across all types for a specific Event.

sourcetype=linux:audit type=CWD 
| fields + msg, cwd
| JOIN msg [ search sourcetype=linux:audit  NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown
| fields + _time, msg, auid]
| JOIN msg [ search sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)    | fields + _time, msg, host, a0, a1, a2, a3]  | table _time, host, msg, auid, a0, a1, a2, a3, cwd

Search works fine and gets the results but it takes exceptionally long to execute and almost every time I see the following under job summary.

info : [subsearch]: Search Processor: Subsearch produced 50000 results, truncating to maxout 50000.
warn : The limit has been reached for log messages in info.csv. 20 messages have not been written to info.csv. Please refer to search.log for these messages or limits.conf to configure this limit.
Is the search missing out on some data? Is there a way to make search more efficient and ensure it searches every event in the given time range?

Any helps is appreciated..

~ Abhi

0 Karma

Path Finder

Hi abhijittikekar,

Try to run below query.

sourcetype=linux:audit type=CWD
| table msg, cwd
| map
[ search sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown msg=$msg$
|eval cwd=$cwd$
| table _time, msg, auid,cwd]
|map

[ search sourcetype=linux:audit NOT auid=4294967295 msg=$msg$ type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)

| eval auid=$auid$ , cwd=$cwd$
| table + _time, msg, host, a0, a1, a2, a3,auid,cwd]]
| table _time, host, msg, auid, a0, a1, a2, a3, cwd

Thanks,
Bhavik

0 Karma

Esteemed Legend

Like this:

(index=YouShouldAlwaysSpecifyIndexValues AND sourcetype=linux:audit) AND 
((type="CWD")
OR
(NOT auid="4294967295" AND type="SYSCALL" AND (comm="chmod" OR comm="rm" OR comm="chown"))
OR
(NOT auid="4294967295" AND type="EXECVE" AND (a0="chmod" AND (a1="-R" AND (a2="777" OR a2="755")) OR (a1="777")) OR (a0="rm" AND a2="-r*") OR (a0="chown")))
| fields _time msg, cwd, auid, host, a0, a1, a2, a3
| stats values(*) AS * BY msg
0 Karma

Contributor

index=?? sourcetype=linux:audit type=CWD OR ( index= ?? sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown or (INDEX=?? sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown) )

| fields + _time, msg, auid, cwd,host, a0, a1, a2, a3]
| stats values(*) as * by msg

0 Karma

Explorer

You can use stats commands to accomplish the same thing:

| stats values(field1) as field1, values(field2) as field2 by msg
0 Karma

SplunkTrust
SplunkTrust

Watch this video, apply what you learned and your search will be significantly faster

https://conf.splunk.com/files/2018/recordings/master-joining-datasets-without-fn1784.mp4

0 Karma

Thanks for the reference. There is tons of information there.. going through the content. i'll re post updated search after making changes.

~ Abhi

Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!