Splunk Search
Highlighted

Can you help me improve the performance of a custom search?

Contributor

Hi,

We have Linux Auditd data coming into Splunk with sourcetype=linux:audit. In Auditd logs, Record Types define events based on what information is being presented, e.g. a type of "CWD" is triggered to record current working directory, type "SYSCALL" records a system call to kernel etc. Full list here in case anyone's interested. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-aud...

To find out when any user runs a potentially risky command e.g. rm -rf XYZ or chmod 777 XYZ, I use the following search which joins three types(CWD, EXECVE and SYSCALL) based on msg field which holds the id unique across all types for a specific Event.

sourcetype=linux:audit type=CWD 
| fields + msg, cwd
| JOIN msg [ search sourcetype=linux:audit  NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown
| fields + _time, msg, auid]
| JOIN msg [ search sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)    | fields + _time, msg, host, a0, a1, a2, a3]  | table _time, host, msg, auid, a0, a1, a2, a3, cwd

Search works fine and gets the results but it takes exceptionally long to execute and almost every time I see the following under job summary.

info : [subsearch]: Search Processor: Subsearch produced 50000 results, truncating to maxout 50000.
warn : The limit has been reached for log messages in info.csv. 20 messages have not been written to info.csv. Please refer to search.log for these messages or limits.conf to configure this limit.
Is the search missing out on some data? Is there a way to make search more efficient and ensure it searches every event in the given time range?

Any helps is appreciated..

~ Abhi

0 Karma
Highlighted

Re: Can you help me improve the performance of a custom search?

SplunkTrust
SplunkTrust

Watch this video, apply what you learned and your search will be significantly faster

https://conf.splunk.com/files/2018/recordings/master-joining-datasets-without-fn1784.mp4

0 Karma
Highlighted

Re: Can you help me improve the performance of a custom search?

Contributor

Thanks for the reference. There is tons of information there.. going through the content. i'll re post updated search after making changes.

~ Abhi

Highlighted

Re: Can you help me improve the performance of a custom search?

Explorer

You can use stats commands to accomplish the same thing:

| stats values(field1) as field1, values(field2) as field2 by msg
0 Karma
Highlighted

Re: Can you help me improve the performance of a custom search?

Contributor

index=?? sourcetype=linux:audit type=CWD OR ( index= ?? sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown or (INDEX=?? sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown) )

| fields + _time, msg, auid, cwd,host, a0, a1, a2, a3]
| stats values(*) as * by msg

0 Karma
Highlighted

Re: Can you help me improve the performance of a custom search?

Esteemed Legend

Like this:

(index=YouShouldAlwaysSpecifyIndexValues AND sourcetype=linux:audit) AND 
((type="CWD")
OR
(NOT auid="4294967295" AND type="SYSCALL" AND (comm="chmod" OR comm="rm" OR comm="chown"))
OR
(NOT auid="4294967295" AND type="EXECVE" AND (a0="chmod" AND (a1="-R" AND (a2="777" OR a2="755")) OR (a1="777")) OR (a0="rm" AND a2="-r*") OR (a0="chown")))
| fields _time msg, cwd, auid, host, a0, a1, a2, a3
| stats values(*) AS * BY msg
0 Karma
Highlighted

Re: Can you help me improve the performance of a custom search?

Path Finder

Hi abhijittikekar,

Try to run below query.

sourcetype=linux:audit type=CWD
| table msg, cwd
| map
[ search sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown msg=$msg$
|eval cwd=$cwd$
| table _time, msg, auid,cwd]
|map

[ search sourcetype=linux:audit NOT auid=4294967295 msg=$msg$ type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)

| eval auid=$auid$ , cwd=$cwd$
| table + _time, msg, host, a0, a1, a2, a3,auid,cwd]]
| table _time, host, msg, auid, a0, a1, a2, a3, cwd

Thanks,
Bhavik

0 Karma