Splunk Search

correlating events within a given time window

sushil909
Explorer

Hi,
I have a file containing events in the format given below
Time system parameter value

12jun2013:14:00:00 system1 memoryusage 12345221233
12jun2013:14:00:00 system1 userprocesses 129

I have created my own custom source type. I am able to make splunk parse the data according to the fields.
I want to be able to run queries like
whenever memoryusage > 10000000 show the userprocesses within that time window(1sec)

How can i run this kind of query ?

Tags (1)

jameshgibson
Path Finder

Try something like:

sourcetype=whatever (memoryusage OR userprocesses) | rex field=_raw ".*memoryusage (?P< memoryusage>[0-9]+)" | rex field=_raw ".*userprocess (?P< userprocess>[0-9]+)"  | transaction _time maxspan=1s | search memoryusage>10000000 |  table memoryusage userprocess

formatting is a bit messed up so the < word> should really be <word>

sushil909
Explorer

Removing regex from James answer since I had field extractions in place:
sourcetype=whatever (memoryusage OR userprocesses) | eval memoryusage=case(parameter=="memoryusage",value)|eval userprocess=case(parameter=="userprocess",value)|transaction _time maxspan=1s | search memoryusage>10000000 | table memoryusage userprocess

0 Karma

jameshgibson
Path Finder

if the log file contained paramater=value then Splunk should parse out the fields automagically for you yes. If you can't change the log format then it may be worth setting up some field extractions.

0 Karma

sushil909
Explorer

Thanks James...the solution works perfectly
only issue i see is that having multiple regular expression slows down the search. Since i already know the format of the event, the individual field, isn't there a way to avoid regex. For example a way to specify that whenever the parameter="memoryusage" memoryuse=value

0 Karma

linu1988
Champion

I tried it on sample event, i didn't get it. Let the actual person give it a go 🙂

0 Karma

jameshgibson
Path Finder

the transaction command joins all the events for a particular second in a single event. So you should have 1 memoryusage and several userprocess per event, so no need to use joins/stats/etc. Give it a go anyway 🙂

0 Karma

linu1988
Champion

Hey James,
Great analysis, but without join how the Table will show different value belonging to separate events? I faced the same in my query to get the userprocess value..

0 Karma

linu1988
Champion

sourcetype=_Name "userprocesses"|eval a=strptime(_time,"%d:%m:%y %H:%M:%S")|fields a [|search index=main sourcetype=_Name "memoryusage"|where MCount >10000000|eval a=strptime(_time,"%d:%m:%y %H:%M:%S")|fields a]

the above query will give you the event containing the Process at the same time when the memory usage is high. You can also remove the %s parameter if you are okay with comparing minute wise.

0 Karma

sushil909
Explorer

this would only display the 'memoryusage' events.Based on this condition i want to display the userprocesses events that may have occured some time prior (eg with 5 secs) to the memoryusage event

0 Karma

linu1988
Champion

Extract the data into a field named Mem_count. You can use rex / UI field extraction. Thanks.

Sourcetype=_Name "memoryusage"| where Mem_count>10000000

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...