Splunk Search

Splunk query with multiple for subsearches

gnandini
Observer

Hi Team,

@DalJeanis 

I am trying to achieve below splunk search query to find out all the errors that are causing JVM instability.

 

 

 

for-each host : hosts (list of hosts)
	for-each jvmerrorevent(event_time, early15minofevent) : jvmerrorevents (search1 will result a table (list of event_time, even_time-15 minutes as early15minofevent))
		result+ = list of errors (search2 = search1+select list of errors occurred between early15minofevent and event_time)
return result

 

 

                            

Below query resulting error. Please suggest if any better way to achieve this. Thanks in advance.

index="123apigee"  sourcetype="msg_system_log" (host="123") "ERROR JVM OUT OF MEMORY ERROR" | eval customtime= strftime(_time, "%Y-%m-%d %I:%M:%S.%3Q") | eval 15MinEarlyofEvent= strftime(_time - 900, "%Y-%m-%d %I:%M:%S.%3Q") | table 15MinEarlyofEvent,customtime | map search="search index=123apigee sourcetype=msg_system_log host=123 ERROR | _time=strftime($customtime$, "%s")"         

Regards,

Nandini G

Labels (2)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Can you describe what your data looks like and _what_ you want to achieve.

Not how you're trying to do it because it shows that you're not used to splunk data manipulation paradigm and you're trying to "program" instead of searching.

In general, there are very few cases when you want to use map cpmmand since  it has its limitations, is heavy on the servers and can faik silently and give inconsistent results.

0 Karma

gnandini
Observer

Hi @PickleRick 

Thanks for the prompt response.

I have to list all the distinct errors within the time period of 15 minutes for each occurance of specific error(Error JVM), for each host and below is the sample error log. Please let me know how to get this distinct error list.

2021-05-10 23:48:49,754 main-SendThread(48.252.152.188:2181) WARN o.a.z.s.ZooKeeperThread - ZooKeeperThread.handleException() : Exception occurred from thread main-SendThread(xx.252.xxx.188:xxxx) java.lang.OutOfMemoryError: Java heap space

 

Thanks,

Nandini G

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The easiest solution to write is to generate time ranges with a subquery and then search for errors during those periods.

<your general search for errors> [ search <your search for zookeeper errors>
| table _time
| rename _time as latest
| eval earliest=latest-15*60 ]

But it has all the limitations of a subquery (the result limit which is probably not important here and the execution time limit and possibility of silent termination).

You could also try something with streamstats along the line of

<your search for errors>
| streamstats max(_time) as bigerrortime reset_before="like(_raw,\"%your_Zookeeper_error%\")"
| where _time-bigerrortime<=900
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...