Splunk query with multiple for subsearches

[gnandini]

Hi Team,

@DalJeanis 

I am trying to achieve below splunk search query to find out all the errors that are causing JVM instability.

for-each host : hosts (list of hosts)
	for-each jvmerrorevent(event_time, early15minofevent) : jvmerrorevents (search1 will result a table (list of event_time, even_time-15 minutes as early15minofevent))
		result+ = list of errors (search2 = search1+select list of errors occurred between early15minofevent and event_time)
return result

Below query resulting error. Please suggest if any better way to achieve this. Thanks in advance.

index="123apigee"  sourcetype="msg_system_log" (host="123") "ERROR JVM OUT OF MEMORY ERROR" | eval customtime= strftime(_time, "%Y-%m-%d %I:%M:%S.%3Q") | eval 15MinEarlyofEvent= strftime(_time - 900, "%Y-%m-%d %I:%M:%S.%3Q") | table 15MinEarlyofEvent,customtime | map search="search index=123apigee sourcetype=msg_system_log host=123 ERROR | _time=strftime($customtime$, "%s")"         

Regards,

Nandini G

[PickleRick]

Can you describe what your data looks like and _what_ you want to achieve.

Not how you're trying to do it because it shows that you're not used to splunk data manipulation paradigm and you're trying to "program" instead of searching.

In general, there are very few cases when you want to use map cpmmand since  it has its limitations, is heavy on the servers and can faik silently and give inconsistent results.

[gnandini]

Hi @PickleRick 

Thanks for the prompt response.

I have to list all the distinct errors within the time period of 15 minutes for each occurance of specific error(Error JVM), for each host and below is the sample error log. Please let me know how to get this distinct error list.

2021-05-10 23:48:49,754 main-SendThread(48.252.152.188:2181) WARN o.a.z.s.ZooKeeperThread - ZooKeeperThread.handleException() : Exception occurred from thread main-SendThread(xx.252.xxx.188:xxxx) java.lang.OutOfMemoryError: Java heap space

Thanks,

Nandini G

[PickleRick]

The easiest solution to write is to generate time ranges with a subquery and then search for errors during those periods.

<your general search for errors> [ search <your search for zookeeper errors>
   | table _time
   | rename _time as latest
   | eval earliest=latest-15*60 ]

But it has all the limitations of a subquery (the result limit which is probably not important here and the execution time limit and possibility of silent termination).

You could also try something with streamstats along the line of

<your search for errors>
| streamstats max(_time) as bigerrortime reset_before="like(_raw,\"%your_Zookeeper_error%\")"
| where _time-bigerrortime<=900