Hi, after one year I think it's time to give some feedback 🙂 Kidding, I was searching for something else and found this article by accident. This information event_count=12146884, result_count=7, available_count=0, scan_count=2942980670 in the audit.log output will tell you that this search scanned almost 3 billion events on disk and just returned/used 12 million. So, 99.6% of the data read from disk was thrown away... sounds a little bit like boiling the ocean. Looking at all the "searchmatch" patterns, you might put the TERMS() like "JwtTokenUtil" or "sendEmail"before the first pipe using "() OR ()" because this will leverage the TSIDX Index (NOT the compressed raw events) and is 1000x faster than the full event scan you have done. Another thought: You seem to mix a couple of sourcetypes in one index which is fine. Using the "sourcetype=" in the OR-Clause might also help to speed up searching because Splunk Enterprise will eliminate all events which are not matching the sourcetypes you want to see. A shortened example: index=<index name> Error* ( (sourcetype=<st1> JwtTokenUtil) OR (sourcetype=<st2> SQLException) ... )
| eval ERROR_TYPE = case(searchmatch("SQLException"), "SQL Exceptions",
searchmatch("JwtTokenUtil"), "JWTToken Error",
searchmatch("JwtAuthenticationTokenFilter"), "Authentication error",
searchmatch("java.lang.OutOfMemoryError: Java heap space"), "Out of memory error",
searchmatch("sendEmail"), "Send Mail error",
searchmatch("message was not delivered"), "IMAP error",
searchmatch("sever not reponding"), "Server error",
searchmatch("Timed Out"),"Timed Out Error",
true(),"OTHER")
| timechart count by ERROR_TYPE Will only return events with "Error*" and the two tokens I defined (with the right sourcetype). This might save you from reading 99.6% of the events from disk, decompress them and do the field extraction for every single event... Depending of the structure of your data, you might even use the "append" command and split the big search into multiple shorter ones but might avoid the "searchmatch" for all your 12 million results. index=<index name> Error* (sourcetype=<st1> JwtTokenUtil) | eval ERROR_TYPE="JWTToken Error" | fields _time ERROR_TYPE |
append [ index=<index name> Error* sourcetype=<st2> SQLException | eval ERORR_TYPE="SQL Exceptions" | fields _time ERROR_TYPE ] |
append [] ... | timechart count by ERROR_TYPE Try it. You need to fix typos and maybe some syntax errors because I have written this by heart without testing in the "Search&Reporting" App. Hope this all make sense. Feel free to ask questions. Greetings, Holger
... View more