topic Re: Search Error labels and level=Error both in same bucket in Splunk Search

Search strings and conditions together

satyajit2007 — Fri, 13 Nov 2020 05:47:16 GMT

Splunk is too powerful. But i wish the search criteria language would have been more generic something like sql 🙂

I have 3 buckets for error, warning and info for each source type.

Need help from experts

1) to add condition in error bucket like this.

level="ERROR" or log contains any of these ("Failed","Exception","Fatal")

2) also in dashboard line chart if i clicked on the error line, it should actually take me those error logs. Is it possible ?

<dashboard> <label>application Name</label> <description>Spark application logs</description> <row> <panel> <title>logs</title> <chart> <title>Streaming Error Count</title> <search> <query>index=myindex sourcetype=mysourceType1 | timechart count as total_logs count(eval(level="INFO")) as total_info count(eval(level="WARN")) as total_warn count(eval(level="ERROR")) as total_error span=1h</query> <earliest>-7d@h</earliest> <latest>now</latest> </search> <option name="charting.chart">line</option> <option name="charting.chart.showDataLabels">minmax</option> <option name="charting.drilldown">all</option> <option name="charting.layout.splitSeries">0</option> </chart> </panel> </row> </dashboard>

Re: Search Error labels and level=Error both in same bucket

richgalloway — Thu, 12 Nov 2020 20:14:38 GMT

Have you seen the Splunk SPL for SQL Users manual? https://docs.splunk.com/Documentation/Splunk/8.1.0/SearchReference/SQLtoSplunk

1) Without knowing the context, you can try

level="ERROR" OR ("Failed" OR "Exception" OR "Fatal")

2) The easiest way to do that is via the UI. Edit the dashboard and click on the hamburger (triple-dot) icon in the panel Select "Edit Drilldown". Choose "Link to search" from the dropdown then select Custom. Enter the search you want to run. Use tokens to employ fields from the dashboard.

Re: Search Error labels and level=Error both in same bucket

satyajit2007 — Thu, 12 Nov 2020 22:02:03 GMT

It gives following error.

Error in 'timechart' command: The eval expression for dynamic field 'level="ERROR" OR ("Failed" OR "Exception" OR "Fatal") ' is invalid. Error='Type checking failed. 'OR' only takes boolean arguments.'.

<query>index=myindex sourcetype=mySourceTYpe | timechart count as total_logs count(eval(level="INFO")) as total_info count(eval(level="WARN")) as total_warn count(eval(level="ERROR" OR ("Failed" OR "Exception" OR "Fatal") ) ) as total_error span=1h</query> <earliest>-7d@h</earliest>

Re: Search Error labels and level=Error both in same bucket

satyajit2007 — Fri, 13 Nov 2020 05:45:55 GMT

The OR condition is not working. will appreciate any help.

Re: Search Error labels and level=Error both in same bucket

richgalloway — Fri, 13 Nov 2020 18:16:07 GMT

I think I understand why that failed. Try this alternative.

index=myindex sourcetype=mySourceTYpe | eval error = if(level="ERROR" OR searchmatch("Failed") OR searchmatch("Exception") OR searchmatch("Fatal"), 1, 0) | timechart count as total_logs count(eval(level="INFO")) as total_info count(eval(level="WARN")) as total_warn sum(error) as total_error span=1h