Okay, I have an issue between the results from "index in ( index1 index2 )" and "index=index1 OR index=index2".
What I have is a dashboard that looks like this...
<form>
<label>SCK Dashboard Test</label>
<fieldset submitButton="false">
<input type="time" token="TimeRange" searchWhenChanged="true">
<label>Time Frame for Dashboard</label>
<default>
<earliest>-1d</earliest>
<latest>now</latest>
</default>
</input>
<input type="multiselect" token="selected_indexes" searchWhenChanged="true">
<label>Indexes (can select multiple)</label>
<default>prod</default>
<fieldForLabel>Index</fieldForLabel>
<fieldForValue>index</fieldForValue>
<search>
<query>index=* source=/var/log/tripwire.log | fields index | dedup index</query>
<earliest>-1h</earliest>
<latest>now</latest>
</search>
<delimiter> </delimiter>
</input>
</fieldset>
<row>
<panel>
<table>
<search>
<query>index IN ( $selected_indexes$ ) SEARCH THINGS HERE</query>
<earliest>$TimeRange.earliest$</earliest>
<latest>$TimeRange.latest$</latest>
</search>
<option name="count">10</option>
<option name="drilldown">none</option>
</table>
</panel>
</row>
</form>
So, it pulls a list of the indexes that are currently being updated (just chose a noisy search field values of for the multiselect) and then let's the dashboard user select which indexes to run the selected queries for.
I wish I could find the original post that I found the use of the "in" command in my dashboard, since they were doing the same type of query against a selected group of indexes. However, here's the part of the search manual that references "IN" as an operator for Multiple field-value comparisons.
The issue is that I'm seeing different results and have narrowed it down to the following two different queries, each with a version with "Index in ()" and a version with "Index="
First we try a working "index in ()" one...
index IN ( prod ) source="/var/log/tripwire.log"
"Total violations found: " AND NOT ("Total violations found: 0") ("Added:" OR "Modified:")
| rex max_match=0 field=_raw "Modified:\n\"(?P<Modified>[^\"]+)"
| rex max_match=0 field=_raw "Added:\n\"(?P<Added>[^\"]+)"
| where ( isnotnull(Modified) OR isnotnull (Added))
| fillnull value="" Added, Modified
| dedup index, host, Added, Modified
| table index, host, Added, Modified
Which results in "770 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)", so that's good.
And when I change it to "index=" the results are...
index=prod source="/var/log/tripwire.log"
"Total violations found: " AND NOT ("Total violations found: 0") ("Added:" OR "Modified:")
| rex max_match=0 field=_raw "Modified:\n\"(?P<Modified>[^\"]+)"
| rex max_match=0 field=_raw "Added:\n\"(?P<Added>[^\"]+)"
| where ( isnotnull(Modified) OR isnotnull (Added))
| fillnull value="" Added, Modified
| dedup index, host, Added, Modified
| table index, host, Added, Modified
Which results in "770 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)", and they match! So, that's working!!!!
So, I get the same results. But, in these different queries, I get different results.
index in ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g"
| eval cmd=urldecode(cmd)
| table _time, index, host, uid, pid, cwd, cmd, res
Which results in "0 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)". Hmmm, I should have seen something... Let's try it again with "index="
index=prod sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g"
| eval cmd=urldecode(cmd)
| table _time, index, host, uid, pid, cwd, cmd, res
Which results in "12 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)". Okay, now I'm confused...
So...
Either
"What is a better way of having a Dashboard user select which indexes to look at?"
or
"How can I fix the failing 'index in ()' queries?"
Thanks!
You'll need to replace in
with IN
on your first line:
index IN ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g"
| eval cmd=urldecode(cmd)
| table _time, index, host, uid, pid, cwd, cmd, res
I believe Splunk is interpreting in
as a keyword search at the moment rather than a reference to the function IN()
.
As elliotproebstel pointed out in the comments...
"in" != "IN"
So, check your capitalization!
Queries work with "IN"
You'll need to replace in
with IN
on your first line:
index IN ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g"
| eval cmd=urldecode(cmd)
| table _time, index, host, uid, pid, cwd, cmd, res
I believe Splunk is interpreting in
as a keyword search at the moment rather than a reference to the function IN()
.
Thank you! Was tearing my hair out trying to figure this out and kept not seeing the difference. LOL!
My pleasure. It's the little things that really trip us all up!
Is it just a transcription error that your non-working query looks like:
index in ( prod )
rather than
index IN ( prod )
? The two are not equivalent in Splunk.
SON OF A....
That was it! I had some of them as "IN" and others as "in" and did not notice the problem... 🙂