Splunk Search

Why am I seeing different results with dashboard/query results using "index in ( A B C )" vs. "index=A"?

scknogas
Path Finder

Okay, I have an issue between the results from "index in ( index1 index2 )" and "index=index1 OR index=index2".

What I have is a dashboard that looks like this...

<form>
  <label>SCK Dashboard Test</label>
  <fieldset submitButton="false">
    <input type="time" token="TimeRange" searchWhenChanged="true">
      <label>Time Frame for Dashboard</label>
      <default>
        <earliest>-1d</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="selected_indexes" searchWhenChanged="true">
      <label>Indexes (can select multiple)</label>
      <default>prod</default>
      <fieldForLabel>Index</fieldForLabel>
      <fieldForValue>index</fieldForValue>
      <search>
        <query>index=* source=/var/log/tripwire.log | fields index | dedup index</query>
        <earliest>-1h</earliest>
        <latest>now</latest>
      </search>
      <delimiter> </delimiter>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <search>
          <query>index IN ( $selected_indexes$ )  SEARCH THINGS HERE</query>
          <earliest>$TimeRange.earliest$</earliest>
          <latest>$TimeRange.latest$</latest>
        </search>
        <option name="count">10</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
</form>

So, it pulls a list of the indexes that are currently being updated (just chose a noisy search field values of for the multiselect) and then let's the dashboard user select which indexes to run the selected queries for.

I wish I could find the original post that I found the use of the "in" command in my dashboard, since they were doing the same type of query against a selected group of indexes. However, here's the part of the search manual that references "IN" as an operator for Multiple field-value comparisons.

http://docs.splunk.com/Documentation/Splunk/7.1.0/SearchReference/Search#Multiple_field-value_compar...

The issue is that I'm seeing different results and have narrowed it down to the following two different queries, each with a version with "Index in ()" and a version with "Index="

First we try a working "index in ()" one...

index IN ( prod )  source="/var/log/tripwire.log" 
              "Total violations found: " AND NOT ("Total violations found: 0") ("Added:" OR "Modified:") 
              | rex max_match=0 field=_raw "Modified:\n\"(?P<Modified>[^\"]+)" 
              | rex max_match=0 field=_raw "Added:\n\"(?P<Added>[^\"]+)" 
              | where ( isnotnull(Modified) OR isnotnull (Added))
              | fillnull value="" Added, Modified
              | dedup index, host, Added, Modified
              | table index, host, Added, Modified

Which results in "770 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)", so that's good.

And when I change it to "index=" the results are...

index=prod source="/var/log/tripwire.log" 
              "Total violations found: " AND NOT ("Total violations found: 0") ("Added:" OR "Modified:") 
              | rex max_match=0 field=_raw "Modified:\n\"(?P<Modified>[^\"]+)" 
              | rex max_match=0 field=_raw "Added:\n\"(?P<Added>[^\"]+)" 
              | where ( isnotnull(Modified) OR isnotnull (Added))
              | fillnull value="" Added, Modified
              | dedup index, host, Added, Modified
              | table index, host, Added, Modified

Which results in "770 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)", and they match! So, that's working!!!!

So, I get the same results. But, in these different queries, I get different results.

index in ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD 
              | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" 
              | eval cmd=urldecode(cmd) 
              | table _time, index, host, uid, pid, cwd, cmd, res

Which results in "0 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)". Hmmm, I should have seen something... Let's try it again with "index="

index=prod sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD 
              | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" 
              | eval cmd=urldecode(cmd) 
              | table _time, index, host, uid, pid, cwd, cmd, res

Which results in "12 events (4/25/18 3:00:00.000 PM to 4/25/18 4:00:00.000 PM)". Okay, now I'm confused...

So...

Either
"What is a better way of having a Dashboard user select which indexes to look at?"
or
"How can I fix the failing 'index in ()' queries?"

Thanks!

0 Karma
1 Solution

elliotproebstel
Champion

You'll need to replace in with IN on your first line:

index IN ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD 
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" 
| eval cmd=urldecode(cmd) 
| table _time, index, host, uid, pid, cwd, cmd, res

I believe Splunk is interpreting in as a keyword search at the moment rather than a reference to the function IN().

View solution in original post

scknogas
Path Finder

As elliotproebstel pointed out in the comments...

"in" != "IN"

So, check your capitalization!

Queries work with "IN"

0 Karma

elliotproebstel
Champion

You'll need to replace in with IN on your first line:

index IN ( prod ) sourcetype=linux_audit source="/var/log/audit/audit.log" type=USER_CMD 
| rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" 
| eval cmd=urldecode(cmd) 
| table _time, index, host, uid, pid, cwd, cmd, res

I believe Splunk is interpreting in as a keyword search at the moment rather than a reference to the function IN().

View solution in original post

scknogas
Path Finder

Thank you! Was tearing my hair out trying to figure this out and kept not seeing the difference. LOL!

elliotproebstel
Champion

My pleasure. It's the little things that really trip us all up!

0 Karma

elliotproebstel
Champion

Is it just a transcription error that your non-working query looks like:

index in ( prod )

rather than

index IN ( prod )

? The two are not equivalent in Splunk.

0 Karma

scknogas
Path Finder

SON OF A....

That was it! I had some of them as "IN" and others as "in" and did not notice the problem... 🙂

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!