How do I search specified fields with the same keyword list without searching the entire index?

[abour]

Assume Splunk is indexing a bunch of structured JSON data and a keyword search such as "foo" OR "bar".

Now I want to search a list of specific fields for these keywords without writing queries like field1="foo" or field1="bar" or field2="foo" or field3="bar" as the number of keywords I have is fairly large.

This is not a problem, if the events I am searching only consists of the fields that I'm interested in. However, I have more fields for which I don't want to perform this search.

I was hoping I could do something like that:
* | fields field1,field2 | search "foo" or "bar"

However, the search command is working on the entire index again and not only the extracted values of field1,field2.

Also, I need to account for not all events having field1 and field2. Some may just have field1, others may have field2, others may have both.

Is there any way to achieve what I want here in splunk?

[martin_mueller · Accepted solution]

You can build yourself a filter without specifying values twice like this:

index=your_index sourcetype=your_sourcetype
[ stats count as fields | eval fields = "field1 field2 field3" | eval values = "foo bar baz"
  | makemv fields | makemv values | mvexpand fields | mvexpand values
  | eval {fields} = values | fields - fields values]
| ...

The subsearch will evaluate to this filter expression:

( ( field1="foo" ) OR ( field1="bar" ) OR ( field1="baz" ) OR ( field2="foo" ) OR ( field2="bar" ) OR ( field2="baz" ) OR ( field3="foo" ) OR ( field3="bar" ) OR ( field3="baz" ) ) 

To facilitate reuse and maintainability, move the content of the subsearch to a macro with two parameters for the field and value list to have your search look like this:

index=your_index sourcetype=your_sourcetype [`awesome_macro("field1 field2 field3", "foo bar baz")`]

[martin_mueller]

I see... it's always a good idea to fully specify requirements first off!

You could replace the subsearch with this:

That'll generate this filter:

( ( field1="*foo*" AND foo ) OR ( field1="*bar*" AND bar ) OR ( field1="*baz*" AND baz )
OR ( field2="*foo*" AND foo ) OR ( field2="*bar*" AND bar ) OR ( field2="*baz*" AND baz )
OR ( field3="*foo*" AND foo ) OR ( field3="*bar*" AND bar ) OR ( field3="*baz*" AND baz ) ) 

That'll make sure your event contains the token foo and the field contains the string foo. There's corner cases going wrong, ie fields that contain foobar in events that also contain foo as a token elsewhere. Those would be caught by the regex-based search I suggested in your other question.

[abour]

Did you forget the subsearch part? Other than that this sounds exactly like what I want!

Thanks!

[martin_mueller]

Weird... luckily, | history | search search=*baz* still had it in my Splunk!

[ stats count as fields | eval fields = "field1 field2 field3" | eval values = "foo bar baz"
  | makemv fields | makemv values | mvexpand fields | mvexpand values
  | eval {fields} = "*".values."*" | fields - fields | rename values as search]

[abour]

This is great! Is there a way to do the same, but as a word search? It seems when I search "foo" this would consider foo to be a word, while when I use this macro, I would need to use wildcards to achieve the same. However, in that case, I would also have matches for events where "foo" is contained in a word, but not itself a word.

[martin_mueller]

There's no wildcards whatsoever in my answer, why do you need wildcards at all?

[abour]

The problem I ran into is that the field match this way is exact and (I admittedly didn't say that precisely, also because I didn't know the difference at that time) I need this to work the exact same way as search "keyword", just that keyword is searched in field1, field2,... So in order to get close to that with the above approach, I would need to use wildcards.

Is there a better way to do that? Because other than that, this macro is pretty damn sweet.

Thanks!

[MuS]

Nice one !

[wrangler2x]

Ditto. That's very cool.