Field from Automatic Lookup (KV Store) not usable

[ejwade]

I have a KV Store with replicate turned on, a lookup definition with WILDCARD(match_field), and an automatic configured to output a numeric lookup_field. When I run a search on the relevant source type, I see the lookup_field. However, when I search with the lookup_field (e.g., "lookup_field=1"), the search finishes quickly and doesn't return anything.

This is an example of the lookup.

mac,exception
00ABCD*,1
11EEFF*,1

This is an example of the lookup definition.

WILDCARD(mac)

This is an example of the automatic lookup.

lookup mac_addresses mac OUTPUT exception

Here is an example of a search that does not return the expected results:

index=mac_index exception=1

Here's what's really strange. It works for some events, but not others.

When I run this, I get five events

earliest=7/29/2024:00:00:00 latest=7/30/2024:00:00:00 index=logs exception=1

When I run this (adding the manual lookup), I get 109 (which is accurate).

earliest=7/29/2024:00:00:00 latest=7/30/2024:00:00:00 index=logs 
| lookup exception_lookup mac OUTPUTNEW exception
| search exception=1

Any ideas of what could cause this? Any ideas on how to troubleshoot it?

[ejwade · Accepted solution]

Figured it out. The kvstore is populated with my Python script, and I noticed that the "exception" field was written as a True/False boolean type. This would translate as 1 and 0 (number/integer) in Splunk search results, but the boolean type created issues for automatic lookups (for some reason it thought it was an indexed field). I changed this data type to integer in my script, and it fixed the problem.

[PickleRick]

Check the search job log, especially the lispy search performed.

[ejwade]

Thank you, @PickleRick. What would I look for specifically? I've reviewed the search.log extensively, and nothing seems to point to the problem.

I should also mention that I created a support case with Splunk, but I wanted to post on Splunk Community because often the community solves my problem. 🙂

[PickleRick]

I didn't have anything specific in mind. More like "that's where I'd look first and see if anything seems off".

What's the lispy search for such a lookup-value-based search?

[ejwade]

The lispy includes. Didn't see anything else remarkable.

[ OR 1 exception::1 ]

[PickleRick]

Ahhh... so you see. Splunk tries to find events which have your "1" as indexed field of the name you've given (which of course it will not have because it's a search-time lookup) or at least a verbatim "1" indexed value.

So you would probably find some events with both your lookup matching properly as well as having the value of "1" _somewhere else_ in the raw event. (these are probably these 5 results you're getting)

It's just how Splunk parses your search I'm not 100 sure if you can call it a bug. (there are other similar issues with Splunk searching data with "wrong" conditions).

EDIT: Is it a whole lispy search? I don't think so.

EDIT2: Also - before Splunk constructs the lispy search, it creates an expanded SPL search and before that it includes the knowledge-objects processing which led to this search. You can dig there a bit more to see why it does what it does.

[ejwade]

I think what's happening is the kv store replication isn't consistent. Lookups that use CSVs are, but not kvstores. I'm trying to figure out how to troubleshoot this.

@PickleRick- it only appears to be happening with a integer (number) field. String fields are working fine, and not included in the lispy. It seems like the search is assuming the integer field is indexed, but like you said, it isn't. Can you explain that more?