topic Why is search command TERM not working in one system but does in another? in Splunk Search

Why is search command TERM not working in one system but does in another?

sansay — Fri, 03 May 2019 21:02:29 GMT

We just found out that the search command TERM does NOT work when used on extracted fields in one of our Splunk Enterprise environments. But it does in another. The 2 systems have some different configurations and handle a different amount of data, but they are both Enterprise and as such, I would expect search commands to work the same way.

For instance:
index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
will return nothing in system 1 while it returns the correct events in system 2.

However using CASE works in both:
index=_internal host=indexer CASE(group=thruput) CASE(name=thruput)
returns the correct events in both system 1 and system 2

Why does TERM work in system 2 but not in system 1
What method can I use to identify the root cause
Why do we need to use TERM or CASE if the fields are extracted? I thought that looking for group=thruput name=thruput would work faster since these are extracted, and therefore indexed, fields.

The following section shows the 3 main queries I ran over a 24 hours time range and results.

SYSTEM 1 (AWS)

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
0 events

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=idx CASE(group=thruput) CASE(name=thruput)
found: 354,372 events
runtime: 18.818 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=idx group=thruput name=thruput
found: 351,096 events
runtime:28.892 seconds

SYSTEM 2 (Data Center)

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer TERM(group=thruput) TERM(name=thruput)
found: 86,893 events
runtime: 2.183 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer CASE(group=thruput) CASE(name=thruput)
found: 86,893 events
runtime: 2.187 seconds

earliest=5/2/2019:0:0:0 latest=5/3/2019:0:0:0 index=_internal host=indexer group=thruput name=thruput
found: 86,397 events
runtime: 6.2 seconds

Re: Why is search command TERM not working in one system but does in another?

koshyk — Fri, 03 May 2019 21:24:04 GMT

you may need to post the actual event message in System1 and System2 too

Re: Why is search command TERM not working in one system but does in another?

sansay — Wed, 30 Sep 2020 00:20:54 GMT

The events are pretty standard logs written by splunkd. So their content shouldn't be a determinant factor whether they are found or not by Splunk.

Anyway, here is a sample event:
"05-02-2019 23:59:56.202 +0000 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.92954826907947, instantaneous_eps=6.484249982611887, average_kbps=4004.078166704291, total_k_processed=27241737783, kb=59.8125, ev=201, load_average=1.8"
Source=/opt/splunk/var/log/splunk/metrics.log
sourcetype=splunkd

Re: Why is search command TERM not working in one system but does in another?

acharlieh — Sun, 05 May 2019 02:44:14 GMT

This might be an anonymization error, or is your search that's returning 0 events on "SYSTEM 1 (AWS)" searching for the correct host?

I note that your TERM search for SYSTEM 1 is using host=indexer like the queries from SYSTEM 2, whereas the other queries in SYSTEM 1 are using host=idx (Obviously if you're trying to search for logs of a host that Splunk is neither collecting, nor distributing search to, you're going to get back no results)

The other question I have is are the two systems the exact same versions of Splunk Enterprise (we're not looking for a bug in one version or the other), and the configuration differences are not around segmentation and metrics for these logs? (although I haven't figured out exactly to make CASE work but TERM not yet, other than searching for the wrong host...)

Re: Why is search command TERM not working in one system but does in another?

sansay — Sun, 05 May 2019 21:12:02 GMT

Of course, it has nothing to do with the indexer name. I see that I posted one version of the query in the wrong system, because I just quickly copy pasted, not remembering that they have different indexer names.
Segmentation? I have no idea how that would impact the functionality of this command. Please elaborate.
Thank you.

Ahh and yes, both systems use the same version 7.2.4.2

Re: Why is search command TERM not working in one system but does in another?

maciep — Mon, 06 May 2019 15:32:32 GMT

I'd probably start with segmenters.conf to be sure they are the same between systems. That's the file that defines the rules for major/minor breakers that ultimately create the terms that are indexed with your events. I think TERM is essentially matching your term filter against the terms found using these breakers.

This session might help (more than I will)
https://conf.splunk.com/files/2016/recordings/fields-indexed-tokens-and-you.mp4