Is it possible to treat a _meta field as a non-indexed field?

[jonow]

I have seen conflicting answers on this and am confused about what should and shouldn't work.

In inputs.conf on our forwarders we configure the following (as an example):

[default]
_meta=environment::production cluster:: region::ap-southeast-2 role::logging subrole:: instance_type::t2.large

On the IX/SH we configure the following in fields.conf (snippet):

[environment]
INDEXED = False
INDEXED_VALUE = False

[cluster]
INDEXED = False
INDEXED_VALUE = False

[region]
INDEXED = False
INDEXED_VALUE = False

The docs lead me to believe that by specifying INDEXED = False, the field would not be treated as an indexed value.
But no matter what combination of values in fields.conf, I get the following behaviour in searches:

environment=production ==> 0 results
environment::production ==> all results matching that meta tag
environment="*production*" --> all results matching that meta tag, + some spurious matches of production in the raw text

The reason this is important is because non-technical users use the interface to try and drill down on search results, and find no end of frustrating where clicking on the auto-extracted interesting or selected fields on the left, selecting for example the production value for cluster, which appends cluster=production to the search, and immediately yields no results.

What I want to know is how can I get meta fields to simply be treated as a regular non-indexed field?

[martin_mueller · Accepted solution]

You want INDEXED = true, with no setting necessary for INDEXED_VALUE. Then Splunk will automatically translate field=value into a lispy of [ AND field::value ] instead of a lispy of [ AND value ] which wouldn't match your raw event.

[martin_mueller]

If that's the issue then make sure fields are set to be exported to system in the metadata/local.meta of that app:

[fields]
export=system

[jonow]

Super, I'll give that a whirl.

[martin_mueller]

The btool output looks good to me. Are you searching from a user/app context that can see that config? Check with splunk btool --debug --app=the_app --user=the_user fields list environment

[jonow]

You know what? I think that was the problem.

If in splunk I brought up the search on the default 'Search & Reporting' app - that was where I saw the results pasted previously.

If instead I brought up the search in the 'CompanyConfiguration' app, I see as you expect:

02-05-2019 10:29:37.558 INFO  UnifiedSearch - Expanded index search = environment=production
02-05-2019 10:29:37.558 INFO  UnifiedSearch - base lispy: [ AND environment::production ]

I want to thank you so much Martin, have been hitting my head against the wall for a number of weeks over this!

[martin_mueller]

No, that's not what I'd expect. The lispy for #2 is translated as if it were a search time field.

Gave it a quick re-test on 7.2.3, fields.conf:

[environment]
INDEXED=true

Searched for environment=foo, got search.log:

02-04-2019 21:55:09.007 INFO  UnifiedSearch - Expanded index search = environment=foo
02-04-2019 21:55:09.007 INFO  UnifiedSearch - base lispy: [ AND environment::foo ]

Note, there's no need to restart splunk as each search should reload the fields.conf on launch.

Where are you storing your fields.conf? Is your setting showing up when you run splunk btool --debug fields list environment?

[jonow]

Thanks Martin,

On the IX/SH machine:

/opt/splunk/bin/splunk btool --debug fields list environment
/opt/splunk/etc/apps/CompanyConfiguration/default/fields.conf [environment]
/opt/splunk/etc/apps/CompanyConfiguration/default/fields.conf INDEXED = true
/opt/splunk/etc/system/default/fields.conf                    INDEXED_VALUE = True
/opt/splunk/etc/system/default/fields.conf                    TOKENIZER =

Where /opt/splunk/etc/system/default/fields.conf contains:

cat /opt/splunk/etc/system/default/fields.conf
#   Version 7.2.3
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file contains possible attribute and value pairs for creating
# dynamic field extractions.
#

TOKENIZER =
INDEXED = False
INDEXED_VALUE = True

[source]
INDEXED = True
INDEXED_VALUE = False

[index]
INDEXED = True
INDEXED_VALUE = False

[sourcetype]
INDEXED = True
INDEXED_VALUE = False

[_sourcetype]
INDEXED = True
INDEXED_VALUE = False

[_indextime]
INDEXED = True
INDEXED_VALUE = False

[host]
INDEXED = True
INDEXED_VALUE = False

[linecount]
INDEXED = True
INDEXED_VALUE = False

[punct]
INDEXED = True
INDEXED_VALUE = False

[evtlog_id]
INDEXED = True
INDEXED_VALUE = False

[evtlog_category]
INDEXED = True
INDEXED_VALUE = False

[evtlog_severity]
INDEXED = True
INDEXED_VALUE = False

[evtlog_account]
INDEXED = True
INDEXED_VALUE = False

[evtlog_domain]
INDEXED = True
INDEXED_VALUE = False

[evtlog_sid]
INDEXED = True
INDEXED_VALUE = False

[evtlog_sid_type]
INDEXED = True
INDEXED_VALUE = False

[date_year]
INDEXED = True
INDEXED_VALUE = False

[date_month]
INDEXED = True
INDEXED_VALUE = False

[date_mday]
INDEXED = True
INDEXED_VALUE = False

[date_wday]
INDEXED = True
INDEXED_VALUE = False

[date_hour]
INDEXED = True
INDEXED_VALUE = False

[date_minute]
INDEXED = True
INDEXED_VALUE = False

[date_second]
INDEXED = True
INDEXED_VALUE = False

[date_zone]
INDEXED = True
INDEXED_VALUE = False

[timeendpos]
INDEXED = True
INDEXED_VALUE = False

[timestartpos]
INDEXED = True
INDEXED_VALUE = False

[splunk_server]
INDEXED = True
INDEXED_VALUE = False

[splunk_server_group]
INDEXED = True
INDEXED_VALUE = False

#[To]
#TOKENIZER = (\w[\w.\-]*@[\w.\-]*\w)

#[From]
#TOKENIZER = (\w[\w.\-]*@[\w.\-]*\w)

#[Cc]
#TOKENIZER = (\w[\w.\-]*@[\w.\-]*\w)

[jonow]

Thanks for your advice. So I changed fields.conf back to INDEXED = true.

[environment]
INDEXED=true

[cluster]
INDEXED=true

[region]
INDEXED=true

Restarted splunk, and I see the following:

For search environment::production --> 1,418,990 results

02-05-2019 09:24:10.759 INFO  UnifiedSearch - Expanded index search = environment::production
02-05-2019 09:24:10.759 INFO  UnifiedSearch - base lispy: [ AND environment::production ]

For search environment=production --> 136,822 results

02-05-2019 09:26:55.450 INFO  UnifiedSearch - Expanded index search = environment=production
02-05-2019 09:26:55.450 INFO  UnifiedSearch - base lispy: [ AND production ]

For search environment="*production*" --> 1,421,106 results

02-05-2019 09:28:27.763 INFO  UnifiedSearch - Expanded index search = environment="*production*"
02-05-2019 09:28:27.763 INFO  UnifiedSearch - base lispy: [ AND ]

Is this what you would expect to see? (7.2.3 by the way)

Thanks for your help.

[martin_mueller]

Open job inspector for your search, open search.log, search for lispy - or increase your log level for search jobs' info.csv in limits.conf, then your lispy will be displayed again at the top of the job inspector like it used to be the default until 6.2 or so.

[jonow]

Thanks for your advice. So I changed fields.conf back to INDEXED = true.

[environment]
INDEXED=true

[cluster]
INDEXED=true

[region]
INDEXED=true

Restarted splunk, and I see the following:

For search environment::production --> 1,418,990 results

02-05-2019 09:24:10.759 INFO  UnifiedSearch - Expanded index search = environment::production
02-05-2019 09:24:10.759 INFO  UnifiedSearch - base lispy: [ AND environment::production ]

For search environment=production --> 136,822 results

02-05-2019 09:26:55.450 INFO  UnifiedSearch - Expanded index search = environment=production
02-05-2019 09:26:55.450 INFO  UnifiedSearch - base lispy: [ AND production ]

For search environment="*production*" --> 1,421,106 results

02-05-2019 09:28:27.763 INFO  UnifiedSearch - Expanded index search = environment="*production*"
02-05-2019 09:28:27.763 INFO  UnifiedSearch - base lispy: [ AND ]

Is this what you would expect to see? (7.2.3 by the way)

Thanks for your help.

[jonow]

Thanks, we had just reverted from INDEXED = true as that seemed to exhibit the same behaviour, but I'll try to switch back and investigate further. Out of curiosity, is there a way to see the 'translated' search as you describe above?