Need some clarification on search-time _meta field extraction.

[AndreAtNN]

I have got a question about using _meta fields in the /opt/splunkforwarder/etc/system/local/inputs.conf
of a Splunk Universal Forwarder (deployed on an AWS EC2 Instance)

In our inputs.conf of the Splunk Universal Forwarder in developement we use one _meta field, vendor, e.g.

[default] 
host = $decideOnStartup
_meta =  vendor::devops

In our organization the Indexers and Search Heads are managed by a dedicated tooling team. I did NOT requested the tooling team to update The fields.conf on the Search Head with e.g. the following statements

[vendor]
INDEXED=true;

If I execute the following search in the Search & Reporting Splunk App

index=os host="ip-10-106-152-169.aws.misin.zbi" 

I see in Selected Fields the following fields, host, source, sourcetype and vendor. The default fields host, source and sourcetype are index time extracted fields, but I don’t understand how the vendor field is extracted by Splunk because the fields.conf on the Search Head was not updated yet.

Can anybody explain why the vendor _meta field is automatically extracted when I execute the search index=os host="ip-10-106-152-169.aws.misin.zbi", without updating the fields.conf in the Search Head.

I also noticed that the following search

index=os host="ip-10-106-152-169.aws.misin.zbi” vendor=devops 

gives no result, but if I change the search to this

index=os host="ip-10-106-152-169.aws.misin.zbi” vendor::devops 

I get the same result set as the search index=os host="ip-10-106-152-169.aws.misin.zbi"

[AndreAtNN]

Hi Skali

Now it makes a little bit more sense to me.

When you mentioned that the _meta field definition in the /opt/splunkforwarder/etc/system/local/inputs.conf overrides the fields.conf definition on the Search Head I now understand why the _meta field vendor can be searched on.

Just out of curiosity, if I change the fields.conf on the Search Head with the following statements

[vendor]
INDEXED=false;   
INDEXED_VALUE=true;   

Can I perform the following search?

index=os host="ip-10-106-152-169.aws.misin.zbi” vendor=devops

According to what I read for INDEXED_VALUE at http://docs.splunk.com/Documentation/Splunk/latest/Admin/Fieldsconf it should work.
"Setting this (the INDEXED_VALUE) to true expands any search for key=value into a search of value AND key=value (since value is indexed)"

Thanks

André

[skalliger]

Should work. 🙂

[AndreAtNN]

Hi Skalli

Great, I will instruct our tooling team to change the fields.conf on the Search Head.

Thanks
André

[skalliger · Accepted solution]

Hi,

I think you're confusing things here. What you did by adding

_meta = a::b

is an override of the fields.conf setting. This tags all your events and make them searchable by defining "a::b" in your search.
Your search with vendor=devops however will only return results once you have set the "INDEXED = true" setting on your Search Head.

Ah, I knew it was here: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html
Take a look at the end of the post. That should explain it as well. 🙂

Skalli