I added a field to all my events so I could search for specific results, but why can't Splunk read the custom field values?

[muellernc]

I added a field cluster to all my events, so that I can search for results in a Hadoop cluster specified. I edited inputs.conf on each node for example with

[default]
_meta = cluster::Test8

and fields.conf with

[cluster]
INDEXED=true

The cluster information is displayed fine in Splunk Search:

If I try to search for a specific cluster, however, I get no results:

When I search for cluster=*Test8, the search works fine again. When I try to plot data (CPU_Load) with timechart and plot it by cluster, it messes up the diagram because it doesn't show any data points.

Thanks for your help!

[jkat54]

Try this instead:

[cluster]
INDEXED = true
INDEXED_VALUE = false

As per inputs.conf document:

INDEXED_VALUE = [true|false|<sed-cmd>|<simple-substitution-string>]
* Set this to true if the value is in the raw text of the event.  
* Set this to false if the value is not in the raw text of the event.
* Setting this to true expands any search for key=value into a search of value AND key=value 
(since value is indexed).
....
* Defaults to true.

Another approach is to change your search to this maybe?:

source="/etc/bmw.hadoop/log/cluster/performance.log" Test8

Final suggestion: Use Tags, Sourcetypes, or even lookups to identify the cluster instead of this meta jazz. Right now you're setting EVERY event to have this meta, and the docs warn against this unless absolutely required.

[vinceskahan]

Do you have an example of how to use a tag to do this on the forwarder from the shell only ? I can't find any reference of what to edit where on the universal forwarder client hosts (we deploy with puppet, so no gui there) and what if anything (hoping nothing) we need to edit (and where) on the indexer or search head boxes.....

[jkat54]

Can you just give the performance.log input a sourcetype of HADOOP_CLUSTER_TEST8 instead of messing around with _meta?