In Splunk hec, what should you check if you cannot search fields from http event collector in SHC?

[larshaugan]

With Splunk HEC it is possible to send a HTTP POST with Json payload to services/collector/event. This supports the fields Json key, that enables you to add additional data to an event that is not present in the _raw (or event) data.

Given the following json payload you should be able to search search index=* k8s_node="node01*":

{ "event": "datadata", "fields": {"k8s_node":"node01.domain.tld", "k8s_namespace","namespacename"}}

However when searching for fields that are not present in the _raw data, the search will not give you a match, and you will not be able to match searches to the items in the fields key.

What could be done to resolve this?

[larshaugan · Accepted solution]

There are additional configuration that is needed to use fields in some cases. The data is automatically indexed (without the need for props and transformations on the HF/peer), but to be able to utilize the data configuration is needed at the SHC.

With a configuration that is incomplete it will be possible to view the field while searching, but not possible to search with the field specified. The following steps helped resolve the issue:

First check that the events actually are indexed: |tstast count where index=* k8s_node="node01.domain.tld". If count > 0 the field is indexed (and you have access to it). search index=* k8s_node="node01.domain.tld" would still show 0 events.

Then on the searchhead cluster you need to specify that the field you are looking for is an indexed field. This is done in fields.conf.

[k8s_node]
INDEXED = true

If you are using a shdeployer (or another app on the searchhead for configuration rather than system) you must ensure that metadata/default.meta contains the following.

[fields]
export = system

Additional search time extraction will need to be specified in props.conf with settings like KV_MODE=auto depending on your source/sourcetypes

This will allow you to search for fields with field=value rather than field::value like described in https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/IFXandHEC#Search_for_index-extracted_fields