In Splunk hec, what should you check if you cannot search fields from http event collector in SHC?

larshaugan — Tue, 19 Jun 2018 16:41:31 GMT

With Splunk HEC it is possible to send a HTTP POST with Json payload to services/collector/event. This supports the fields Json key, that enables you to add additional data to an event that is not present in the _raw (or event) data.

Given the following json payload you should be able to search search index=* k8s_node="node01*":

{ "event": "datadata", "fields": {"k8s_node":"node01.domain.tld", "k8s_namespace","namespacename"}}

However when searching for fields that are not present in the _raw data, the search will not give you a match, and you will not be able to match searches to the items in the fields key.

What could be done to resolve this?

Re: In Splunk hec, what should you check if you cannot search fields from http event collector in SHC?

larshaugan — Tue, 19 Jun 2018 18:31:37 GMT

There are additional configuration that is needed to use fields in some cases. The data is automatically indexed (without the need for props and transformations on the HF/peer), but to be able to utilize the data configuration is needed at the SHC.

With a configuration that is incomplete it will be possible to view the field while searching, but not possible to search with the field specified. The following steps helped resolve the issue:

First check that the events actually are indexed: |tstast count where index=* k8s_node="node01.domain.tld". If count > 0 the field is indexed (and you have access to it). search index=* k8s_node="node01.domain.tld" would still show 0 events.

Then on the searchhead cluster you need to specify that the field you are looking for is an indexed field. This is done in fields.conf.

[k8s_node]
INDEXED = true

If you are using a shdeployer (or another app on the searchhead for configuration rather than system) you must ensure that metadata/default.meta contains the following.

[fields]
export = system

Additional search time extraction will need to be specified in props.conf with settings like KV_MODE=auto depending on your source/sourcetypes

This will allow you to search for fields with field=value rather than field::value like described in https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/IFXandHEC#Search_for_index-extracted_fields

topic Re: In Splunk hec, what should you check if you cannot search fields from http event collector in SHC? in Getting Data In

In Splunk hec, what should you check if you cannot search fields from http event collector in SHC?

Re: In Splunk hec, what should you check if you cannot search fields from http event collector in SHC?