topic Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector? in Getting Data In

How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

ekst_andwii — Tue, 29 Sep 2020 16:04:35 GMT

We have started using the Http Event Collector (HEC) for logging directly from our Java apps. HEC takes data in JSON format but we have a lot of legacy code that logs key/value pairs and some searches/dashboards that utilize these. Data logged to HEC is by default indexed as the _json sourcetype and I have tried to configure this with KV_MODE=auto (for key/value) and json (for json-format) but none of these seem to trigger Splunk to index key/values. Example log statement:

logger.info("corrId=11-1111-566 aa=88");

However, I have not been able to search on the keys, e.g. search aa=88

The event looks like this:

Raw format: {"severity":"INFO","logger":"splunk.logger","thread":"main","message":"corrId=11-1111-566 aa=88"}

Any ideas?

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

richgalloway — Tue, 03 Oct 2017 13:15:09 GMT

Consider using transforms.

Put this is your props.conf stanza

TRANSFORMS-messageFields = messageFields

Then create this stanza in your transforms.conf file. It should create a field for each word on the left of each '=' in the event.

[messageFields]
REGEX = ([a-z]+)=([\w\-]+)
FORMAT = $1::$2

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

ekst_andwii — Tue, 03 Oct 2017 14:04:34 GMT

Thanks for the reply @richgalloway, however, I can not get this to work. First of all, I think there's a typo, should it not be TRANSFORMS-messageFields? I've added this to props.conf and transforms.conf but Splunk still does not index fields in the message element.

Also, I'm a bit skeptical to adding index-time extractions in the first place as this could have a big performance impact.

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

richgalloway — Tue, 03 Oct 2017 16:27:15 GMT

Yes, it should be TRANSFORMS. I've corrected my answer.
Once you change the config files you need to restart Splunk and then re-index the data as existing events will not be affected.
To do the same thing at search time try this:

<your base search> | rex field=message "corrId=(?<corrId>[^\s]+)\saa=(?<aa>[.*]?)" | ...

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

ekst_andwii — Wed, 04 Oct 2017 10:42:43 GMT

Still cannot get index-time extractions to work, search-time works.

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

richgalloway — Mon, 09 Oct 2017 18:42:02 GMT

Given your concern about the performance of index-time extractions, search-time working is good, right?

Re: How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

ekst_andwii — Mon, 16 Oct 2017 09:17:09 GMT

Yes, but we want this to happen automatically through KV_MODE in props.conf. That's what we had before, without having to use rex and naming each field at search time.