How to configure Splunk to extract key value pairs with JSON log data from Http Event Collector?

[ekst_andwii]

We have started using the Http Event Collector (HEC) for logging directly from our Java apps. HEC takes data in JSON format but we have a lot of legacy code that logs key/value pairs and some searches/dashboards that utilize these. Data logged to HEC is by default indexed as the _json sourcetype and I have tried to configure this with KV_MODE=auto (for key/value) and json (for json-format) but none of these seem to trigger Splunk to index key/values. Example log statement:

logger.info("corrId=11-1111-566 aa=88");

However, I have not been able to search on the keys, e.g. search aa=88

The event looks like this:

Raw format: {"severity":"INFO","logger":"splunk.logger","thread":"main","message":"corrId=11-1111-566 aa=88"}

Any ideas?

[richgalloway]

Consider using transforms.

Put this is your props.conf stanza

TRANSFORMS-messageFields = messageFields

Then create this stanza in your transforms.conf file. It should create a field for each word on the left of each '=' in the event.

[messageFields]
REGEX = ([a-z]+)=([\w\-]+)
FORMAT = $1::$2

[ekst_andwii]

Thanks for the reply @richgalloway, however, I can not get this to work. First of all, I think there's a typo, should it not be TRANSFORMS-messageFields? I've added this to props.conf and transforms.conf but Splunk still does not index fields in the message element.

Also, I'm a bit skeptical to adding index-time extractions in the first place as this could have a big performance impact.

[richgalloway]

Yes, it should be TRANSFORMS. I've corrected my answer.
Once you change the config files you need to restart Splunk and then re-index the data as existing events will not be affected.
To do the same thing at search time try this:

<your base search> | rex field=message "corrId=(?<corrId>[^\s]+)\saa=(?<aa>[.*]?)" | ...

[ekst_andwii]

Still cannot get index-time extractions to work, search-time works.

[richgalloway]

Given your concern about the performance of index-time extractions, search-time working is good, right?

[ekst_andwii]

Yes, but we want this to happen automatically through KV_MODE in props.conf. That's what we had before, without having to use rex and naming each field at search time.