We have started using the Http Event Collector (HEC) for logging directly from our Java apps. HEC takes data in JSON format but we have a lot of legacy code that logs key/value pairs and some searches/dashboards that utilize these. Data logged to HEC is by default indexed as the _json sourcetype and I have tried to configure this with KV_MODE=auto (for key/value) and json (for json-format) but none of these seem to trigger Splunk to index key/values. Example log statement:
logger.info("corrId=11-1111-566 aa=88");
However, I have not been able to search on the keys, e.g. search aa=88
The event looks like this:
Raw format: {"severity":"INFO","logger":"splunk.logger","thread":"main","message":"corrId=11-1111-566 aa=88"}
Any ideas?
Consider using transforms.
Put this is your props.conf stanza
TRANSFORMS-messageFields = messageFields
Then create this stanza in your transforms.conf file. It should create a field for each word on the left of each '=' in the event.
[messageFields]
REGEX = ([a-z]+)=([\w\-]+)
FORMAT = $1::$2
Thanks for the reply @richgalloway, however, I can not get this to work. First of all, I think there's a typo, should it not be TRANSFORMS-messageFields? I've added this to props.conf and transforms.conf but Splunk still does not index fields in the message element.
Also, I'm a bit skeptical to adding index-time extractions in the first place as this could have a big performance impact.
Yes, it should be TRANSFORMS
. I've corrected my answer.
Once you change the config files you need to restart Splunk and then re-index the data as existing events will not be affected.
To do the same thing at search time try this:
<your base search> | rex field=message "corrId=(?<corrId>[^\s]+)\saa=(?<aa>[.*]?)" | ...
Still cannot get index-time extractions to work, search-time works.
Given your concern about the performance of index-time extractions, search-time working is good, right?
Yes, but we want this to happen automatically through KV_MODE in props.conf. That's what we had before, without having to use rex and naming each field at search time.