To expand on this, if you send an event to the HEC, it should look like this:
Header: {"Content-Type":"application/json", "Authorization":"Splunk My-HEC-Token"}
Body: { "time": "1526458944", "index": "main", "sourcetype": "my_sourcetype", "host": "my_host", "event": { "message":"this is an event" } }
This is because the body of the event MUST have the time, index, sourcetype, host fields otherwise they take the default from the HEC endpoint configuration. This means without the time field in the body JSON, it will take the indextime for the _time field.
The problem with using splunk-logging-java is that you piggyback on all of the functionality of logback (or log4j2), but they end up producing body JSONs like this:
{"level":"INFO","logger":"SPLUNK","thread":"splunkHandlerThread","event":{"index":"main","sourcetype":"my_sourcetype","host":"my_host","event":{"message":"this is an event"}}}
Now as you can see, the root level objects do not contain ANY of the information that Splunk needs, so it ends up displaying the root level JSON as the event itself, rather than taking that information as the metadata fields. Because it piggybacks on logback, the logger objects can't be overriden to send different events. This also means that the extractions are a bit of a terror to get working as well.
The solution is to send the JSON object that you want to send, rather than whatever logback/splunk-logging-java wants to send. The downfall of this is that you need to ensure that you have a signed certificate on the Splunk HEC receiver and import that into a java keystore in order to get it to work.
... View more