Setting time field in JSON object sent from Splunk-Logging-Java to Splunk HEC

[althomas]

Hi all,

Currently I'm using the Splunk Logging for Java libary to send HEC messages to Splunk via logback. Currently when you send a message, it formats it as such:

{"severity":"INFO","logger":"SPLUNK","thread":"splunkHandler","message":"timestamp=2018-05-04 10:00:01.013,someotherthings=something"}

when sending a message like this:

Logger splunk = LoggerFactory.getLogger("SPLUNK");
splunk.info("timestamp=2018-05-04 10:00:01.013,someotherthings=something");

The time in this format is not extracted as it's not a top level JSON field, so it completely ignores the timestamp field provided, regardless of configuration in props.conf.

I don't see any way to provide a timestamp in a message like this. Has anyone had any experience of this?

Best regards,
Alex

[althomas · Accepted solution]

The solution to this is to not use Splunk-Logging-Java. There is very little configuration, very little flexibility, no ability to override the time or hostnames and sending messages to it causes the application to never shut down gracefully (the splunk logger never exits by design).
The solution is to get signed certificates and create your own HTTP client, as this one simply does not work.

[aa70627]

Too bad you've answered your own question with "I Quit". We're having similar issues.

[althomas]

To expand on this, if you send an event to the HEC, it should look like this:
Header: {"Content-Type":"application/json", "Authorization":"Splunk My-HEC-Token"}
Body: { "time": "1526458944", "index": "main", "sourcetype": "my_sourcetype", "host": "my_host", "event": { "message":"this is an event" } }

This is because the body of the event MUST have the time, index, sourcetype, host fields otherwise they take the default from the HEC endpoint configuration. This means without the time field in the body JSON, it will take the indextime for the _time field.

The problem with using splunk-logging-java is that you piggyback on all of the functionality of logback (or log4j2), but they end up producing body JSONs like this:
{"level":"INFO","logger":"SPLUNK","thread":"splunkHandlerThread","event":{"index":"main","sourcetype":"my_sourcetype","host":"my_host","event":{"message":"this is an event"}}}

Now as you can see, the root level objects do not contain ANY of the information that Splunk needs, so it ends up displaying the root level JSON as the event itself, rather than taking that information as the metadata fields. Because it piggybacks on logback, the logger objects can't be overriden to send different events. This also means that the extractions are a bit of a terror to get working as well.

The solution is to send the JSON object that you want to send, rather than whatever logback/splunk-logging-java wants to send. The downfall of this is that you need to ensure that you have a signed certificate on the Splunk HEC receiver and import that into a java keystore in order to get it to work.

[althomas]

Tbh it's just not the right tool for the job. If you like, I can share my implementation of the apache httpclient which works fine, not at a PC right now.

[pmurarka]

Can you please share your piece of code.