Getting Data In

Setting time field in JSON object sent from Splunk-Logging-Java to Splunk HEC

althomas
Communicator

Hi all,

Currently I'm using the Splunk Logging for Java libary to send HEC messages to Splunk via logback. Currently when you send a message, it formats it as such:

{"severity":"INFO","logger":"SPLUNK","thread":"splunkHandler","message":"timestamp=2018-05-04 10:00:01.013,someotherthings=something"}

when sending a message like this:

Logger splunk = LoggerFactory.getLogger("SPLUNK");
splunk.info("timestamp=2018-05-04 10:00:01.013,someotherthings=something");

The time in this format is not extracted as it's not a top level JSON field, so it completely ignores the timestamp field provided, regardless of configuration in props.conf.

I don't see any way to provide a timestamp in a message like this. Has anyone had any experience of this?

Best regards,
Alex

1 Solution

althomas
Communicator

The solution to this is to not use Splunk-Logging-Java. There is very little configuration, very little flexibility, no ability to override the time or hostnames and sending messages to it causes the application to never shut down gracefully (the splunk logger never exits by design).
The solution is to get signed certificates and create your own HTTP client, as this one simply does not work.

View solution in original post

0 Karma

althomas
Communicator

The solution to this is to not use Splunk-Logging-Java. There is very little configuration, very little flexibility, no ability to override the time or hostnames and sending messages to it causes the application to never shut down gracefully (the splunk logger never exits by design).
The solution is to get signed certificates and create your own HTTP client, as this one simply does not work.

0 Karma

aa70627
Communicator

Too bad you've answered your own question with "I Quit". We're having similar issues.

0 Karma

althomas
Communicator

To expand on this, if you send an event to the HEC, it should look like this:
Header: {"Content-Type":"application/json", "Authorization":"Splunk My-HEC-Token"}
Body: { "time": "1526458944", "index": "main", "sourcetype": "my_sourcetype", "host": "my_host", "event": { "message":"this is an event" } }

This is because the body of the event MUST have the time, index, sourcetype, host fields otherwise they take the default from the HEC endpoint configuration. This means without the time field in the body JSON, it will take the indextime for the _time field.

The problem with using splunk-logging-java is that you piggyback on all of the functionality of logback (or log4j2), but they end up producing body JSONs like this:
{"level":"INFO","logger":"SPLUNK","thread":"splunkHandlerThread","event":{"index":"main","sourcetype":"my_sourcetype","host":"my_host","event":{"message":"this is an event"}}}

Now as you can see, the root level objects do not contain ANY of the information that Splunk needs, so it ends up displaying the root level JSON as the event itself, rather than taking that information as the metadata fields. Because it piggybacks on logback, the logger objects can't be overriden to send different events. This also means that the extractions are a bit of a terror to get working as well.

The solution is to send the JSON object that you want to send, rather than whatever logback/splunk-logging-java wants to send. The downfall of this is that you need to ensure that you have a signed certificate on the Splunk HEC receiver and import that into a java keystore in order to get it to work.

0 Karma

althomas
Communicator

Tbh it's just not the right tool for the job. If you like, I can share my implementation of the apache httpclient which works fine, not at a PC right now.

0 Karma

pmurarka
Explorer

Can you please share your piece of code.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...