Getting Data In
Highlighted

Add data programatically

Explorer

How do I add data programatically to splunk using the java sdk but none of the Input Kinds? Is there a way to add a java object as an input and inside splunk tell it how to interpret the object?

Tags (2)
0 Karma
Highlighted

Re: Add data programatically

Splunk Employee
Splunk Employee

Are you looking to just directly add data to an index from a java program?

For example, here is an event getting submitted over HTTP from the docs:

// Retrieve the index for the data
Index myIndex = service.getIndexes().get("test_index");

// Specify  values to apply to the event
Args eventArgs = new Args();
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");

// Submit an event over HTTP
myIndex.submit(eventArgs, "This is my event");

Java SDK docs

Splunk cannot just "index a java object". It needs to be converted into an ascii format at some point and it needs to make sense with some sort of time stamp.

View solution in original post

Highlighted

Re: Add data programatically

Explorer

Just so I'm clear, what is the difference between an index and an input?

I have this java class (let's say all it holds are strings) and I would like to take each intance of the class and add the data into splunk. To do this, you are suggesting that I first convert my object into ascii format and I shoul be able to go from there?

0 Karma
Highlighted

Re: Add data programatically

Splunk Employee
Splunk Employee

An index is where the data is stored in splunk. Later on when you access the data you would specify an index. Generally you want to group similar data in the same index. The main reasons to split off into a new index is if the data has different access requirements or if it has a different retention period.

The code I pasted was a way to programmatically push the event to Splunk using the Java SDK.

I would say though that the best practice would likely be to write an ascii representation of your object out to a file and use the splunk forwarder to send it on.

0 Karma
Highlighted

Re: Add data programatically

Splunk Employee
Splunk Employee

It depends on the use case but the reasons it is optimal to write it to a file is that forwarders can do compression, acknowledgements, you would have the data written somewhere in case something goes wrong, and you could modify it using configuration files rather than code.

0 Karma
Highlighted

Re: Add data programatically

Explorer

What would be the alternative to writing it to a file?

0 Karma
Highlighted

Re: Add data programatically

Splunk Employee
Splunk Employee

The examples in the link I gave show adding events directly to splunk using HTTP or an open socket. That would work fine but you just need to take into consideration reliability aspects if that is important to you. For example, how to handle the splunk indexer being down?

A forwarder would just remember its state and pick back up when the indexer comes back online but you would want code to handle that exception.

0 Karma

Re: Add data programatically

Explorer

Yes I understand that, but it is not pertinent at this point. The example you pasted shows the event being indexed from a log file where you specify the sourcetype property. Does the data that splunk indexes have to exist in a file of some sort or can it be in memory as a program is running?

0 Karma
Highlighted

Re: Add data programatically

Splunk Employee
Splunk Employee

Ok. The example does not actually index a log file. They are just using "accesscombined.log" as the sourcetype. Your sourcetype would be "myjavaapp" or whatever you want to call it.

The actual text of the event is the last call which is "This is my event".

0 Karma
Highlighted

Re: Add data programatically

Explorer

Ok, thanks for all the help and bearing with me but I am receiving a HTTP 404 error when executing the pasted code above. Should I ask a new question for this?

0 Karma