Hi guys I got a trouble on getting data to Splunk by java and I really need your help!
I followed the instructions of To add data directly to an index in http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2#add2index , using the attachWith method. Here is my code:
public static void main(String[] args) {
ServiceArgs serviceArgs = new ServiceArgs();
serviceArgs.setUsername("admin");
serviceArgs.setPassword("admin");
serviceArgs.setHost("local");
serviceArgs.setPort(8089);
Index myIndex = service.getIndexes().get("folder");
try {
myIndex.attachWith(new ReceiverBehavior() {
public void run(OutputStream stream) {
Event event = service.
DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
String date = dateFormat.format(new Date());
String eventText = date+" text=Testing!";
try {
stream.write(eventText.getBytes("UTF8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
});
} catch (IOException e) {
e.printStackTrace();
}
}
The event was successfully indexed by Splunk, but with sourcetype "http-stream-too_small". By submit method in To add data directly to an index, it provides arguments to set the values of host, source, sourcetype and so on as we want. However, I don't see any argument as such in the example of attachwith method. Is it possible to customize the host and sourcetype by this way?
Have you tried something like:
Args eventArgs = new Args();
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");
...
String eventText = date+eventArgs+" text=Testing!";
...
stream.write(eventText.getBytes("UTF8"));
In the docs it implies the methods for submit and attachWith are the same, eg
- Use the submit method to send an event over HTTP. You'll need to provide the event as a string, and can specify values to apply to the event (host, source, and sourcetype).
- Use the attach and attachWith methods to send events over a writeable socket. You can also specify the values to apply to these events (host, source, and sourcetype).
Have you tried something like:
Args eventArgs = new Args();
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");
...
String eventText = date+eventArgs+" text=Testing!";
...
stream.write(eventText.getBytes("UTF8"));
In the docs it implies the methods for submit and attachWith are the same, eg
- Use the submit method to send an event over HTTP. You'll need to provide the event as a string, and can specify values to apply to the event (host, source, and sourcetype).
- Use the attach and attachWith methods to send events over a writeable socket. You can also specify the values to apply to these events (host, source, and sourcetype).
Thank you for your answer. I tried this and found that the content of eventArgs would be upload to Splunk as a part of event. But it didn't make any different to the values of host, source and so on. And it's weird that when I set the timestamp with some certain time ( for example 2000-01-01 01:01:01, which is a little different from the example ), Splunk set the time I run the code as timestamp, not the time I set above. Any ideas about this?
The timestamp issue is probably because Splunk doesn't allow dates from that far in the past without modification.
I notice you are adding the data directly to an index. I recommend that you use an input rather than submitting to an index so you can easily change the input settings without changing your sourcecode.
Otherwise I'd say you just need to submit the args properly - Have you tried listening to the normal submit event example in the docs and seeing the format of the sting that gets sent?
Well I tried using submit method and I could set the sourcetype correctly. But the reason I would rather to use attachWith method is the performance problem as mentioned in the docs. I'm not sure what you mean by "using an input". Is there any docs about this? Thx again.
Inputs is the first section of the documentation you linked to: http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2#inputs
If you are unclear on the difference between submitting data direvtly to an index and using an index, I encourage you to drop your development work and run through the splunk tutorial. It's important that you understand the non SDK methods of getting data into splunk before writing your own application. Have you tried getting data into splunk using the universal forwarder or a syslog listener? A syslog input can handle millions of events an hour if you have enough indexers, so you could just use log4j instead of creating your own library!
Actually what I need to monitor is not the content of the logs, but the size, the modified time and the other properties of the log files. This script gets these properties and sends them to Splunk as events. Can I do this by using data input?
The official advice from splunk is to simply
Use the auditd daemon on *nix systems and monitor output from the daemon.
(See: http://docs.splunk.com/Documentation/Splunk/6.3.1/Data/Monitorchangestoyourfilesystem)
Have a look a google for auditd, of have a look on splunk answers for how to monitor file system changes. It's a very common use case so there will doubtless be a common method. It should save you a lot of grief!
Really need help!