Getting Data In

How to customize the sourcetype in java?

LuiesCui
Communicator

Hi guys I got a trouble on getting data to Splunk by java and I really need your help!
I followed the instructions of To add data directly to an index in http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2#add2index , using the attachWith method. Here is my code:

public static void main(String[] args) {
    ServiceArgs serviceArgs = new ServiceArgs();
    serviceArgs.setUsername("admin");
    serviceArgs.setPassword("admin");
    serviceArgs.setHost("local");
    serviceArgs.setPort(8089);
    Index myIndex = service.getIndexes().get("folder");
    try {
        myIndex.attachWith(new ReceiverBehavior() {
            public void run(OutputStream stream) {
                Event event = service.
                DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
                String date = dateFormat.format(new Date());
                String eventText = date+" text=Testing!";
                try {
                    stream.write(eventText.getBytes("UTF8"));
                } catch (UnsupportedEncodingException e) {
                    e.printStackTrace();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        });
    } catch (IOException e) {
        e.printStackTrace();
    }
}

The event was successfully indexed by Splunk, but with sourcetype "http-stream-too_small". By submit method in To add data directly to an index, it provides arguments to set the values of host, source, sourcetype and so on as we want. However, I don't see any argument as such in the example of attachwith method. Is it possible to customize the host and sourcetype by this way?

0 Karma
1 Solution

jplumsdaine22
Influencer

Have you tried something like:

Args eventArgs = new Args(); 
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");
...
String eventText = date+eventArgs+" text=Testing!";
...
stream.write(eventText.getBytes("UTF8"));

In the docs it implies the methods for submit and attachWith are the same, eg

  • Use the submit method to send an event over HTTP. You'll need to provide the event as a string, and can specify values to apply to the event (host, source, and sourcetype).
  • Use the attach and attachWith methods to send events over a writeable socket. You can also specify the values to apply to these events (host, source, and sourcetype).

View solution in original post

0 Karma

jplumsdaine22
Influencer

Have you tried something like:

Args eventArgs = new Args(); 
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");
...
String eventText = date+eventArgs+" text=Testing!";
...
stream.write(eventText.getBytes("UTF8"));

In the docs it implies the methods for submit and attachWith are the same, eg

  • Use the submit method to send an event over HTTP. You'll need to provide the event as a string, and can specify values to apply to the event (host, source, and sourcetype).
  • Use the attach and attachWith methods to send events over a writeable socket. You can also specify the values to apply to these events (host, source, and sourcetype).
0 Karma

LuiesCui
Communicator

Thank you for your answer. I tried this and found that the content of eventArgs would be upload to Splunk as a part of event. But it didn't make any different to the values of host, source and so on. And it's weird that when I set the timestamp with some certain time ( for example 2000-01-01 01:01:01, which is a little different from the example ), Splunk set the time I run the code as timestamp, not the time I set above. Any ideas about this?

0 Karma

jplumsdaine22
Influencer

The timestamp issue is probably because Splunk doesn't allow dates from that far in the past without modification.

I notice you are adding the data directly to an index. I recommend that you use an input rather than submitting to an index so you can easily change the input settings without changing your sourcecode.

Otherwise I'd say you just need to submit the args properly - Have you tried listening to the normal submit event example in the docs and seeing the format of the sting that gets sent?

0 Karma

LuiesCui
Communicator

Well I tried using submit method and I could set the sourcetype correctly. But the reason I would rather to use attachWith method is the performance problem as mentioned in the docs. I'm not sure what you mean by "using an input". Is there any docs about this? Thx again.

0 Karma

jplumsdaine22
Influencer

Inputs is the first section of the documentation you linked to: http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2#inputs

If you are unclear on the difference between submitting data direvtly to an index and using an index, I encourage you to drop your development work and run through the splunk tutorial. It's important that you understand the non SDK methods of getting data into splunk before writing your own application. Have you tried getting data into splunk using the universal forwarder or a syslog listener? A syslog input can handle millions of events an hour if you have enough indexers, so you could just use log4j instead of creating your own library!

0 Karma

LuiesCui
Communicator

Actually what I need to monitor is not the content of the logs, but the size, the modified time and the other properties of the log files. This script gets these properties and sends them to Splunk as events. Can I do this by using data input?

0 Karma

jplumsdaine22
Influencer

The official advice from splunk is to simply

Use the auditd daemon on *nix systems and monitor output from the daemon.
(See: http://docs.splunk.com/Documentation/Splunk/6.3.1/Data/Monitorchangestoyourfilesystem)

Have a look a google for auditd, of have a look on splunk answers for how to monitor file system changes. It's a very common use case so there will doubtless be a common method. It should save you a lot of grief!

0 Karma

LuiesCui
Communicator

Really need help!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...