topic Re: Java sdk Splunk queries performance challenge in Monitoring Splunk

Java sdk Splunk queries performance challenge

1234testtest — Fri, 25 Jan 2013 12:09:08 GMT

Hi, I am using JAVA SDK of Splunk and using struts2 as the framework. I have a performance challenge.
Each time the page loads, the queries take too long time to execute (i have 6 of them in each page). So, I implemented Ajax.
But even then, the first time the page loads - takes too long (about 30-40 seconds) (for splunk server connection and then querying the saved searches) before the user sees a response. I am using non-blocking mode of executing which was included in the website (http://dev.splunk.com/view/splunk-java-sdk-how-to/SP-CAAAEKY).

Could you pl suggest a better way of improving the performance.

   Job jobSavedSearch = null;

        // Run the saved search
        try {
            jobSavedSearch = savedSearch.dispatch();
            amJob.job = jobSavedSearch;
        } catch (InterruptedException e1) {
            e1.printStackTrace();
        }

        System.out.println("Waiting for the job to finish...\n");

        // Wait for the job to finish
        while (!jobSavedSearch.isDone()) {
            try {
                Thread.sleep(500);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

Re: Java sdk Splunk queries performance challenge

Neeraj_Luthra — Fri, 25 Jan 2013 22:28:57 GMT

The connection to Splunk is most likely not adding any significant delay. My initial guess is that your saved searches are expensive and are taking quite some time to return.

Unless these queries absolutely need to return real-time data, I would recommend scheduling these saved searches on Splunk to run at regular intervals. You can do this from the Splunk UI and pick time intervals like once a minute, hour, day, month etc., whatever suits your business need.

Finally, try the following in your Java code:

SavedSearch mySavedSearch = service.getSavedSearches().get("mySavedSearchName");
Job[] jobs = mySavedSearch.history();
Job myJob = null;
try {
    myJob = (jobs.length > 0) ? jobs[0] : mySavedSearch.dispatch();
} catch (InterruptedException e) {
    e.printStackTrace();
}
System.out.println("Waiting for the job to finish for saved search - " + mySavedSearch.getName() + " ...\n");

while (!myJob.isDone()) {
    try {
        Thread.sleep(500);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}
System.out.println("Job for saved search - " + mySavedSearch.getName() + " finished.\n");

Re: Java sdk Splunk queries performance challenge

1234testtest — Sat, 26 Jan 2013 04:11:43 GMT

Thank you. It worked!

Re: Java sdk Splunk queries performance challenge

1234testtest — Tue, 29 Jan 2013 13:56:32 GMT

I have now tried with multiple queries. There are few observations.
1. I dont have scheduled search- but still am getting the results from the savedsearch with the above code. What could be the reason.
2. Could you kindly clarify if there is any splunk web equivalent for the above code - how do I check from where it is getting savedsearch.history.
Kindly help

Re: Java sdk Splunk queries performance challenge

Neeraj_Luthra — Tue, 29 Jan 2013 18:47:01 GMT

The code is set up to dispatch the saved search in the event that the history is empty. That is probably why you are still getting results.
After you login into Splunk web, select the appropriate app from the "App" drop down in the top right corner, then click on "Jobs". The URL is something like this - http://$host:$port/en-US/app/$app/job_management. Here you should see jobs for your saved searches, assuming they are scheduled.

Re: Java sdk Splunk queries performance challenge

1234testtest — Wed, 13 Feb 2013 08:04:39 GMT

Does usage of indexes help in improving the performance? Am just evaluating other options as well. Thank you.

Re: Java sdk Splunk queries performance challenge

Neeraj_Luthra — Wed, 13 Feb 2013 08:40:00 GMT

That won't help. An index in Splunk is not like one in a typical RDBMS where indexes are created for performance improvement. If you have scheduled your saved searches and are still looking for further improvement, I recommend reading through the Splunk search manual - http://docs.splunk.com/Documentation/Splunk/latest/SearchReference.

Re: Java sdk Splunk queries performance challenge

1234testtest — Wed, 13 Feb 2013 09:53:27 GMT

Thank you. I see that in 5.x version of SPlunk,there is summary indexing, which seemingly is much faster. http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Usesummaryindexing

Re: Java sdk Splunk queries performance challenge

1234testtest — Thu, 14 Feb 2013 12:17:47 GMT

Hi Neeraj,
Thank you for the guidance. Could you kindly suggest does summary indexing help.

Re: Java sdk Splunk queries performance challenge

Neeraj_Luthra — Thu, 14 Mar 2013 16:26:43 GMT

Summary Indexing can definitely help but I suggest that you read through Report Acceleration and Summary Indexing along with their use cases at http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutsummaryindexing.

Re: Java sdk Splunk queries performance challenge

1234testtest — Mon, 18 Mar 2013 06:42:20 GMT

Thank you..