Change Default limit of 100 results using JAVA SDK

[cbauza]

Hi,

I've read several reports about how to get more than 100 results, but didn't find one addressing the Splunk Java API.

The query is something like:
search sourcetype="xxx" | table _raw

My constraints are:
- I want each event to be wrapped in a XML tag (result/field/v, as per Splunk response) to be XSL transformed later, so adding "| outputcsv" at the end of the query doesn't help for me as it wraps the whole response in a single xml tag I cannot process after.
- Also, I cannot change the Splunk Server setup

if anyone know how and what parameter to pass the code below to retrieve ALL results, please let me know:

    Job job = splunkService.getService().getJobs().create("my query");
    while (!job.isDone())
    {
        try
        {
            Thread.sleep(2000);
        }
        catch (InterruptedException e){}
        job.refresh();
    }

    println("--- Event Count = " + job.getEventCount());

    InputStream stream = job.getResults();

    BufferedInputStream bis = new BufferedInputStream(stream);
    ByteArrayOutputStream buf = new ByteArrayOutputStream();
    int result;
    try
    {
        result = bis.read();
        while(result != -1)
        {
            byte b = (byte)result;
            buf.write(b);
            result = bis.read();
        }
    }
    catch (IOException e){}

[wcolgate_splunk]

One is still limited to the splunk configured maximum results returned via the REST API. This default setting is 50,000. Even if you set the count to 0, you will only get 50,0000 events.

The correct mechanism to return events larger than the configured maximum is to issue the search request, and "page" through the results using offset and count output arguments.

In order to determine the maximum number of events returned from splunk via the REST API is as follows. Note the default configuration setting is 50,000.

    // how many events is splunk configured to return?
    Entity restApi = service.getConfs().get("limits").get("restapi");
    int maxresults = Integer.parseInt((String)restApi.get("maxresultrows"));

After the search job has been completed, you can query the number of events that match the search query:

    // how many events returned by this search?
    int eventCount = job.getEventCount();

A simple loop to page through the results is as follows:

    int getOffset = 0;

    while (getOffset < eventCount) {
        Args outputArgs = new Args();
        outputArgs.put("output_mode", "csv"); // arbitrarily choose csv output
        outputArgs.put("count", maxresults);
        outputArgs.put("offset", getOffset);

        stream = job.getResults(outputArgs);
        InputStreamReader reader = new InputStreamReader(stream);

        byte[] bytes = new byte[1024];
        while ((read = inputStream.read(bytes)) != -1) {
            out.write(bytes, 0, read);
        }            

        reader.close();

        getOffset = getOffset + maxresults;
    }

The while-loop effective returns maxresults per loop and simply adjusts the base (getOffset) to the beginning of the next page.

Good luck,

Wim

[andras_kerekes]

The above iteration can be optimized, see comment on this answer: http://splunk-base.splunk.com/answers/25411/upper-limit-for-rest-api-limitsconf-maxresultrows/25440

[Damien_Dallimor]

The instance method of the Job Class, getResults, is overloaded.

The method you are calling ,public InputStream getResults() , is simply using the endpoint defaults, which in the case of count is 100.

REST API Reference For Search Results

So you can use the method signature ,public InputStream getResults(Map args) , and supply a count value of 0, which will return all results.

Before :

InputStream stream = job.getResults();

After :

Map <String,Object> arguments = new HashMap<String,Object>();
arguments.put("count",0);
InputStream stream = job.getResults(arguments);

[yunusemreakbaba]

totally solved my problem. tx