topic Re: C# API Query TImeOut in Getting Data In

C# API Query TImeOut

michaudel — Fri, 29 Mar 2013 18:49:30 GMT

Hello,
I edited the c# splunk API search example and made it into the method below. It basically creates a connection then does a search and puts all the results into a list of csv. Which i use later.

the problem i am having is if the search takes a while to run, the connection gets closed so when i to too read each line from the stream i get an error saying "The request was aborted".

is there a way to set the timeout value?

or can i create the search, wait for it to finish, then re-connect to pull down the results.

   public static List<string> SplunkSearch(string strQuery)
    {
        var cli = Command.Splunk("search");
        cli.AddRule("search", typeof(string), "search string");

        cli.Opts.Add("host", "host");
        cli.Opts.Add("port", "8089");
        cli.Opts.Add("scheme", "https");
        cli.Opts.Add("username", "uname");
        cli.Opts.Add("password", "pwd");
        cli.Opts.Add("search",strQuery);
        cli.Opts.Add("Timeout", "1000");

        var service = Service.Connect(cli.Opts);

        var jobs = service.GetJobs();
        var job = jobs.Create((string)cli.Opts["search"]);

        while (!job.IsDone)
        {
            Thread.Sleep(1000);
        }



        var outArgs = new Args
        {
            { "output_mode", "csv" },

            // Return all entries.
            { "count", "0" }
        };
        int intRun = 0;
        string columnnames = "";
        List<string> lstResults = new List<string>();
        using (var stream = job.Results(outArgs))
        {
            using (StreamReader reader = new StreamReader(stream))
            {
                while (!reader.EndOfStream)
                {
                    lstResults.Add(reader.ReadLine());
                }
            }
        }

        return lstResults;
    }

Re: C# API Query TImeOut

ywu — Fri, 29 Mar 2013 19:27:12 GMT

In your code, at the time the stream is read, search has finished already. It is a separate httpwebrequest to get the result. It is essentially what you said -- "wait for it to finish, then re-connect to pull down the results."

Do you have to more detailed info to share?

What many lines are there in the search result?

Re: C# API Query TImeOut

michaudel — Fri, 29 Mar 2013 19:59:32 GMT

Thank you for the explanation on the http request part. I think you are on to something, i think it is the size of result that is getting me. When i run a long query, but bucket into hour of day (so 24 results) it works even though the query took 5 mins. If I run another query which returns about 10K results,it aborts the connection. I am guessing i need to chunk my result set. Ask for 1K at a time or something?

Re: C# API Query TImeOut

ywu — Fri, 29 Mar 2013 20:29:57 GMT

Chunking the result set should be a good approach. There's first class support for this in Splunk REST API. Refer to 'count' and 'offset' parameters of

GET search/jobs/{search_id}/results

http://docs.splunk.com/Documentation/Splunk/5.0.2/RESTAPI/RESTsearch#search.2Fjobs.2F.7Bsearch_id.7D.2Fresults

Using C# SDK, you can supply the two parameters using Args object which is a dictionary of name and value pairs. Let me know if you have any questions on this.

Btw, I am interested in exactly where it fails in your case now. So please send me any other info about the error if any.

Re: C# API Query TImeOut

ywu — Mon, 29 Jul 2013 16:07:03 GMT

Please take a look at:

http://splunk-base.splunk.com/answers/93382/timeout-error-in-resultsreaderxml-c-sdk-example-search-program

It might be the same issue.