topic Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint? in Splunk Search

Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

sk4l — Sun, 07 Aug 2016 19:58:48 GMT

Hi,

I am trying to do a real-time Splunk search using the REST API. The endpoint I am sending a request to is services/search/jobs/export and if I understand the documentation correctly, I should be getting a stream of events that match my search. My problem is that I am not receiving ANY data back. I am 100% sure the events are happening and getting into Splunk, because I can see them through Splunk Web.

More info about the request I am making:
- earliest_time and latest_time are set to rt
- search_mode is set to realtime. I tried every possible value and still I couldn't get anything back.

The only way I get some data back is if I set the auto_cancel parameter to some value. After the search cancels, I get the accumulated results back. What I don't understand is why am I not getting the data streamed back? What am I missing?

I would be really grateful if someone points me in the right direction. Thanks!

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

MuS — Mon, 08 Aug 2016 00:06:06 GMT

Hi sk4l,

First thing that should not work the way you think is the rest command; rest cannot be used in real-time searches and give this error:

 Error in 'rest' command: This command is not supported in a real-time search

and the other thing is your used rest end point /search/jobs/export. I'm not sure if this is correct because the docs http://docs.splunk.com/Documentation/Splunk/6.4.2/RESTREF/RESTsearch#search.2Fjobs.2Fexport mention /services/search/jobs/export

Hope this helps ...

cheers, MuS

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

sk4l — Mon, 08 Aug 2016 07:17:07 GMT

Hi, thanks for the response!

I didn't mean a rest command, but the REST-ful API. I am also using the endpoint you suggested.

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

MuS — Mon, 08 Aug 2016 20:21:07 GMT

Hmmm, still the rest command uses the REST API to show the information and I never heard / saw a REST API call being made in real-time.....maybe you want to elaborate your use cases a bit more?

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

sk4l — Tue, 09 Aug 2016 14:55:42 GMT

I want to be able to monitor the progress of some infrastructure commands. The command's chain of activated services produces lots of Splunk logs, which I want to see as they arrive. That's why I want to start a real-time search, activate the command, and have the results arrive at an endpoint of my choice.

The big problem here is that the logs are visible in Splunk Web almost instantly, but don't get streamed to my real-time search for at least another 30 seconds.

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

gblock_splunk — Thu, 11 Aug 2016 16:29:52 GMT

How are you making the request? Have you tested with curl?

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

sk4l — Fri, 12 Aug 2016 13:06:41 GMT

Yes, cURL gives me results delayed with around 30-40 seconds. Node.js doesn't give me ANY results, which is weird, considering I am using the standard request from the documentation. I've re-checked my code 10 times.

function startRealTimeSearch(searchQuery, callback) {

  // I've replaced the data with placeholders
  const options = {
    hostname: 'splunk_instance_address',
    port: 'port_number',
    path: '/services/search/jobs/export',
    method: 'POST',

    headers: {
      'Authorization': 'Basic base64_encoded_data'
    }
  };

  // POST body
  let search_body = qs.stringify({
    'search': searchQuery,
    'earliest_time': 'rt',
    'latest_time': 'rt',
    'output_mode': 'json'
  });


  // HTTPS POST request to Splunk that starts the RT search
  let searchRequestToSplunk = https.request(options, function onResponse(res) {

      let receivedData = '';

      // The stream of events should be received here.
      res.on('data', (chunk) => {
        receivedData += chunk;
        console.log(`Received: ${receivedData}`);
      });

      res.on('end', () => {
        console.log(`Stream ended.`);
      });


  });
  // Send the POST request to the Splunk API
  searchRequestToSplunk.write(search_body);
  searchRequestToSplunk.end();

  // Receive request errors here
  searchRequestToSplunk.on('error', (err) => {
    console.error(err.message);
  });

}

Re: Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

gblock_splunk — Fri, 12 Aug 2016 19:34:20 GMT

OK, since you are using node, I did a blog post on export from node. The post is here. I use a slightly different approach using the request module and pipes.

Can you try it out and see if it works, and then maybe tweak it to your use case?