Why am I not getting any search results using the REST API to send a request to the /search/jobs/export endpoint?

[sk4l]

Hi,

I am trying to do a real-time Splunk search using the REST API. The endpoint I am sending a request to is services/search/jobs/export and if I understand the documentation correctly, I should be getting a stream of events that match my search. My problem is that I am not receiving ANY data back. I am 100% sure the events are happening and getting into Splunk, because I can see them through Splunk Web.

More info about the request I am making:
- earliest_time and latest_time are set to rt
- search_mode is set to realtime. I tried every possible value and still I couldn't get anything back.

The only way I get some data back is if I set the auto_cancel parameter to some value. After the search cancels, I get the accumulated results back. What I don't understand is why am I not getting the data streamed back? What am I missing?

I would be really grateful if someone points me in the right direction. Thanks!

[gblock_splunk]

How are you making the request? Have you tested with curl?

[sk4l]

Yes, cURL gives me results delayed with around 30-40 seconds. Node.js doesn't give me ANY results, which is weird, considering I am using the standard request from the documentation. I've re-checked my code 10 times.

function startRealTimeSearch(searchQuery, callback) {

  // I've replaced the data with placeholders
  const options = {
    hostname: 'splunk_instance_address',
    port: 'port_number',
    path: '/services/search/jobs/export',
    method: 'POST',

    headers: {
      'Authorization': 'Basic base64_encoded_data'
    }
  };

  // POST body
  let search_body = qs.stringify({
    'search': searchQuery,
    'earliest_time': 'rt',
    'latest_time': 'rt',
    'output_mode': 'json'
  });


  // HTTPS POST request to Splunk that starts the RT search
  let searchRequestToSplunk = https.request(options, function onResponse(res) {

      let receivedData = '';

      // The stream of events should be received here.
      res.on('data', (chunk) => {
        receivedData += chunk;
        console.log(`Received: ${receivedData}`);
      });

      res.on('end', () => {
        console.log(`Stream ended.`);
      });


  });
  // Send the POST request to the Splunk API
  searchRequestToSplunk.write(search_body);
  searchRequestToSplunk.end();

  // Receive request errors here
  searchRequestToSplunk.on('error', (err) => {
    console.error(err.message);
  });

}

[gblock_splunk]

OK, since you are using node, I did a blog post on export from node. The post is here. I use a slightly different approach using the request module and pipes.

Can you try it out and see if it works, and then maybe tweak it to your use case?

[MuS]

Hi sk4l,

First thing that should not work the way you think is the rest command; rest cannot be used in real-time searches and give this error:

 Error in 'rest' command: This command is not supported in a real-time search

and the other thing is your used rest end point /search/jobs/export. I'm not sure if this is correct because the docs http://docs.splunk.com/Documentation/Splunk/6.4.2/RESTREF/RESTsearch#search.2Fjobs.2Fexport mention /services/search/jobs/export

Hope this helps ...

cheers, MuS

[sk4l]

Hi, thanks for the response!

I didn't mean a rest command, but the REST-ful API. I am also using the endpoint you suggested.

[MuS]

Hmmm, still the rest command uses the REST API to show the information and I never heard / saw a REST API call being made in real-time.....maybe you want to elaborate your use cases a bit more?

[sk4l]

I want to be able to monitor the progress of some infrastructure commands. The command's chain of activated services produces lots of Splunk logs, which I want to see as they arrive. That's why I want to start a real-time search, activate the command, and have the results arrive at an endpoint of my choice.

The big problem here is that the logs are visible in Splunk Web almost instantly, but don't get streamed to my real-time search for at least another 30 seconds.