Extracting Data from Splunk for reporting in an external system

[DanielFordWA]

I would like to extract the search results from searches performed by Splunk on a regular basis.

This data will then be added into a database which our reporting tool sits on top of (Qlikview)

I have had a look at the REST API and the CLI but I am a novice and a little confused.

I have setup all the required searches that produce the data the business would require, can someone talk me through a simple way of extracting this via REST so I can populate the data into Excel or have the data delivered in csv format.

I am doing a Proof of Concept at the moment so I am just trying to run a test to understand how it works.

Thanks,

Dan

[steverimar]

We actually have a QlikView connector available for Splunk. E-mail me at steverimar@analyticaconsulting.com if you are interested. We're a QlikView and Splunk partner. -Steve

[Gilberto_Castil]

This process would require an explicit search to extract data either from the Splunk UI or the REST API. Assume for example that you have a Splunk saved search named “Alert Index Capacity”. You can then use the REST API to run the search on demand.

curl -k -u admin:splunk -d  "search=|savedsearch %22Alert%3A%20Index%20Capacity%22" https://10.0.1.20:8089/services/search/jobs/

At that point the REST API will respond with something like this:

---

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <sid>1367356709.287</sid>
</response>

---

To obtain the results into a CSV, then use the following:

curl -u admin:splunk -k https://10.0.1.21:8089/services/search/jobs/1367356709.287 --get -d output_mode=csv > mySplunkCSVData.csv

This should be formatted in a table format and may satisfactory for your third party tool. If you already have all of the knowledge items in Splunk, then you can use this method to extract the data.

[Mgrease]

Do we have similar examples for wget?

[barakreeves · Accepted solution]

Let's take this step-by-step. Start out with your favorite language. I will use Python.

1- Install the Python SDK; video link: https://www.youtube.com/watch?list=UUG5B6oQWWsIt-yO826IwG2Q&feature=player_detailpage&v=TUQcjSWzZg8;
Installation link: http://dev.splunk.com/goto/sdk-python

2- Connect to Splunk via the API: http://dev.splunk.com/view/SP-CAAAEE4
3- Run a simple search and display results in command line: http://dev.splunk.com/view/SP-CAAAEE5 and use "To create a basic oneshot search and display results" code example towards the bottom
4- See how the results are returned
5- From the results, modify the code to either create your own CSV or via the API

You can also use the REST API directly using curl. Here's a link with a code example: http://docs.splunk.com/Documentation/Splunk/5.0.3/RESTAPI/RESTsearch#search.2Fjobs.2Fexport

Hope this helps.

[DanielFordWA]

Thanks,

I am doing a Proof of Concept with Splunk to sit along side our web analytics tools. I have to say Splunk is great but getting data out of it is no easy task. I would have expected to be able to query Splunk from most other tools (Excel, Access, Qlikview etc) and load the data without much effort.

I'll follow your steps and see how I go.

Thanks,

Dan