topic Re: automatically forward splunk data to database in Getting Data In

automatically forward splunk data to database

nielsenr — Mon, 06 Aug 2012 23:05:37 GMT

Ok so I am new to splunk and have an instance set up with logs from several servers feeding into it.
My question is can i get the data from splunk into a database(probably mysql) automatically.
There seems to be no clear explanation to say whether this is possible or not.

I have read a bit about the splunk forwarder would this be able to do it? once more there is no real explanation I can find.

Any suggestions would be helpful
thanks

Re: automatically forward splunk data to database

Takajian — Mon, 06 Aug 2012 23:29:48 GMT

Splunk forwarder can forward data to splunk other instance or third party software like syslogd with text format. I do not think splunk forwarder can foward data to mysql. you will need to make script to geta data from splunk and put into the mysql.

Re: automatically forward splunk data to database

Damien_Dallimor — Tue, 07 Aug 2012 02:15:34 GMT

A couple of approaches :

1) The Splunk MYSQL connector includes a search command, mysqloutput, that you can use to insert or update records in a table in a MySQL database based on fields resulting from your Splunk search.

2) you could use one of the Developer SDKs, write a custom program to execute a Splunk search , process the XML/JSON/CSV result and roll this up into a SQL statement to insert/update tables in your database.

3) you could write your own custom search command , and insert this at the end of you search pipeline to insert/update your DB tables with Splunk search fields ie: index=foo sourcetype=goo | stats count by host | myCustomOutputToDBCommand

Re: automatically forward splunk data to database

nielsenr — Tue, 07 Aug 2012 21:57:21 GMT

Thanks for the feedback
I have already looked at MYSQL connector and unless I'm mistaken it seems to only to be able to insert into a table data that has been searched form the database(please correct me if I'm wrong)

I am currently using the SDK's provided but was hoping for an easier solution (just being lazy)

umm, a custom search command never crossed my mind I'll have a look at that and see if it's useful.

Ill keep this updated with what I do. I have a feeling I'm not the first person to wonder about this, of course anymore suggestions would be welcome

Re: automatically forward splunk data to database

nielsenr — Tue, 07 Aug 2012 21:57:35 GMT

Thanks for clearing that up

Re: automatically forward splunk data to database

RohiniJindam — Wed, 14 Aug 2013 11:31:23 GMT

@nielsenr I am trying to achieve something similar to what you stated. Could you find a solution?