Transfer data to Splunk from database on different server

[harry2007gsp]

We are right now transferring data from MongoDb to Splunk with the help of 3rd party JDBC driver via DBX addon app for Splunk.
Now our scenario is to replace MongoDb with MySql and we don't wanna use any JDBC driver for it.

So which is the best way to transfer data from performance point of view?
1. Use Splunk Python SDK and push data with code from MySql to Splunk?
2. Use common network drive for both MySql and Splunk servers and do scp linux command?
3. Something else?

Please help me.

[cpetterborg · Accepted solution]

From a purely performance standpoint, using a process that grabs the data from the database and sends it into Splunk via HEC (Http Event Collector) would be one of the most performant ways, but it also would be more maintenance, and would require you to do some custom set up.

From a purely get-the-data-in maintainability perspective, use DBX. DBX isn't the best way to get data into and out of Splunk, but it is a safe way and typically gets the job done well enough.

[harry2007gsp]

Thanks @cpetterborg. I was thinking to copy data with scp into Splunk's staging directory with a script on common network drive. So is this HEC even faster than that?

[cpetterborg]

Yes, significantly.

[harry2007gsp]

@cpetterborg, can we use HEC for lookup(mapping tables in mysql) data transfer to lookups(Splunk's lookups) inside Splunk rather than indexes?

I could not find how to save data in Splunk's lookups with HEC.

[cpetterborg]

You cannot create a lookup using the HEC. It is only for getting events into Splunk. In order to create/modify a lookup you will have to use the REST API.

[harry2007gsp]

Hi @cpetterborg,
Can we push lookup table data from outside database(mongoDb lookukp collection) to splunk with splunk python sdk?

We have been pushing normal data to splunk with the help of third party JDBC unity drivers but now planning to push it with python splunk sdk. This case is possible and we know how to do it.

Problem is how can we push lookup data to splunk lookup tables instead of indexes.

[cpetterborg]

I have not done that, so I don't know, but I should think that it is possible, given that most things are available through the API's. I have been told that the SDK is not as robust as the REST API, but I don't have first hand knowledge of that.

[harry2007gsp]

Thanks. I will then do a little more research about that.