Getting Data In

Anyone have a good working DB polling scripts written in Python?


Anyone have a good working python DB table dump scripts that keeps track of last row marker?

I guess it would be in-efficient for every Splunk users to come up with their own scripts. Which I already have spent quete some to the get it to work since I am not a hardcore develper, but I am not quote 100% satisfied with the result.

Path Finder

I have written a script which periodically checks the DB (up to once a second) for new transactional entries and writes the log. Like Lowell's one it is not a product: no installer or docs, only runs with SQL server, and queries are hardcoded into a module. Maybe if there's enough interest.

It writes pipe-separated key-value pairs (K1=V1|K2=V2), with the timestamp key first. The files are named like /{basepath}/{queryname}/{YYYYMMDD}.log and rolled at midnight.

A "timestamp" file maintains the date of the last row retrieved for a query, whose contents is used as the start date for the next query execution. A --start YYYYMMDD argument forces it to overwrite old log files for a delete-and-reindex operation.

We have got 9 queries running to pull data from timestamped database tables. They join up to 8 other tables to pull in all possible related fields. Splunk then acts like a data warehouse: denormalised read-only transactional data for charting and statistics.

Splunk Employee
Splunk Employee

Personally, if there were to be a shared canonical DB scripted input, I think it should be written in Java, because of the far better DB interface support.

0 Karma

Super Champion

I have written a python library to take care of what you're talking about. And your right, not everyone should have to reinvent the wheel on this. Unfortunately, I haven't had a chance to publish a demo app yet. I'm much closer than I was, but more cleanup is needed to remove any company specific info.

What platform are you running on? And which database are you trying to connect to? If I could contact you offline, I would be willing to sent you a rough draft version of the app. It would be helpful to me to have some feedback from a less python-savvy individual since I would like to may a generic app that is fairly accessible to both python pros, and novices alike. (At the moment, what I have may be over engineered, so I'd be curious in hearing your requirements.)

Unfortunately, I don't think this site has a user-to-use message feature.... (which I generally don't like anyways.)

Super Champion

Clyde772, send me an email at (my user name) + "87" at gmail dot com. (I'll be deleting this comment after you do)

Get Updates on the Splunk Community!

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...