Getting Data In

Anyone have a good working DB polling scripts written in Python?


Anyone have a good working python DB table dump scripts that keeps track of last row marker?

I guess it would be in-efficient for every Splunk users to come up with their own scripts. Which I already have spent quete some to the get it to work since I am not a hardcore develper, but I am not quote 100% satisfied with the result.

Path Finder

I have written a script which periodically checks the DB (up to once a second) for new transactional entries and writes the log. Like Lowell's one it is not a product: no installer or docs, only runs with SQL server, and queries are hardcoded into a module. Maybe if there's enough interest.

It writes pipe-separated key-value pairs (K1=V1|K2=V2), with the timestamp key first. The files are named like /{basepath}/{queryname}/{YYYYMMDD}.log and rolled at midnight.

A "timestamp" file maintains the date of the last row retrieved for a query, whose contents is used as the start date for the next query execution. A --start YYYYMMDD argument forces it to overwrite old log files for a delete-and-reindex operation.

We have got 9 queries running to pull data from timestamped database tables. They join up to 8 other tables to pull in all possible related fields. Splunk then acts like a data warehouse: denormalised read-only transactional data for charting and statistics.

Splunk Employee
Splunk Employee

Personally, if there were to be a shared canonical DB scripted input, I think it should be written in Java, because of the far better DB interface support.

0 Karma

Super Champion

I have written a python library to take care of what you're talking about. And your right, not everyone should have to reinvent the wheel on this. Unfortunately, I haven't had a chance to publish a demo app yet. I'm much closer than I was, but more cleanup is needed to remove any company specific info.

What platform are you running on? And which database are you trying to connect to? If I could contact you offline, I would be willing to sent you a rough draft version of the app. It would be helpful to me to have some feedback from a less python-savvy individual since I would like to may a generic app that is fairly accessible to both python pros, and novices alike. (At the moment, what I have may be over engineered, so I'd be curious in hearing your requirements.)

Unfortunately, I don't think this site has a user-to-use message feature.... (which I generally don't like anyways.)

Super Champion

Clyde772, send me an email at (my user name) + "87" at gmail dot com. (I'll be deleting this comment after you do)

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!