topic Anyone have a good working DB polling scripts written in Python? in Getting Data In

Anyone have a good working DB polling scripts written in Python?

clyde772 — Mon, 03 May 2010 17:10:04 GMT

Anyone have a good working python DB table dump scripts that keeps track of last row marker?

I guess it would be in-efficient for every Splunk users to come up with their own scripts. Which I already have spent quete some to the get it to work since I am not a hardcore develper, but I am not quote 100% satisfied with the result.

Re: Anyone have a good working DB polling scripts written in Python?

Lowell — Tue, 18 May 2010 04:57:49 GMT

I have written a python library to take care of what you're talking about. And your right, not everyone should have to reinvent the wheel on this. Unfortunately, I haven't had a chance to publish a demo app yet. I'm much closer than I was, but more cleanup is needed to remove any company specific info.

What platform are you running on? And which database are you trying to connect to? If I could contact you offline, I would be willing to sent you a rough draft version of the app. It would be helpful to me to have some feedback from a less python-savvy individual since I would like to may a generic app that is fairly accessible to both python pros, and novices alike. (At the moment, what I have may be over engineered, so I'd be curious in hearing your requirements.)

Unfortunately, I don't think this site has a user-to-use message feature.... (which I generally don't like anyways.)

Re: Anyone have a good working DB polling scripts written in Python?

gkanapathy — Tue, 18 May 2010 18:41:05 GMT

Personally, if there were to be a shared canonical DB scripted input, I think it should be written in Java, because of the far better DB interface support.

Re: Anyone have a good working DB polling scripts written in Python?

Lowell — Tue, 18 May 2010 21:36:23 GMT

Clyde772, send me an email at (my user name) + "87" at gmail dot com. (I'll be deleting this comment after you do)

Re: Anyone have a good working DB polling scripts written in Python?

grahampoulter — Wed, 02 Jun 2010 21:34:45 GMT

I have written a script which periodically checks the DB (up to once a second) for new transactional entries and writes the log. Like Lowell's one it is not a product: no installer or docs, only runs with SQL server, and queries are hardcoded into a module. Maybe if there's enough interest.

It writes pipe-separated key-value pairs (K1=V1|K2=V2), with the timestamp key first. The files are named like /{basepath}/{queryname}/{YYYYMMDD}.log and rolled at midnight.

A "timestamp" file maintains the date of the last row retrieved for a query, whose contents is used as the start date for the next query execution. A --start YYYYMMDD argument forces it to overwrite old log files for a delete-and-reindex operation.

We have got 9 queries running to pull data from timestamped database tables. They join up to 8 other tables to pull in all possible related fields. Splunk then acts like a data warehouse: denormalised read-only transactional data for charting and statistics.