topic Index a specific time range with DB Connect in Getting Data In

Index a specific time range with DB Connect

Yorokobi — Thu, 05 Dec 2013 16:23:41 GMT

I want to import only the last X months/days/whatever of data from my database via DB Connect. Is there a better procedure than the following?

Create a temporary index (restart Splunk, etc.)
Create the DB input to send to the temp index and set tail.follow.only=true
Wait for Splunk to create the state.xml file
Stop Splunk
Change the DB connection's state.xml and set the value for the rising column to the start date I want
Change the inputs.conf to use the "permanent" index and remove the tail.follow.only line
Start Splunk

Is it possible to create the $SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx/HASH/state.xml and manifest.properties manually? What hash function does Splunk use and what does it hash? The input name or something else?

Re: Index a specific time range with DB Connect

aelliott — Thu, 05 Dec 2013 16:55:45 GMT

I add the following outside the brackets: where datefield > "12/1/2010 00:00:00" {AND $rising_column$ > ?}

Then you can set the retention on the index.

Re: Index a specific time range with DB Connect

Yorokobi — Thu, 05 Dec 2013 19:45:24 GMT

It looks like the suggestions do not work "as-is" for this particular Oracle table but they are far better than what I came up with myself and got me started in a much better direction. With the help of a DBA, we settled on using:

SELECT ... WHERE LOG_TIMESTAMP > TO_DATE('2013/07/30 01:00:00 PM', 'YYYY/MM/DD HH:MI:SS PM') {{AND $rising_column$ > ?}}