Configuring dbx tail input on very large table. (70,000,000+ rows)

[johnebgood]

I was able to get the input to work but it would be nice to start the tail from a specified value when the input is configured. I don't want to input all 70,000,000 rows on the first run... I just want the data from the last day or so as I would blow up our license since I have to do this for 20+ databases with similarly sized tables. Any ideas on how to do this?

[piebob · Accepted solution]

solution is in this answer:
http://splunk-base.splunk.com/answers/68572/splunk-db-connect-how-to-reset-tailrising-state

[hirsts]

Creating the monitor with initial SQL to return one row only, updating the state.xml and then remove the 1 row limit worked fine. I did not need to restart Splunk as the state.xml file is read each time the monitor is run.

[hirsts]

I've tied this and the state.xml file is only created after the first run of the monitor. In my the case the first run returned lots of rows which then seem to get batch inserting into the indexing queues as the state.xml files keeps changing until the backlog is indexed.

I'm going to try creating the monitor with SQL that only returns 1 row at a time, disable the monitor, modify the state.xml, modify the SQL to return all the required rows then re-enable the monitor.

[johnebgood]

Note, after you create the new database monitor immediately disable it and go and edit the state.xml file. I have been copying the state.xml from previously configured monitors, makes it fairly easy.

[piebob]

related/same question: http://splunk-base.splunk.com/answers/69895/splunk-db-connect-tail-only-from-latest-value-in-databas...