is there a rising column equivalent for DB Connect V2 Output?

[peterweinstein]

Hi,

I need to transfer Splunk extracts to Sql Server such that every record is written to Sql Server exactly one time. It doesn't need to be real time, for example, an hourly periodicity would be fine.

Currently, I am repeating the output every hour, and searching for the previous hour. This works but it is fragile and, given that Splunk may be down at times, the data saved to the database is incomplete.

We need the equivalent of a "rising column" -- Splunk should output all new records.

In SQL, I would select all records with a timestamp greater than the MAX of the previously written data.

What is the recommended approach using DB Connect V2?

Thanks,
Peter

[cramasta]

Sounds like what you are looking for is the equivalent of the backfill command which is exists for summary indexing.

However that doesn't cover you in situations when in a distributed environment and you have a single indexer down for a period of time. During that time you will be exporting partial results and Splunk just continues on as if everything's all good. The only App that splunk has which deals with this type of failure is the hadoop connect app. When it runs a search to export it will not write out the data to hadoop unless all indexers are up. If even one indexer is down is reruns the search until ALL indexers are up. I really wish Splunk incorporated type of check into more of their Apps/Scheduler.

[jcoates_splunk]

In Splunk, the equivalent would be the earliest search command. http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/SearchTimeModifiers