All Apps and Add-ons

is there a rising column equivalent for DB Connect V2 Output?

peterweinstein
Explorer

Hi,

I need to transfer Splunk extracts to Sql Server such that every record is written to Sql Server exactly one time. It doesn't need to be real time, for example, an hourly periodicity would be fine.

Currently, I am repeating the output every hour, and searching for the previous hour. This works but it is fragile and, given that Splunk may be down at times, the data saved to the database is incomplete.

We need the equivalent of a "rising column" -- Splunk should output all new records.

In SQL, I would select all records with a timestamp greater than the MAX of the previously written data.

What is the recommended approach using DB Connect V2?

Thanks,
Peter

Tags (1)
0 Karma

cramasta
Builder

Sounds like what you are looking for is the equivalent of the backfill command which is exists for summary indexing.

However that doesn't cover you in situations when in a distributed environment and you have a single indexer down for a period of time. During that time you will be exporting partial results and Splunk just continues on as if everything's all good. The only App that splunk has which deals with this type of failure is the hadoop connect app. When it runs a search to export it will not write out the data to hadoop unless all indexers are up. If even one indexer is down is reruns the search until ALL indexers are up. I really wish Splunk incorporated type of check into more of their Apps/Scheduler.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

In Splunk, the equivalent would be the earliest search command. http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/SearchTimeModifiers

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...