All Apps and Add-ons

is there a rising column equivalent for DB Connect V2 Output?

peterweinstein
Explorer

Hi,

I need to transfer Splunk extracts to Sql Server such that every record is written to Sql Server exactly one time. It doesn't need to be real time, for example, an hourly periodicity would be fine.

Currently, I am repeating the output every hour, and searching for the previous hour. This works but it is fragile and, given that Splunk may be down at times, the data saved to the database is incomplete.

We need the equivalent of a "rising column" -- Splunk should output all new records.

In SQL, I would select all records with a timestamp greater than the MAX of the previously written data.

What is the recommended approach using DB Connect V2?

Thanks,
Peter

Tags (1)
0 Karma

cramasta
Builder

Sounds like what you are looking for is the equivalent of the backfill command which is exists for summary indexing.

However that doesn't cover you in situations when in a distributed environment and you have a single indexer down for a period of time. During that time you will be exporting partial results and Splunk just continues on as if everything's all good. The only App that splunk has which deals with this type of failure is the hadoop connect app. When it runs a search to export it will not write out the data to hadoop unless all indexers are up. If even one indexer is down is reruns the search until ALL indexers are up. I really wish Splunk incorporated type of check into more of their Apps/Scheduler.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

In Splunk, the equivalent would be the earliest search command. http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/SearchTimeModifiers

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...