Feeding data from script into splunk while avoiding data duplicates

[skrskr]

Hello all,

upfront: first time Splunk user here, be patient with me 🙂

I've a scenario I would like to describe and which I require some comments on in regards to how this can be archived with Splunk.

Scenario:
- I have a PERL script which is generating data from a target API (stdout or file) on a daily basis - the script requires to be executed with a parameter and only retrieves data with a timestamp that is newer than the value of the supplied parameter
- now I want to forward this CSV formated output into Splunk
- tha data integrity shall be handled by relying on the information which is stored in Splunk (the highest timestamp value stored)

Generally I'm not sure how to assure that splunk does not create duplicates for this data.

Current approach/idea:
1) a daily routine in Splunk is triggered (I assume that would be the job of the forwarder)
2) this input routine checks for the highest timestamp value currently stored in the Splunk index, passes this information towards the PERL script and executes it
3) Splunk takes the output from the PERL script (stdout or file) and feeds it into the index

Does the approach sound reasonable? I'm uncertain how to archive the logic described in 2) - I was thinking about firing the module up as script:// but I'm uncertain how to pass the timestamp value stored in the index. As an alternative I was thinking about just dumping the whole information from the API each time and afterwards somehow filter for data which already was indexed. What can I do to implement a logic for validating for duplicated data?

Any recommendation or pointing in the right direction would be appreciated.

Cheers!

[Damien_Dallimor]

You would be better to implement a modular input that keeps track of its position in the data source and , using the Splunk REST API , persists this positional data (your timestamp) back to inputs.conf so that upon Splunk restarts it remembers where it left off and won't index duplicate data.

Have a look at my REST API Modular input as an example of how to do this.

It comes with an example Twitter handler that keeps track of the tweet stream "since_id" so that it doesn't index duplicate data.This "since_id" is persisted back to Splunk via the REST API. So this is the same paradigm you are trying to achieve with a positional timestamp.