Developing for Splunk Enterprise

How to avoid duplication of events for multiple modular inputs?

umairahmad3985
Path Finder

Dear All,

I have created a Python modular input (of multiple instance type) using Splunk's Add-on builder that polls a REST API and pulls JSON data for indexing into Splunk. The parameters of the API are start and end timestamps, for which the data is required. In order to avoid duplication, I am keeping the last_polled time as a checkpoint in my modular input so that on the next execution, the script knows from where to start fetching the data. This works great when the user creates only one input from the modular input but if the user creates another input to ingest the data in a separate other index, the script will be fetching the last_polled time from the first input as checkpoints are shared within a modular input so it will miss some data if their intervals are not the same.

Is there any technique to isolate checkpoints for each input so that they are not shared between them? Ideally, I would want them to be isolated according to the index and sourcetype defined by the user.

I hope I was able to clear my requirement clearly, let me know if you need more information on this. Will be very happy to receive some direction on this as the documentation has little information.

Regards,
Umair

Labels (1)
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!