Building for the Splunk Platform

How to avoid duplication of events for multiple modular inputs?

Path Finder

Dear All,

I have created a Python modular input (of multiple instance type) using Splunk's Add-on builder that polls a REST API and pulls JSON data for indexing into Splunk. The parameters of the API are start and end timestamps, for which the data is required. In order to avoid duplication, I am keeping the last_polled time as a checkpoint in my modular input so that on the next execution, the script knows from where to start fetching the data. This works great when the user creates only one input from the modular input but if the user creates another input to ingest the data in a separate other index, the script will be fetching the last_polled time from the first input as checkpoints are shared within a modular input so it will miss some data if their intervals are not the same.

Is there any technique to isolate checkpoints for each input so that they are not shared between them? Ideally, I would want them to be isolated according to the index and sourcetype defined by the user.

I hope I was able to clear my requirement clearly, let me know if you need more information on this. Will be very happy to receive some direction on this as the documentation has little information.


Labels (1)
0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...