Splunk Dev

Index access from scripted or modular input

chrisdev
Explorer

Is it possible to access the index from a scripted or modular input? And is the standard way of doing this via the SDK features as shown in examples such as search.py?

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Scripted and modular inputs are input-only.  Technically, they don't access indexes at all, but produce output which Splunk then writes to an index.

What problem are you trying to solve?

---
If this reply helps you, Karma would be appreciated.

chrisdev
Explorer

Thank you.

Weighing up my options for deduplication of events at index time. Doesn't seem like the correct approach if it's bad practice to access an index from a  scripted/modular input.

I know that there's a dedup command at search time, however ideally I wouldn't have duplicate events in the index. The source of these events, which im not in control of, may produce duplicates.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I suggest the modular input cache events internally rather than try to fetch them from an index.  If a new event is in the cache then it's a duplicate and can be discarded; otherwise, index it and add it to the cache.  Yes, the cache will limit your look-back for duplicates, but will perform vastly better than scanning an index for every incoming event.  You'll still need to handle duplicates at search time, but there will be far fewer of them.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...