Splunk Enterprise

Not to forward duplicate events.


Hi Experts,

disabled = false
interval= 300
passAuth = root

disabled = false
index = index_name
sourcetype = _json
crcSalt = <SOURCE>

Content of /opt/splunk/etc/apps/app_name/bin/test1.sh
command > /tmp/output.txt

As shown above script "test1.sh" runs a command and redirects the output to output.txt in json format.
Monitor reads output.txt and sends to splunk enterprise.

The "command" mentioned in test1.sh may or may not return new set of values whenever it runs.
So the forwarder should send only the events from output.txt, that were not indexed earlier.
Forwaarder is unbale to send the logs to splunk enterpirse if "crcSalt = <SOURCE>" is not mentioned.

Please help.


Thank you.



Labels (1)
0 Karma


Your scripted input (test1.sh) runs every 5mins and writes the content to a file in tmp (everytime the script runs, it overwrites the content. Not sure if this is what you need).

Now, your monitor stanza reads the contents off the file, as and when it sees entries and send them to indexer. As the content of the file could be similar (the first 256 chars, without CRCSalt, it can get ignored - you can see an error/warning in splunkd log about this. Check props.conf for CRCSalt).

One option would be to not override the contents, but append to the file and then have a logrotation to manage the size of the file. If you indeed want to override (as in your current approach), when the command returns no json/results, how will you know if the command returned results or not? so perhaps adding some entry to indicate writing to the file and completion status could be helpful.

0 Karma

Super Champion

Thats how UF works when UF is reading file. 

It will keep pointer at end of file and it resumes when new data is written to file.

why don’t you index script output directly to Splunk instead of writing it to file again?



If this helps, give a like below.
0 Karma


Forwarders have no concept of duplicate data.  All they know is the last position they read within each file they monitor.

I suggest removing the redirection from test1.sh.  That will cause Splunk to index the output of the script directly rather than via an intermediate file.

If this reply helps you, Karma would be appreciated.
0 Karma


Thank you for your reply.

Is there anyway to forward only the data to splunk  that got generated after the latest event indexed in splunk. 

0 Karma
Get Updates on the Splunk Community!

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...