Splunk Enterprise

Not to forward duplicate events.


Hi Experts,

disabled = false
interval= 300
passAuth = root

disabled = false
index = index_name
sourcetype = _json
crcSalt = <SOURCE>

Content of /opt/splunk/etc/apps/app_name/bin/test1.sh
command > /tmp/output.txt

As shown above script "test1.sh" runs a command and redirects the output to output.txt in json format.
Monitor reads output.txt and sends to splunk enterprise.

The "command" mentioned in test1.sh may or may not return new set of values whenever it runs.
So the forwarder should send only the events from output.txt, that were not indexed earlier.
Forwaarder is unbale to send the logs to splunk enterpirse if "crcSalt = <SOURCE>" is not mentioned.

Please help.


Thank you.



Labels (1)
0 Karma


Your scripted input (test1.sh) runs every 5mins and writes the content to a file in tmp (everytime the script runs, it overwrites the content. Not sure if this is what you need).

Now, your monitor stanza reads the contents off the file, as and when it sees entries and send them to indexer. As the content of the file could be similar (the first 256 chars, without CRCSalt, it can get ignored - you can see an error/warning in splunkd log about this. Check props.conf for CRCSalt).

One option would be to not override the contents, but append to the file and then have a logrotation to manage the size of the file. If you indeed want to override (as in your current approach), when the command returns no json/results, how will you know if the command returned results or not? so perhaps adding some entry to indicate writing to the file and completion status could be helpful.

0 Karma


Thats how UF works when UF is reading file. 

It will keep pointer at end of file and it resumes when new data is written to file.

why don’t you index script output directly to Splunk instead of writing it to file again?



If this helps, give a like below.
0 Karma


Forwarders have no concept of duplicate data.  All they know is the last position they read within each file they monitor.

I suggest removing the redirection from test1.sh.  That will cause Splunk to index the output of the script directly rather than via an intermediate file.

If this reply helps you, Karma would be appreciated.
0 Karma


Thank you for your reply.

Is there anyway to forward only the data to splunk  that got generated after the latest event indexed in splunk. 

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...