Not to forward duplicate events.

[email2vamsi]

Hi Experts,

inputs.conf
[script:///opt/splunk/etc/apps/app_name/bin/test1.sh]
disabled = false
interval= 300
passAuth = root

[monitor:///tmp/output.txt]
disabled = false
index = index_name
sourcetype = _json
crcSalt = <SOURCE>

Content of /opt/splunk/etc/apps/app_name/bin/test1.sh
command > /tmp/output.txt

Background:
As shown above script "test1.sh" runs a command and redirects the output to output.txt in json format.
Monitor reads output.txt and sends to splunk enterprise.

Problem:
The "command" mentioned in test1.sh may or may not return new set of values whenever it runs.
So the forwarder should send only the events from output.txt, that were not indexed earlier.
Forwaarder is unbale to send the logs to splunk enterpirse if "crcSalt = <SOURCE>" is not mentioned.

Please help.

Thank you.

[lakshman239]

Your scripted input (test1.sh) runs every 5mins and writes the content to a file in tmp (everytime the script runs, it overwrites the content. Not sure if this is what you need).

Now, your monitor stanza reads the contents off the file, as and when it sees entries and send them to indexer. As the content of the file could be similar (the first 256 chars, without CRCSalt, it can get ignored - you can see an error/warning in splunkd log about this. Check props.conf for CRCSalt).

One option would be to not override the contents, but append to the file and then have a logrotation to manage the size of the file. If you indeed want to override (as in your current approach), when the command returns no json/results, how will you know if the command returned results or not? so perhaps adding some entry to indicate writing to the file and completion status could be helpful.

[thambisetty]

Thats how UF works when UF is reading file. 

It will keep pointer at end of file and it resumes when new data is written to file.

why don’t you index script output directly to Splunk instead of writing it to file again?

[richgalloway]

Forwarders have no concept of duplicate data.  All they know is the last position they read within each file they monitor.

I suggest removing the redirection from test1.sh.  That will cause Splunk to index the output of the script directly rather than via an intermediate file.

[email2vamsi]

Thank you for your reply.

Is there anyway to forward only the data to splunk  that got generated after the latest event indexed in splunk.