Splunk Enterprise

Not to forward duplicate events.

email2vamsi
Explorer

Hi Experts,


inputs.conf
[script:///opt/splunk/etc/apps/app_name/bin/test1.sh]
disabled = false
interval= 300
passAuth = root

[monitor:///tmp/output.txt]
disabled = false
index = index_name
sourcetype = _json
crcSalt = <SOURCE>

Content of /opt/splunk/etc/apps/app_name/bin/test1.sh
command > /tmp/output.txt

Background:
As shown above script "test1.sh" runs a command and redirects the output to output.txt in json format.
Monitor reads output.txt and sends to splunk enterprise.

Problem:
The "command" mentioned in test1.sh may or may not return new set of values whenever it runs.
So the forwarder should send only the events from output.txt, that were not indexed earlier.
Forwaarder is unbale to send the logs to splunk enterpirse if "crcSalt = <SOURCE>" is not mentioned.

Please help.

 

Thank you.

 

 

Labels (1)
0 Karma

lakshman239
Influencer

Your scripted input (test1.sh) runs every 5mins and writes the content to a file in tmp (everytime the script runs, it overwrites the content. Not sure if this is what you need).

Now, your monitor stanza reads the contents off the file, as and when it sees entries and send them to indexer. As the content of the file could be similar (the first 256 chars, without CRCSalt, it can get ignored - you can see an error/warning in splunkd log about this. Check props.conf for CRCSalt).

One option would be to not override the contents, but append to the file and then have a logrotation to manage the size of the file. If you indeed want to override (as in your current approach), when the command returns no json/results, how will you know if the command returned results or not? so perhaps adding some entry to indicate writing to the file and completion status could be helpful.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

Thats how UF works when UF is reading file. 

It will keep pointer at end of file and it resumes when new data is written to file.

why don’t you index script output directly to Splunk instead of writing it to file again?

 

 

————————————
If this helps, give a like below.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Forwarders have no concept of duplicate data.  All they know is the last position they read within each file they monitor.

I suggest removing the redirection from test1.sh.  That will cause Splunk to index the output of the script directly rather than via an intermediate file.

---
If this reply helps you, Karma would be appreciated.
0 Karma

email2vamsi
Explorer

Thank you for your reply.

Is there anyway to forward only the data to splunk  that got generated after the latest event indexed in splunk. 

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...