Splunk Enterprise

Not to forward duplicate events.

email2vamsi
Explorer

Hi Experts,


inputs.conf
[script:///opt/splunk/etc/apps/app_name/bin/test1.sh]
disabled = false
interval= 300
passAuth = root

[monitor:///tmp/output.txt]
disabled = false
index = index_name
sourcetype = _json
crcSalt = <SOURCE>

Content of /opt/splunk/etc/apps/app_name/bin/test1.sh
command > /tmp/output.txt

Background:
As shown above script "test1.sh" runs a command and redirects the output to output.txt in json format.
Monitor reads output.txt and sends to splunk enterprise.

Problem:
The "command" mentioned in test1.sh may or may not return new set of values whenever it runs.
So the forwarder should send only the events from output.txt, that were not indexed earlier.
Forwaarder is unbale to send the logs to splunk enterpirse if "crcSalt = <SOURCE>" is not mentioned.

Please help.

 

Thank you.

 

 

Labels (1)
0 Karma

lakshman239
SplunkTrust
SplunkTrust

Your scripted input (test1.sh) runs every 5mins and writes the content to a file in tmp (everytime the script runs, it overwrites the content. Not sure if this is what you need).

Now, your monitor stanza reads the contents off the file, as and when it sees entries and send them to indexer. As the content of the file could be similar (the first 256 chars, without CRCSalt, it can get ignored - you can see an error/warning in splunkd log about this. Check props.conf for CRCSalt).

One option would be to not override the contents, but append to the file and then have a logrotation to manage the size of the file. If you indeed want to override (as in your current approach), when the command returns no json/results, how will you know if the command returned results or not? so perhaps adding some entry to indicate writing to the file and completion status could be helpful.

0 Karma

thambisetty
Super Champion

Thats how UF works when UF is reading file. 

It will keep pointer at end of file and it resumes when new data is written to file.

why don’t you index script output directly to Splunk instead of writing it to file again?

 

 

————————————
If this helps, give a like below.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Forwarders have no concept of duplicate data.  All they know is the last position they read within each file they monitor.

I suggest removing the redirection from test1.sh.  That will cause Splunk to index the output of the script directly rather than via an intermediate file.

---
If this reply helps you, Karma would be appreciated.
0 Karma

email2vamsi
Explorer

Thank you for your reply.

Is there anyway to forward only the data to splunk  that got generated after the latest event indexed in splunk. 

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...