Forwarder not sending data to indexer

[722624]

Please check the splunkd.log

08-30-2017 21:03:32.004 -0400 INFO TcpOutputProc - Connected to idx=10.100.xxx.1:9997, pset=0, reuse=0.
08-30-2017 21:03:32.008 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
08-30-2017 21:03:32.009 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_access.log'.
08-30-2017 21:03:32.011 -0400 INFO WatchedFile - Will begin reading at offset=57592 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
08-30-2017 21:03:32.013 -0400 INFO WatchedFile - Will begin reading at offset=969 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
08-30-2017 21:03:32.014 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
08-30-2017 21:03:32.016 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
08-30-2017 21:03:32.017 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
08-30-2017 21:03:32.019 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
08-30-2017 21:03:32.020 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
08-30-2017 21:03:32.022 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
08-30-2017 21:03:32.024 -0400 INFO WatchedFile - Will begin reading at offset=369 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
08-30-2017 21:03:32.025 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
08-30-2017 21:03:32.102 -0400 INFO WatchedFile - Will begin reading at offset=20365668 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
08-30-2017 21:14:07.561 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.
08-30-2017 21:29:06.640 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.

I need the file /opt/scripts/rsda.txt to be indexed , this is file is recreated every 15 mins....
but this is not coming to indexer
both UF and Indexer are in Linux, ping is working both ways....

I have searched , there are so many posts but none is addressing this problem..

Thank you
AB

[woodcock]

You need to modify CHECK_METHOD in props.conf to modtime (checks only modification time of file):
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf?utm_source=answers&utm_medium=in...

[muszyngr]

ok, so what did it what fixed it? this is so frustrating finding unanswered threads, not your fault, just the Splunk Documentation is so lacking and here I am three years later with a very similar issue

[gcusello]

Hi AB,
some questions, to better understand the situation:

when the file is recreated, it's different, the same or both the possibilities?

Surely Splunk don't index it when it's the same, but only when updated.

When you update file, do you modify first chars?

Bye.
Giuseppe

[gcusello]

ok for different content, but the first 256 chars hare different or the same?

When you say that the only two servers aren't sending logs, do you mean that the problem is only on two UF and correctly runs on the other 13?
If yes, delete the first question.

Bye.
Giuseppe

[722624]

Hello Giuseppe,
Thanks for quick response
Yes..first 250 chars are also different
We have same version of UF installed on each of our 15 hosts...13 hosts are sending data to indexer..but 2 hosts are not sending the data

[gcusello]

Yes, the problem that you don't index updates there is only on two Forwarders or in all Forwarders?
if the first, you have to check if the two Forwarders send other logs to Indexer ( index=_internal host=your_host1 OR host=your_host2 ).
If the second, it's a different problem.
Bye.
Giuseppe

[722624]

only two forwrarders are not sending... index=_internal host=your_host1 ...is not giving any data

[gcusello]

This means that the problem isn't in the ingestion of the variation of the file, te problem in in connection!

at first check if firewalls rules are open, using telnet IP_Indexer 9997

if ok, check hostname in $SPLUNK_HOME/etc/system/local/inputs.conf and $SPLUNK_HOME/etc/system/local/server.conf (beware if you have the same hostname of another forwarder sometimes it happens!)

if ok, check if outputs.conf is correctly configurated (usually is in $SPLUNK_HOME/etc/system/local/ or in a dedicated App): you must have something like this:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = IP_Indexer:9997
disabled=false
[tcpout-server://IP_Indexer:9997]

otherwise see at http://docs.splunk.com/Documentation/Forwarder/6.6.3/Forwarder/Troubleshoottheuniversalforwarder

Bye.
Giuseppe

[722624]

Hello Giuseppe....
the file is created every 15 mins with same file name,,,,but with different content ,
I have total 15 hosts, same configuration , same OS, same UF...13 hosts are sending but 2 hosts are not sending...