Getting Data In

Forwarder not sending data to indexer

Path Finder

Please check the splunkd.log

08-30-2017 21:03:32.004 -0400 INFO TcpOutputProc - Connected to idx=10.100.xxx.1:9997, pset=0, reuse=0.
08-30-2017 21:03:32.008 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
08-30-2017 21:03:32.009 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_access.log'.
08-30-2017 21:03:32.011 -0400 INFO WatchedFile - Will begin reading at offset=57592 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
08-30-2017 21:03:32.013 -0400 INFO WatchedFile - Will begin reading at offset=969 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
08-30-2017 21:03:32.014 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
08-30-2017 21:03:32.016 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
08-30-2017 21:03:32.017 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
08-30-2017 21:03:32.019 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
08-30-2017 21:03:32.020 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
08-30-2017 21:03:32.022 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
08-30-2017 21:03:32.024 -0400 INFO WatchedFile - Will begin reading at offset=369 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
08-30-2017 21:03:32.025 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
08-30-2017 21:03:32.102 -0400 INFO WatchedFile - Will begin reading at offset=20365668 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
08-30-2017 21:14:07.561 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.
08-30-2017 21:29:06.640 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.

I need the file /opt/scripts/rsda.txt to be indexed , this is file is recreated every 15 mins....
but this is not coming to indexer
both UF and Indexer are in Linux, ping is working both ways....

I have searched , there are so many posts but none is addressing this problem..

Thank you
AB

0 Karma

Esteemed Legend

You need to modify CHECK_METHOD in props.conf to modtime (checks only modification time of file):
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf?utm_source=answers&utm_medium=in...

0 Karma

New Member

ok, so what did it what fixed it? this is so frustrating finding unanswered threads, not your fault, just the Splunk Documentation is so lacking and here I am three years later with a very similar issue

0 Karma

Legend

Hi AB,
some questions, to better understand the situation:

when the file is recreated, it's different, the same or both the possibilities?

Surely Splunk don't index it when it's the same, but only when updated.

When you update file, do you modify first chars?

Bye.
Giuseppe

0 Karma

Legend

ok for different content, but the first 256 chars hare different or the same?

When you say that the only two servers aren't sending logs, do you mean that the problem is only on two UF and correctly runs on the other 13?
If yes, delete the first question.

Bye.
Giuseppe

0 Karma

Path Finder

Hello Giuseppe,
Thanks for quick response
Yes..first 250 chars are also different
We have same version of UF installed on each of our 15 hosts...13 hosts are sending data to indexer..but 2 hosts are not sending the data

0 Karma

Legend

Yes, the problem that you don't index updates there is only on two Forwarders or in all Forwarders?
if the first, you have to check if the two Forwarders send other logs to Indexer ( index=_internal host=your_host1 OR host=your_host2 ).
If the second, it's a different problem.
Bye.
Giuseppe

0 Karma

Path Finder

only two forwrarders are not sending... index=_internal host=your_host1 ...is not giving any data

0 Karma

Legend

This means that the problem isn't in the ingestion of the variation of the file, te problem in in connection!

at first check if firewalls rules are open, using telnet IP_Indexer 9997

if ok, check hostname in $SPLUNK_HOME/etc/system/local/inputs.conf and $SPLUNK_HOME/etc/system/local/server.conf (beware if you have the same hostname of another forwarder sometimes it happens!)

if ok, check if outputs.conf is correctly configurated (usually is in $SPLUNK_HOME/etc/system/local/ or in a dedicated App): you must have something like this:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = IP_Indexer:9997
disabled=false
[tcpout-server://IP_Indexer:9997]

otherwise see at http://docs.splunk.com/Documentation/Forwarder/6.6.3/Forwarder/Troubleshoottheuniversalforwarder

Bye.
Giuseppe

0 Karma

Path Finder

Hello Giuseppe....
the file is created every 15 mins with same file name,,,,but with different content ,
I have total 15 hosts, same configuration , same OS, same UF...13 hosts are sending but 2 hosts are not sending...

0 Karma