Why can't the forwarder index and populate data?

[shawno]

We're unable to get the forwarder to index/re-index and populate data - any make out what is happening here? Thanks

03-06-2018 22:08:21.280 +0000 INFO  TailReader - Ignoring file '/tmp/hsperfdata_root/3843' due to: binary
03-06-2018 22:08:39.078 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" which: no tshark in (/opt/splunk/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
03-06-2018 22:08:39.104 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" /opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh: line 8: -v: command not found
03-06-2018 22:08:39.111 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" /opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh: line 31: [: : integer expression expected
03-06-2018 22:08:39.153 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/rlog.sh" Redirecting to /bin/systemctl status auditd.service
03-06-2018 22:08:40.347 +0000 WARN  FileClassifierManager - The file '/tmp/hsperfdata_root/3843' is invalid. Reason: binary
03-06-2018 22:08:40.347 +0000 INFO  TailReader - Ignoring file '/tmp/hsperfdata_root/3843' due to: binary
03-06-2018 22:08:48.320 +0000 WARN  LineBreakingProcessor - Truncating line because limit of 1000000 bytes has been exceeded with a line length >= 1003520 - data_source="lsof", data_host="harplg01.stag.defence.gov.au", data_sourcetype="lsof"
03-06-2018 22:09:08.887 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" which: no tshark in (/opt/splunk/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
03-06-2018 22:09:08.936 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" /opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh: line 8: -v: command not found
03-06-2018 22:09:08.947 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh" /opt/splunk/etc/apps/SplunkForPCAP/bin/pcap2csv.sh: line 31: [: : integer expression expected
03-06-2018 22:09:10.449 +0000 WARN  FileClassifierManager - The file '/tmp/hsperfdata_root/3843' is invalid. Reason: binary
03-06-2018 22:09:10.449 +0000 INFO  TailReader - Ignoring file '/tmp/hsperfdata_root/3843' due to: binary
03-06-2018 22:09:19.336 +0000 WARN  DateParserVerbose - Accepted time format has changed ((?i)(?

[lloydknight]

Hello shawno,

Bunch of messages are happening from your posted splunkd.log.
You might want to address some of them if they're part of your requirement too.
Most of the messages are straightforward and can addressed individually.

For example, /tmp/hsperfdata_root/3843 is being ignored due to binary.
Check this https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf#Binary_file_configuration

[lloydknight]

Hello @shawno

Kindly check this similar question below:
https://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html

Hope it helps!

[shawno]

I've already used this article and no joy...

02-12-2018 02:43:58.919 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
02-12-2018 02:43:58.951 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
02-12-2018 02:43:58.975 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
02-12-2018 02:43:58.984 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_access.log'.
02-12-2018 02:43:59.005 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
02-12-2018 02:43:59.023 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_audit.log'.
02-12-2018 02:43:59.049 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
02-12-2018 02:43:59.058 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
02-12-2018 02:43:59.067 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
02-12-2018 02:44:09.087 +0000 INFO DC:HandshakeReplyHandler - Handshake done.
02-12-2018 03:01:12.814 +0000 INFO DeployedApplication - Checksum mismatch 0 <> 594566478266413569 for app=_server_app_bluecoat. Will reload from='10.27.22.218:8089/services/streams/deployment?name=default:bluecoat_ftp:_server_app_bluec
oat'
02-12-2018 03:01:12.896 +0000 INFO DeployedApplication - Downloaded url=10.27.22.218:8089/services/streams/deployment?name=default:bluecoat_ftp:_server_app_bluecoat to file='/opt/splunkforwarder/var/run/bluecoat_ftp/_server_app_bluecoat
-1518404458.bundle' sizeKB=10