Why aren't my logs being forwarded for indexing by my forwarders?

markconlin — Thu, 28 Sep 2017 21:34:28 GMT

Summary
Not all logs are being forwarded for indexing by my splunkforwarders.

Situation
I have 4 instances that run 3 processes I am interesting in.
Each process outputs logs that I am forwarding to Splunk via a splunkforwarder on the instance.
These logs are rotated by logrotate.d.
On some instances all logs are being forwarded, on some instances only some logs are being forwarded.

Problems
I believe the relevant error from the logs is this one (others below):

splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Possible Solutions?
- Increase the initCrcLen?
- WHAT ELSE SHOULD I TRY?
- DO THE OTHER ERRORS IN THE LOG MATTER (ERROR JsonLineBreaker or ERROR TcpOutputProc)

Reference

Splunk Forwarder Config

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
[default]
host = bowman-<myserver>1

[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
disabled = 0
sourcetype = boeinglogjson
index = prod-boeing

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk.myotherserver.com:9997

[tcpout-server://splunk.myotherserver.com:9997]
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat props.conf
TRUNCATE = 2000000

[boeinglogjson]
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = info.created
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q
category = Custom
disabled = false

Other errors from Splunk Logs

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/var/log/splunk# grep ERROR *.log
splunkd.log:09-21-2017 23:44:53.585 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_booking-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-26-2017 08:51:46.621 +0000 ERROR JsonLineBreaker - JSON StreamId:11681658046189288813 had parsing error:String value too long - data_source="/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_default-1.log", data_host="bowman-<myserver>1", data_sourcetype="boeinglogjson"

Example logrotate.d conf..

/<log file path and name>/
{
  size 250M
  missingok
  rotate 3
  compress
  delaycompress
  notifempty
  create 664 ubuntu ubuntu
  su ubuntu ubuntu
  sharedscripts
  postrotate
  service bowman-worker_booking-1 restart
  endscript
}

Re: Why aren't my logs being forwarded for indexing by my forwarders?

harsmarvania57 — Fri, 29 Sep 2017 03:42:51 GMT

Hi @markconlin,

If you have same header (First 256 bytes) in all files which splunkforwarder is trying to read then please change inputs.conf on your forwarder something like this.

[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
 disabled = 0
crcSalt = <SOURCE>
 sourcetype = boeinglogjson
 index = prod-boeing

Additionally props.conf which you have defined on splunk universal forwarder will not work because splunk universal forwarder will not do any parsing those parsing will be done by either heavy forwarder (if you are using it) or indexer. So apply that props.conf configuration on heavy forwarder or indexer whichever comes first in your environment.

I hope this helps.

Thanks,
Harshil

Re: Why aren't my logs being forwarded for indexing by my forwarders?

markconlin — Fri, 29 Sep 2017 13:19:20 GMT

@harsmarvania57 I have concerns with crcSalt and log rotate, because of this comment in the Splunk documentation:

Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed
after it has rolled.

Re: Why aren't my logs being forwarded for indexing by my forwarders?

harsmarvania57 — Fri, 29 Sep 2017 13:28:55 GMT

In this case you need to increase initCrcLength so that splunk forwarder will indentify that these are different file. It looks like in current scenario first 256 bytes of all your log files are same.

topic Why aren't my logs being forwarded for indexing by my forwarders? in Deployment Architecture

Why aren't my logs being forwarded for indexing by my forwarders?

Re: Why aren't my logs being forwarded for indexing by my forwarders?

Re: Why aren't my logs being forwarded for indexing by my forwarders?

Re: Why aren't my logs being forwarded for indexing by my forwarders?