Deployment Architecture

Why aren't my logs being forwarded for indexing by my forwarders?

markconlin
Path Finder

Summary
Not all logs are being forwarded for indexing by my splunkforwarders.

Situation
I have 4 instances that run 3 processes I am interesting in.
Each process outputs logs that I am forwarding to Splunk via a splunkforwarder on the instance.
These logs are rotated by logrotate.d.
On some instances all logs are being forwarded, on some instances only some logs are being forwarded.

Problems
I believe the relevant error from the logs is this one (others below):

splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Possible Solutions?
- Increase the initCrcLen?
- WHAT ELSE SHOULD I TRY?
- DO THE OTHER ERRORS IN THE LOG MATTER (ERROR JsonLineBreaker or ERROR TcpOutputProc)

Reference

Splunk Forwarder Config

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
[default]
host = bowman-<myserver>1

[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
disabled = 0
sourcetype = boeinglogjson
index = prod-boeing

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk.myotherserver.com:9997

[tcpout-server://splunk.myotherserver.com:9997]
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat props.conf
TRUNCATE = 2000000

[boeinglogjson]
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = info.created
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q
category = Custom
disabled = false

Other errors from Splunk Logs

Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/var/log/splunk# grep ERROR *.log
splunkd.log:09-21-2017 23:44:53.585 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_booking-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-26-2017 08:51:46.621 +0000 ERROR JsonLineBreaker - JSON StreamId:11681658046189288813 had parsing error:String value too long - data_source="/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_default-1.log", data_host="bowman-<myserver>1", data_sourcetype="boeinglogjson"

Example logrotate.d conf..

/<log file path and name>/
{
  size 250M
  missingok
  rotate 3
  compress
  delaycompress
  notifempty
  create 664 ubuntu ubuntu
  su ubuntu ubuntu
  sharedscripts
  postrotate
  service bowman-worker_booking-1 restart
  endscript
}
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @markconlin,

If you have same header (First 256 bytes) in all files which splunkforwarder is trying to read then please change inputs.conf on your forwarder something like this.

[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
 disabled = 0
crcSalt = <SOURCE>
 sourcetype = boeinglogjson
 index = prod-boeing

Additionally props.conf which you have defined on splunk universal forwarder will not work because splunk universal forwarder will not do any parsing those parsing will be done by either heavy forwarder (if you are using it) or indexer. So apply that props.conf configuration on heavy forwarder or indexer whichever comes first in your environment.

I hope this helps.

Thanks,
Harshil

0 Karma

markconlin
Path Finder

@harsmarvania57 I have concerns with crcSalt and log rotate, because of this comment in the Splunk documentation:

Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed
after it has rolled.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

In this case you need to increase initCrcLength so that splunk forwarder will indentify that these are different file. It looks like in current scenario first 256 bytes of all your log files are same.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...