When I attempt to enter very large queries into the search bar I get errors in chrome and eventually a "disconnected" from Splunk popup. I do not believe the search is ever started as I do not see it in the "Jobs Manager". Steps to reproduce: - enter a large query into the search bar, mine is 160 lines and 10.4k characters. - hit enter key - errors appear, search is not started, soon "Disconnected from Splunk" popup will appear. A split second after hitting enter... these errors appears in the chrome debugger: DELETE https://splunk.myhost.com/en-US/splunkd/__raw/services/search/jobs/rt_md_1516156026.353?output_mode=json (failed) net::ERR_CONNECTION_CLOSED GET https://splunk.myhost.com/en-US/splunkd/__raw/servicesNS/admin/search/saved/searches/_new?output_mode=json&_=1516156026267 (failed) net::ERR_CONNECTION_CLOSED GET https://splunk.myhost.com/en-US/splunkd/__raw/services/saved/searches/_new?output_mode=json&_=1516156026268 (failed) net::ERR_CONNECTION_CLOSED POST https://splunk.myhost.com/en-US/splunkd/__raw/servicesNS/admin/search/search/ast (failed) net::ERR_CONNECTION_CLOSED and so on.... Errors in Chrome Debugger (this happens right away) ...and then eventually disconnected from Splunk ... View more

Summary Not all logs are being forwarded for indexing by my splunkforwarders. Situation I have 4 instances that run 3 processes I am interesting in. Each process outputs logs that I am forwarding to Splunk via a splunkforwarder on the instance. These logs are rotated by logrotate.d. On some instances all logs are being forwarded, on some instances only some logs are being forwarded. Problems I believe the relevant error from the logs is this one (others below): splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. Possible Solutions? - Increase the initCrcLen? - WHAT ELSE SHOULD I TRY? - DO THE OTHER ERRORS IN THE LOG MATTER (ERROR JsonLineBreaker or ERROR TcpOutputProc) Reference Splunk Forwarder Config Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat inputs.conf [default] host = bowman-<myserver>1 [monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log] disabled = 0 sourcetype = boeinglogjson index = prod-boeing Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat outputs.conf [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = splunk.myotherserver.com:9997 [tcpout-server://splunk.myotherserver.com:9997] Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat props.conf TRUNCATE = 2000000 [boeinglogjson] INDEXED_EXTRACTIONS = json NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIMESTAMP_FIELDS = info.created TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q category = Custom disabled = false Other errors from Splunk Logs Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/var/log/splunk# grep ERROR *.log splunkd.log:09-21-2017 23:44:53.585 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf. splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_booking-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. splunkd.log:09-26-2017 08:51:46.621 +0000 ERROR JsonLineBreaker - JSON StreamId:11681658046189288813 had parsing error:String value too long - data_source="/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_default-1.log", data_host="bowman-<myserver>1", data_sourcetype="boeinglogjson" Example logrotate.d conf.. /<log file path and name>/ { size 250M missingok rotate 3 compress delaycompress notifempty create 664 ubuntu ubuntu su ubuntu ubuntu sharedscripts postrotate service bowman-worker_booking-1 restart endscript } ... View more

Objective Using the Splunk Add-on for Amazon Web Services to ingest events from AWS Kinesis with a custom sourcetype. Issue Ingested json event fields are not extracted when using custom sourcetype. What I have tried I have created Kinesis inputs to read from the stream. One with the sourcetype = aws:kinesis (as specified in the documentation here http://docs.splunk.com/Documentation/AddOns/released/AWS/Kinesis) and one with a custom sourcetype. The custom sourcetype events do not have extracted json fields. (see picture attached). The standard sourceytpe events do have extracted json fields. I have tested this sourcetype using oneshot to place json data into a test index and the fields were extracted correctly. Create indices /opt/splunk/bin# ./splunk add index fromkinesis /opt/splunk/bin# ./splunk add index bythebookkn /opt/splunk/bin# ./splunk add index oneshottest Test sourcetype with oneshot /opt/splunk/bin# ./splunk add oneshot /opt/splunk/data/test.json -sourcetype myevents -index oneshottest Kinesis inputs /opt/splunk/etc/apps/Splunk_TA_aws/local# cat aws_kinesis_tasks.conf [bythebookkn] account = splunk encoding = format = CloudWatchLogs index = bythebookkn init_stream_position = LATEST region = us-east-1 sourcetype = aws:kinesis stream_names = stage-my-events [fromkinesis] account = splunk encoding = format = CloudWatchLogs index = fromkinesis init_stream_position = LATEST region = us-east-1 sourcetype = myevents stream_names = stage-my-events Sourcetype /opt/splunk/etc/system/local# cat props.conf TRUNCATE = 800000 [myevents] INDEXED_EXTRACTIONS = json TIMESTAMP_FIELDS = info.created TIME_FORMAT = %Y-%d-%m %H:%M:%S.%3Q TZ = UTC detect_trailing_nulls = auto SHOULD_LINEMERGE = false KV_MODE = none AUTO_KV_JSON = false category = Custom disabled = false ... View more