TailReader - Insufficient permissions - Reindexing

[tb5821]

TailReader - Insufficient permissions - errors in my logs - will splunk attempt to re-read those at some interval? thus far I only see it doing it once a few hours back and not since 😞

I also see several databaseDirectory events in the splunkd log that relates to the index that these logs should of went to so I'm not sure whats going on, perhaps just a delay?

00 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 19:43:49.481 +0000 INFO  HotBucketRoller - finished moving hot to warm bid=kinesis~20~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF idx=kinesis from=hot_v1_20 to=db_1590613020_1589312100_20 size=956243968 caller=size_exceeded _maxHotBucketSize=786432000 (750MB), bucketSize=1015918592 (968MB)
06-04-2020 19:43:49.483 +0000 INFO  IndexWriter - Creating hot bucket=hot_v1_21, idx=kinesis, event timestamp=1590429480, reason="suitable bucket not found, number of hot buckets=1, max=3; closest bucket localid=0, earliest=1577836800, latest=1577836800"
06-04-2020 19:43:49.484 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Adding bucket, bid=kinesis~21~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF'
06-04-2020 19:43:49.485 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 19:44:15.461 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
06-04-2020 19:44:15.463 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 19:44:16.399 +0000 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=kinesis~20~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF
06-04-2020 19:44:16.454 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=1 .  Reason='Updating manifest: bucketUpdates=1'
06-04-2020 19:44:16.458 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 20:22:02.413 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Updating bucket, bid=kinesis~21~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF'
06-04-2020 20:22:02.415 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 20:22:02.417 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Updating bucket, bid=kinesis~21~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF'
06-04-2020 20:22:02.418 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 20:22:02.419 +0000 INFO  HotBucketRoller - finished moving hot to warm bid=kinesis~21~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF idx=kinesis from=hot_v1_21 to=db_1590613020_1589312100_21 size=789688320 caller=size_exceeded _maxHotBucketSize=786432000 (750MB), bucketSize=789729280 (753MB)
06-04-2020 20:22:14.438 +0000 INFO  IndexWriter - Creating hot bucket=hot_v1_22, idx=kinesis, event timestamp=1590605700, reason="suitable bucket not found, number of hot buckets=1, max=3; closest bucket localid=0, earliest=1577836800, latest=1577836800"
06-04-2020 20:22:14.439 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Adding bucket, bid=kinesis~22~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF'
06-04-2020 20:22:14.440 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 20:22:18.375 +0000 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=kinesis~21~BC057F8A-75D0-4CDC-9BD0-EA5E0076B4AF
06-04-2020 20:22:18.455 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=1 .  Reason='Updating manifest: bucketUpdates=1'
06-04-2020 20:22:18.457 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db
06-04-2020 20:23:15.459 +0000 INFO  DatabaseDirectoryManager - idx=kinesis Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/kinesis/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
06-04-2020 20:23:15.460 +0000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/kinesis/db