indexer has stopped working

[Orange_girl]

Hello Splunk community, 

One of my indexes doesn't seem to have indexed any data for the last two weeks or so. This is the logs I see when searching for index="_internal" index_name:

26/05/2024 02:19:36.947 // 05-26-2024 02:19:36.947 -0400 INFO Dashboard - group=per_index_thruput, series="index_name", kbps=7940.738, eps=17495.842, kb=246192.784, ev=542437, avg_age=0.039, max_age=1

26/05/2024 02:19:07.804 // 05-26-2024 02:19:07.804 -0400 INFO DatabaseDirectoryManager [12112 IndexerService] - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/…/db duration=0.013

26/05/2024 02:19:07.799 // 05-26-2024 02:19:07.799 -0400 INFO DatabaseDirectoryManager [12112 IndexerService] - idx=index_name writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/…/db' pendingBucketUpdates=0 innerLockTime=0.009. Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'

26/05/2024 02:19:05.944 // 05-26-2024 02:19:05.944 -0400 INFO Dashboard - group=per_index_thruput, series="index_name", kbps=10987.030, eps=24200.033, kb=340566.581, ev=750132, avg_age=0.032, max_age=1

26/05/2024 02:18:59.981 // 05-26-2024 02:18:59.981 -0400 INFO LicenseUsage - type=Usage s="/opt/splunk/etc/apps/…/…/ABC.csv" st="name" h=host o="" idx="index_name" i="41050380-CA05-4248-AFCA-93E310A1E6A9" pool="auto_generated_pool_enterprise" b=6343129 poolsz=5368709120

What could be a reason for this and how could I address it? Thank you for all your help!

[gcusello]

Hi @Orange_girl ,

please check the time format of your timestamps: maybe they are in european format (dd/mm/yyyy) and you didn't configured TIME_FORMAT in your sourcetype definition, so Splunk uses the american format (mm/dd/yyyy).

Ciao.

Giuseppe

[Orange_girl]

Hi Giuseppe, 

I haven't changed anything in SPLUNK and the indexing used to work well, would this just randomly change by itself? 

I'm happy to check it though, could you let me know where and what I should be looking for? Are you referring to the time value in logs?

thank you. 

[gcusello]

Hi @Orange_girl ,

check if you received logs until the 31st of May, if yes and data flow stopped at 1st of June, check the timestamp format because probably you missed a configuration, but until the 31st of May you didn't discover it.

the check the time forma of your data.

Ciao.

Giuseppe 

[Orange_girl]

Thanks Giuseppe. The logs I shared here are the last logs I received for this index. 

I also checked logs for ABC.csv which is used by the index, and same here - logs only until May 26th:

26/05/2024 02:19:39.647 // 05-26-2024 02:19:39.647 -0400 WARN TailReader [12321 tailreader0] - Access error while handling path: failed to open for checksum: '/opt/splunk/etc/apps/.../.../ABC.csv' (No such file or directory)

26/05/2024 02:19:38.208 // 05-26-2024 02:19:38.208 -0400 INFO WatchedFile [12321 tailreader0] - Will begin reading at offset=0 for file='/opt/splunk/etc/apps/.../.../ABC.csv'.

26/05/2024 02:19:38.208 // 05-26-2024 02:19:38.208 -0400 INFO WatchedFile [12321 tailreader0] - Checksum for seekptr didn't match, will re-read entire file='/opt/splunk/etc/apps/.../.../ABC.csv'.

26/05/2024 02:19:37.621 // 05-26-2024 02:19:37.621 -0400 WARN TailReader [12321 tailreader0] - Insufficient permissions to read file='/opt/splunk/etc/apps/.../.../ABC' (hint: No such file or directory , UID: 0, GID: 0).

26/05/2024 02:19:37.512 // 05-26-2024 02:19:37.512 -0400 INFO WatchedFile [12321 tailreader0] - Will begin reading at offset=0 for file='/opt/splunk/etc/apps/.../.../ABC.csv'.

26/05/2024 02:19:37.512 // 05-26-2024 02:19:37.512 -0400 WARN LineBreakingProcessor [12299 parsing] - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 50968856 - data_source="/opt/splunk/etc/apps/.../.../ABC.csv", data_host="host", data_sourcetype="sourcetype"

26/05/2024 02:19:37.512 // 05-26-2024 02:19:37.512 -0400 INFO WatchedFile [12321 tailreader0] - Will begin reading at offset=0 for file='/opt/splunk/etc/apps/.../.../ABC.csv'.

26/05/2024 02:19:37.143 // 05-26-2024 02:19:37.143 -0400 WARN LineBreakingProcessor [12299 parsing] - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 50276856 - data_source="/opt/splunk/etc/apps/.../.../ABC.csv", data_host="host", data_sourcetype="sourcetype"

26/05/2024 02:19:36.947 // 05-26-2024 02:19:36.947 -0400 INFO Dashboard - group=per_source_thruput, series="/opt/splunk/etc/apps/.../.../ABC.csv", kbps=219.057, eps=482.877, kb=6791.592, ev=14971, avg_age=0.000, max_age=0

Would this be of any help?

[gcusello]

Hi @Orange_girl ,

it seems that something changed: Splunk hasn't more the requested permissions on the files to read: check them.

Ciao.

Giuseppe

[Orange_girl]

I haven't been able to look into this as much as I'd like, however over the past 2 weeks this has randomly worked couple of times - no errors and no issues. I still don't understand how it can complain about not having the right permissions and then suddenly work well the very next day to only again give the errors 2 days later.... 

[isoutamo]

Hi

as @gcusello said there are issues with file permissions.

You should check that those files are owned by your splunk user (usually splunk). Those can be changed e.g. if someone has restarted splunk as root user etc.

One other option is that your file system has remounted as RO due to some OS/storage level issue. Check also this and fix if needed.

r. Ismo