Errors casuing unusable Indexer Linux VMs, slowdowns for entire vcenter cluster until powered off

mevangelisti — Wed, 30 Sep 2020 05:24:05 GMT

We are seeing two types of dmesg errors on the linux VMs which are acting as our indexers:

1) “task blocked for more than 120 seconds, hung_task_timeout_secs, call trace” - https://del.dog/120blk.txt

2) “sd 0:0:0:0: [sda] task abort on host 0” - https://del.dog/tskabrt.txt

These issues seem to be also causing high I/O for all VMs on the same vcenter cluster that the indexers are on. As soon as we get the indexers powered down, performance is restored across the cluster.

Things I've tried:
-Full yum update
-Disabling Huge Pages
-Tried booting with older kernel two versions back, and latest ones.
-Initially we were on RDM backed by EMC VNX SAN for hot space. This was converted to VMDK (still backed by VNX).
-Initially the hot drives were thin. They were converted to Eager Thick.
-Initially the hot drives were formatted with XFS. I have migrated them to EXT4.
-I tried tuning system cacheing / flushes per this explanation: https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue--fix-hung_task_timeout_secs-blocked-120-seconds-problem/

Re: Errors casuing unusable Indexer Linux VMs, slowdowns for entire vcenter cluster until powered off

williaml_splunk — Thu, 27 Aug 2020 05:44:05 GMT

Is it helpful to resolve this blocked issue with the two parameters?

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

topic Errors casuing unusable Indexer Linux VMs, slowdowns for entire vcenter cluster until powered off in Getting Data In

Errors casuing unusable Indexer Linux VMs, slowdowns for entire vcenter cluster until powered off

Re: Errors casuing unusable Indexer Linux VMs, slowdowns for entire vcenter cluster until powered off