Hello everyone, I recently migrated from and old running hardware to a newer hardware and started to index the same that that I was indexing before (njmon - json version of nmon). In the old infra, same Splunk version running on both environments, it has no big issues and it has lower capacity than in the new one. Also, in the newer one, we're using faster disks (nmvme). After 1-3 days injecting njmon data, the indexer crashes and during this time, I can see a lot of splunkd recover-metada processes and also splunkd fsck --log-to--splunkd-log repair processes: [root@splunk]# ps -ef | grep splunkd splunk 21828 16396 99 12:00 ? 02:59:44 splunkd fsck --log-to--splunkd-log repair --try-warm-then-cold --one-bucket --index-name=njmon --bucket-name=db_1683179867_1683170671_48 --bloomfilter-only splunk 21829 21828 0 12:00 ? 00:00:00 splunkd fsck --log-to--splunkd-log repair --try-warm-then-cold --one-bucket --index-name=njmon --bucket-name=db_1683179867_1683170671_48 --bloomfilter-only splunk 41284 16396 99 12:20 ? 02:40:30 splunkd recover-metadata /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683195586_1683179630_51 --handle-roll njmon /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683195586_1683179630_51 --write-level 4 --tsidx-target-size 1572864000 --msidx-comp-block-size 1024 splunk 80067 16396 99 12:59 ? 02:00:53 splunkd recover-metadata /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683197989_1683180366_54 --handle-roll njmon /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683197989_1683180366_54 --write-level 4 --tsidx-target-size 1572864000 --msidx-comp-block-size 1024 splunk 136806 16396 99 13:44 ? 01:16:45 splunkd recover-metadata /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683200654_1683180434_53 --handle-roll njmon /net/splunk/fs0/splunk-hotwarm/njmon/db/db_1683200654_1683180434_53 --write-level 4 --tsidx-target-size 1572864000 --msidx-comp-block-size 1024   The server is running RHEL 8, 128 RAM, 48 Physical Procs, 96 Logical. Splunk Version: Splunk 8.2.10 (build 417e74d5c950)   The difference between old infra and new one is the tsidxlevel. In the old infra, we're using 2, in the newer one, we're using 4, but all the environments using the data are using the version greater or equal to 8.2.   Any hints from the community? ... View more

Hello, This is my first question here, since I don't know how to look for the solution. I tried to resolve this case on the past three days, but I was not able to. From the NJMON (JSON format for NMON) I got some multilevel fields and I'm trying to perform a graph to display the cpu utilization of sys, user, wait versus total (sys+wait+idle+user). The problem is not with the math, but how to group this information by CPU number level. This is how I got the structured data from NJMON: cpu_physical: { [-]      cpu0: { [-]        idle: 0        sys: 0        user: 0        wait: 0      }      cpu1: { [-]        idle: 0        sys: 0        user: 0        wait: 0      }      cpu10: { [-]        idle: 0        sys: 0        user: 0        wait: 0      } So, basically, I have the field: cpu_physical.cpu0.idle, cpu_physical.cpu0.sys, cpu_physical.cpu0.wait, cpu_physical.cpu0.user, cpu_physical.cpu1.idle, cpu_physical.cpu1.wait, cpu_physical.cpu1.sys, cpu_physical.cpu1.user ... I could have in the same System 4 CPUs to 64 CPUs, so I cannot perform that only using | eval CPU0.idle+CPU0.wait+.... Basically, this is what I need as the final output in a table format: CPU#  (Idle)  (Wait)  (Sys)  (user) CPU0   10%  12%     13%    16% CPU1  15%   15%     44%     67% CPU2    XX%  16%     X%     X% CPU3  XY%     X%       ...          ...     So, with this results, I would be able to calculate the Wait+User+System / Wait+User+System+Idle per CPU.   Any hints or ideas about how to make this? Sorry if the text is quite confusing, I don't know how to explain it shortly and maybe this is why I was not able to find the solution.   Thank you very much!     ... View more