The issue is definitevely that i have to add some indexers and maybe also 1 or 2 SH to cluster. Infrastructure is currently undersized, it can't manage all actual data and jobs. Due to a very high ...
See more...
The issue is definitevely that i have to add some indexers and maybe also 1 or 2 SH to cluster. Infrastructure is currently undersized, it can't manage all actual data and jobs. Due to a very high data burst during office time (9 to 17), delays (for very very massive log files) and cpu saturation indexers side, infrastructure can't manage all data/users interaction/scheduled jobs all at once. So Indexers stop responding durings some times. Pipelines is 1, if i raise it to 2 System collapses. Monitoring.Console delined some heavy queries during that time.range that also writes directly on some indexes. But i have my own Dashboard Monitoring.Console on SHs that delines a strong delay for heavy logs (from 15 to 90 minutes before they reach the 0 minutes delay and indexes can write queues), some blocked queues (i have 1000MB size for many queue set) and all that can the easily delines a collapsing infrastructure 🤷 Infrastructure grew last months, so it's time to add some servers. I began with a 2 Indexers, then 4, now i really have to go to 8/12. Also Splunk Best-Practices suggests a 12 Indexers Infrastructure for my actual data flow (2-3 TB per day). Meanwhile, i fixed actual situation disabling heavy logs and heavy jobs on SHs 🤷 i also lowered the thruput for UFs, from maximum to 10MB/s. System works, but disabling some features and data. Thanks all.