Any of you guys upgraded / installed Splunk Enterprise 7.1 / 7.1.1 and have extensive use of accelrated datamodels ? Fx Enterprise Security 5.1 ?
We have seen a jump in load to 100% CPU usage at the time of upgrade. (And yes this is several weekas ago and the datamodels have been rebuild) And we run on 32 core machines that used to have a load of about 5-6, jumped to a load of 30-31 and has not come down since, neither via rebuild datamodels manually, or by rebooting. Only by disabling datamodel acceleration it helped, which is not doable when running ES.
We see many errors on all datamodel accerelerations when looking in the search.log in the dispatch catalogue on our indexeres.
ERROR BucketSummaryProcessor - Timed out waiting for bucket summary creation thread
Any of you guys have seen same issues ?
We have a support case running wil support also, but we haven't got any reply yet on these errors.
There is a known issue with Data Model Acceleration and the backfill time.
acceleration.backfill_time = (some time no earlier than data in the index)
This can be set in the CIM setup in Enterprise Security or in the datamodels.conf in etc/apps/Splunk_SA_CIM/local
What mitigated this problem for us, was to add
acceleration.backfill_time to all accelerated data models (under the DM stanza in datamodels.conf). Set this to <=
acceleration.earliest_time. (I assume this is only a issue with 7.1.1).