Datamodel Search "Job terminated unexpectedly"

[vsommer]

Hello,

I have a problem regarding a datamodel search.

My datamodel consists of different boolean values with a span of 5s. The summary range of the data model is defined for two weeks.

I want to walk through the data and count events which contain the value "true" (GROUPBY _time span=5s ....). When I'm running the search with earliest=-2d@d I get an error "Job terminated unexpectedly". In the search.log I found "ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!"

When I'm running the same search for earliest=-1d@d I don't get an error. In the OS logs of my server I've noticed that the cpu nearly reaches 100% of the capacity.

THP is disabled.

Thank you 🙂

[codebuilder]

I've actually ran into this myself, the hard way.

It sounds like your datamodel acceleration either didn't complete or isn't configured correctly.
When using acceleration, if a search reaches out beyond the date/time range (even a few seconds) of the accelerated results the search will revert to searching the index instead. If your index is large and/or your search will return a lot of events, that can cause the issue you're seeing.

Other causes can be including a field from the index in your search that is not part of the datamodel, or your search hitting a date/time range that has not yet been accelerated (meaning acceleration is working but still in progress). Both will trigger the same end result.

[vsommer]

Thanks for your answer but unfortunately this doesn't solve my problem. When I remove some fields from my tstats search the search is successful.

My datamodel contains 178 fields. Is it possible that there are restrictions regarding the limit of a datamodel?

[codebuilder]

Did you check the status of the acceleration before and after you removed the fields? And do those fields contain a lot of data?

[vsommer]

I only removed the fields from my search, not from the datamodel. The acceleration status is 100% completed. The size on disk is only about a few MB...

[codebuilder]

That's strange. Are the fields you removed from the search part of the datamodel?

Also, there are some aggregation restrictions regarding tstats and datamodels.
Without seeing your search it's difficult to tell, but it might be worth reviewing this documentation:

https://docs.splunk.com/Documentation/Splunk/8.2.2/SearchReference/Tstats#Complex_aggregate_function...

[vsommer]

---

That's strange. Are the fields you removed from the search part of the datamodel?

---

---

Yes..

Thanks for your help! I will look at the documentation.