topic Re: Datamodel Search "Job terminated unexpectedly" in Splunk Search

Datamodel Search "Job terminated unexpectedly"

vsommer — Thu, 26 Aug 2021 15:16:10 GMT

Hello,

I have a problem regarding a datamodel search.

My datamodel consists of different boolean values with a span of 5s. The summary range of the data model is defined for two weeks.

I want to walk through the data and count events which contain the value "true" (GROUPBY _time span=5s ....). When I'm running the search with earliest=-2d@d I get an error "Job terminated unexpectedly". In the search.log I found "ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!"

When I'm running the same search for earliest=-1d@d I don't get an error. In the OS logs of my server I've noticed that the cpu nearly reaches 100% of the capacity.

THP is disabled.

Thank you 🙂

Re: Datamodel Search "Job terminated unexpectedly"

codebuilder — Thu, 26 Aug 2021 16:18:33 GMT

I've actually ran into this myself, the hard way.

It sounds like your datamodel acceleration either didn't complete or isn't configured correctly.
When using acceleration, if a search reaches out beyond the date/time range (even a few seconds) of the accelerated results the search will revert to searching the index instead. If your index is large and/or your search will return a lot of events, that can cause the issue you're seeing.

Other causes can be including a field from the index in your search that is not part of the datamodel, or your search hitting a date/time range that has not yet been accelerated (meaning acceleration is working but still in progress). Both will trigger the same end result.

Re: Datamodel Search "Job terminated unexpectedly"

vsommer — Mon, 13 Sep 2021 15:19:10 GMT

Thanks for your answer but unfortunately this doesn't solve my problem. When I remove some fields from my tstats search the search is successful.

My datamodel contains 178 fields. Is it possible that there are restrictions regarding the limit of a datamodel?

Re: Datamodel Search "Job terminated unexpectedly"

codebuilder — Mon, 13 Sep 2021 18:30:07 GMT

Did you check the status of the acceleration before and after you removed the fields? And do those fields contain a lot of data?

Re: Datamodel Search "Job terminated unexpectedly"

vsommer — Tue, 14 Sep 2021 06:50:56 GMT

I only removed the fields from my search, not from the datamodel. The acceleration status is 100% completed. The size on disk is only about a few MB...

Re: Datamodel Search "Job terminated unexpectedly"

codebuilder — Tue, 14 Sep 2021 15:28:48 GMT

That's strange. Are the fields you removed from the search part of the datamodel?

Also, there are some aggregation restrictions regarding tstats and datamodels.
Without seeing your search it's difficult to tell, but it might be worth reviewing this documentation:

https://docs.splunk.com/Documentation/Splunk/8.2.2/SearchReference/Tstats#Complex_aggregate_functions

Re: Datamodel Search "Job terminated unexpectedly"

vsommer — Wed, 15 Sep 2021 07:20:19 GMT

That's strange. Are the fields you removed from the search part of the datamodel?

Yes..

Thanks for your help! I will look at the documentation.