Splunk Search

Datamodel Search "Job terminated unexpectedly"

vsommer
Explorer

Hello,

I have a problem regarding a datamodel search.

My datamodel consists of different boolean values with a span of 5s. The summary range of the data model is defined for two weeks.

I want to walk through the data and count events which contain the value "true" (GROUPBY _time span=5s ....). When I'm running the search with earliest=-2d@d I get an error "Job terminated unexpectedly". In the search.log I found "ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!"

When I'm running the same search for earliest=-1d@d I don't get an error. In the OS logs of my server I've noticed that the cpu nearly reaches 100% of the capacity.

THP is disabled.

Thank you 🙂

Labels (1)
Tags (2)
0 Karma

codebuilder
Influencer

I've actually ran into this myself, the hard way.


It sounds like your datamodel acceleration either didn't complete or isn't configured correctly.
When using acceleration, if a search reaches out beyond the date/time range (even a few seconds) of the accelerated results the search will revert to searching the index instead. If your index is large and/or your search will return a lot of events, that can cause the issue you're seeing.

Other causes can be including a field from the index in your search that is not part of the datamodel, or your search hitting a date/time range that has not yet been accelerated (meaning acceleration is working but still in progress). Both will trigger the same end result.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

Thanks for your answer but unfortunately this doesn't solve my problem. When I remove some fields from my tstats search the search is successful.

My datamodel contains 178 fields. Is it possible that there are restrictions regarding the limit of a datamodel?

0 Karma

codebuilder
Influencer

Did you check the status of the acceleration before and after you removed the fields? And do those fields contain a lot of data?

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

I only removed the fields from my search, not from the datamodel. The acceleration status is 100% completed. The size on disk is only about a few MB...

0 Karma

codebuilder
Influencer

That's strange. Are the fields you removed from the search part of the datamodel?

Also, there are some aggregation restrictions regarding tstats and datamodels.
Without seeing your search it's difficult to tell, but it might be worth reviewing this documentation:

https://docs.splunk.com/Documentation/Splunk/8.2.2/SearchReference/Tstats#Complex_aggregate_function...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

That's strange. Are the fields you removed from the search part of the datamodel?



Yes..

Thanks for your help! I will look at the documentation.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!