Splunk Search

Datamodel Search "Job terminated unexpectedly"

vsommer
Explorer

Hello,

I have a problem regarding a datamodel search.

My datamodel consists of different boolean values with a span of 5s. The summary range of the data model is defined for two weeks.

I want to walk through the data and count events which contain the value "true" (GROUPBY _time span=5s ....). When I'm running the search with earliest=-2d@d I get an error "Job terminated unexpectedly". In the search.log I found "ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!"

When I'm running the same search for earliest=-1d@d I don't get an error. In the OS logs of my server I've noticed that the cpu nearly reaches 100% of the capacity.

THP is disabled.

Thank you 🙂

Labels (1)
Tags (2)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

I've actually ran into this myself, the hard way.


It sounds like your datamodel acceleration either didn't complete or isn't configured correctly.
When using acceleration, if a search reaches out beyond the date/time range (even a few seconds) of the accelerated results the search will revert to searching the index instead. If your index is large and/or your search will return a lot of events, that can cause the issue you're seeing.

Other causes can be including a field from the index in your search that is not part of the datamodel, or your search hitting a date/time range that has not yet been accelerated (meaning acceleration is working but still in progress). Both will trigger the same end result.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

Thanks for your answer but unfortunately this doesn't solve my problem. When I remove some fields from my tstats search the search is successful.

My datamodel contains 178 fields. Is it possible that there are restrictions regarding the limit of a datamodel?

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Did you check the status of the acceleration before and after you removed the fields? And do those fields contain a lot of data?

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

I only removed the fields from my search, not from the datamodel. The acceleration status is 100% completed. The size on disk is only about a few MB...

0 Karma

codebuilder
SplunkTrust
SplunkTrust

That's strange. Are the fields you removed from the search part of the datamodel?

Also, there are some aggregation restrictions regarding tstats and datamodels.
Without seeing your search it's difficult to tell, but it might be worth reviewing this documentation:

https://docs.splunk.com/Documentation/Splunk/8.2.2/SearchReference/Tstats#Complex_aggregate_function...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

vsommer
Explorer

That's strange. Are the fields you removed from the search part of the datamodel?



Yes..

Thanks for your help! I will look at the documentation.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...