topic Re: AggregatorMiningProcessor in Deployment Architecture

AggregatorMiningProcessor

nls21 — Wed, 28 Aug 2013 11:34:01 GMT

Any idea what this means or more to the point, why?? Came in to a dead Splunk...

WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded
-0400 FATAL ProcessRunner - Unexpected EOF from process runner child!
-0400 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!

Re: AggregatorMiningProcessor

rturk — Wed, 28 Aug 2013 13:37:05 GMT

Right... after a bit of searching I've come up with the following.

WARN  AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded

When parsing for multiline events, Splunk has an inbuilt limit of 256 lines. This has been exceeded, and this error was thrown, however I do not think this was the problem....

-0400 FATAL ProcessRunner - Unexpected EOF from process runner child!

I don't think we're seeing the full event here... signal 9 is a KILL signal from an external process. From the second reference below:

"It is likely that your OS has some
kind of monitor or other setting on it
that kills processes that do certain
things. Perhaps your administrator is
watching for memory usage, access to
certain files, or other things. You
should consult with your system admin
to find out what they have put in
place."

I hope this sets you on the right path.

REFERENCES:

Re: AggregatorMiningProcessor

nls21 — Mon, 28 Sep 2020 14:40:21 GMT

We see this:

Yes, the OS kernel decided to kill off splunkd, not a good choice.

Aug 27 16:46:35 splunk01 kernel: [58064] 10074 58064 911880 897929 0 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58065] 10074 58065 12792 1163 2 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58067] 10074 58067 12791 1163 2 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58070] 10074 58070 12792 1163 0 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [63522] 0 63522 1014 17 3 0 0 sleep
Aug 27 16:46:35 splunk01 kernel: [67229] 0 67229 1014 17 2 0 0 sleep
Aug 27 16:46:35 splunk01 kernel: Out of memory: Kill process 58064 (splunkd) score 439 or sacrifice child
Aug 27 16:46:35 splunk01 kernel: Killed process 58067, UID 10074, (splunkd) total-vm:51164kB, anon-rss:4648kB, file-rss:4kB
Aug 27 16:46:35 splunk01 kernel: splunkd invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0

Re: AggregatorMiningProcessor

rturk — Wed, 28 Aug 2013 13:52:48 GMT

Yep - Out of memory will do it 😞

If you could mark this question as answered, that will help other people who are searching for an answer to the same problem.

Good luck!

Re: AggregatorMiningProcessor

nls21 — Wed, 28 Aug 2013 14:41:09 GMT

Turns out we had some users running large, numerous and mixed with real-time searches. Evidently, this causes the OS to shut-down splunkd via OOM killer.