topic Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded in Getting Data In

Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Mon, 28 Sep 2020 16:54:31 GMT

Hi Team,

I was indexing the WebSphere logs into SPLUNK, all of a sudden it stopped indexing. When I looked into the logs found the below error:

06-23-2014 10:10:12.855 -0400 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="/opt/IBM/WebSphereND64/AppServer/profiles/AppSrv01/logs/JVM2/SystemOut_14.06.23_10.10.12.log", data_host="SEP01XVP-004", data_sourcetype="systemout"
06-23-2014 10:10:12.856 -0400 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Jun 23 01:15:17 2014). Context: source::/opt/IBM/WebSphereND64/AppServer/profiles/AppSrv01/logs/JVM2/SystemOut_14.06.23_10.10.12.log|host::SEP01XVP-004|systemout|430

How can I overcome this? How should I make the websphere logs back to indexing.
Kindly help on priority basis.

Regards,
Sushma.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Mon, 28 Sep 2020 16:54:45 GMT

If you can provide an example event, we can help you more.
You can try to use the following settings in props.conf (edit / adjust for ur env.) for the affected sourcetype
TIME_PREFIX = TIMESTAMP=
MAX_TIMESTAMP_LOOKAHEAD = 25
MAX_EVENTS =

and if you know the time format you can also specify this with;
TIME_FORMAT =

The docs is here; http://docs.splunk.com/Documentation

How to configure time stamps;
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Tue, 24 Jun 2014 07:45:05 GMT

Hi,

I have included the lines MAX_EVENTS=10000 and TRUNCATE=0 in the props.conf(etc/system/local), assuming this would not break the events after reaching default value since I have set it to 10000, but this did not solve the issue, the websphere logs are not getting indexed and when I check the logs I find the same error as above even after setting the above values. Let me know if you need any more information from my side, to investigate/advise on the issue.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Tue, 24 Jun 2014 07:46:55 GMT

if you can give us (paste here) an event from your logs (well at lest the start) it would help us to help you.

Also read the docs, as it is described there how to solve this problem

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Mon, 28 Sep 2020 16:54:47 GMT

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Tue, 24 Jun 2014 08:21:57 GMT

The above mentioned lines is what I retrieved from my logs, the same lines re-appear number of times.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Tue, 24 Jun 2014 08:42:23 GMT

Well, this is your splunkd.log, what we need to see is your websphere log. To be able to determine timestamp / event breaking etc etc ..

ANd you should really read the doc that i linked for you 🙂

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Mon, 28 Sep 2020 16:54:50 GMT

the log that you are trying to index that is;

opt/IBM/WebSphereND64/AppServer/profiles/AppSrv01/logs/JVM2/SystemOut_14.06.23_10.10.12.log

There is suppose to be a log4j sourcetype, you could try to assign this sourcetype to the datainput instead of the (default one) "systemout" sourcetype

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

linu1988 — Tue, 24 Jun 2014 09:29:19 GMT

use
BREAK_ONLY_BEFORE=\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Mon, 28 Sep 2020 16:54:55 GMT

where should I use this line "BREAK_ONLY_BEFORE=\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}.\d{3}"? under inputs.conf of local?

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Tue, 24 Jun 2014 10:36:56 GMT

Hi lmyrefelt,
As said by you I shall try even changing the sourcetype, but just to give you an update that I am even collecting the same Websphere logs from another machine which works absolutely fine, the problem is while collecting the webshere logs from only one of the box.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Mon, 28 Sep 2020 16:54:57 GMT

@sushma7, BREAK_ONLY_BEFORE (as well as the other settings) should be in the props.conf file on your indexers. (please read the docs i linked to you for in depth details) .

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

lmyrefelt — Tue, 24 Jun 2014 12:45:03 GMT

@sushma7,

If you have it working on one machine, why don't you replicate the settings for that data-input ?

What your splunkd.log is saying to you is that; Splunk did not find (or reqnoize ) an timestamp in the indexed event and therefor it don't know how to break / format the given events. So if you have it working for one data-source / input you should be able to get it to work based on the settings from this one.

If your event starts with;
NN-NN-NNNN NN:NN:NN.NNN

24-06-2014 14:42:34.542

Then the setting from linu1988 would work .

And if this is how your timstamp looks like, you should be able to use

TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N

please check
http://strftime.net

and

http://man7.org/linux/man-pages/man3/strftime.3.html

good luck

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

yannK — Tue, 24 Jun 2014 17:06:16 GMT

Guys, "WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded"

Means that your multiline event has been in cut in chunks of 256 lines, because of the default limit.
see props.conf MAX_EVENTS=256

So usually it is followed by a warning that no timestamp was found on the second piece.
You can adapt your sourcetype, and maybe tune your timestamp extraction to improve it.

EDIT :
You can use this search to find the long events.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

sushma7 — Thu, 26 Jun 2014 16:58:07 GMT

Thanks for your replies guys...!!! Changing the MAX_EVENTS=10000 and TRUNCATE=0 in the props.conf file of the indexer and restarting the indexer has resolved has the issue.

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

yannK — Thu, 26 Jun 2014 18:48:29 GMT

beware, the TRUNCATE=0 may bite you if you have very bad events. You may want to have a real limit (default is 10000, why not use 100000 to start)

Re: Error in splunkd.log: Breaking event because limit of 256 has been exceeded

jrodman — Fri, 05 Jun 2015 08:24:31 GMT

Be sure you ACTUALLY have events over 256 lines long.