topic Re: How can I get more than 10,000 lines into a single event? in Getting Data In

How can I get more than 10,000 lines into a single event?

payal23 — Tue, 10 Apr 2018 11:54:33 GMT

I want more than 10,000 lines to merge and show in a single event.

[tally_nightly_prd]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
TRUNCATE=0
disabled=false
BREAK_ONLY_BEFORE=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
MAX_EVENTS=90000
TIME_FORMAT=%+
TIME_PREFIX=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted

Re: How can I get more than 10,000 lines into a single event?

FrankVl — Tue, 10 Apr 2018 12:24:05 GMT

And what exactly is your question? Is your current config not working as expected? If so: what is the expected outcome and what outcome are you now getting?

Also a bit more context around the data you're ingesting and what you are trying to achieve would probably help 🙂

Re: How can I get more than 10,000 lines into a single event?

payal23 — Tue, 10 Apr 2018 13:01:36 GMT

In between my file start and end points there are number of lines in between which is more than 10,000 and i want all the lines to come under one event. But the breaking is not happening in that way. In mid it is breaking anywhere.

Re: How can I get more than 10,000 lines into a single event?

FrankVl — Tue, 10 Apr 2018 13:32:49 GMT

And how are you collecting this data? With a HF or a UF and how/where is it then forwarded?

Re: How can I get more than 10,000 lines into a single event?

somesoni2 — Tue, 10 Apr 2018 18:47:31 GMT

Just want to make sure you're aware that having that many line in a single event will not give you a pleasant Splunk UI experience when viewing the same. Assuming you still want to do it, give this a try

 [tally_nightly_prd]
 SHOULD_LINEMERGE=false
 LINE_BREAKER = ([\r\n]+)(?=(\*){12}\snightlyProcess\sStarted)
 TRUNCATE=0
 MAX_EVENTS=90000
 TIME_FORMAT=%+
 TIME_PREFIX=^\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted

Re: How can I get more than 10,000 lines into a single event?

payal23 — Wed, 11 Apr 2018 07:21:21 GMT

We are collecting from UF

Re: How can I get more than 10,000 lines into a single event?

FrankVl — Wed, 11 Apr 2018 07:42:31 GMT

And is that UF sending to a single indexer/HF or to a load balanced pool of destinations (e.g. indexer cluster, multiple intermediate forwarders...)?

Re: How can I get more than 10,000 lines into a single event?

payal23 — Mon, 16 Apr 2018 01:12:26 GMT

Thanks...Yes, logs are having big xml payload and hence merging in an event will make sense.

I tried the above but now the lines are breaking in single line.

😞

Re: How can I get more than 10,000 lines into a single event?

payal23 — Mon, 16 Apr 2018 01:12:58 GMT

Sending to indexer cluster

Re: How can I get more than 10,000 lines into a single event?

manishankark04 — Thu, 16 Apr 2020 20:03:04 GMT

you can increase the truncate parameter to 40k or 50k.

Re: How can I get more than 10,000 lines into a single event?

jinseong — Fri, 17 Apr 2020 06:46:07 GMT

hello

open the limits.conf and configration maxchars=10240