How can I get more than 10,000 lines into a single event?

[payal23]

I want more than 10,000 lines to merge and show in a single event.

[tally_nightly_prd]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
TRUNCATE=0
disabled=false
BREAK_ONLY_BEFORE=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
MAX_EVENTS=90000
TIME_FORMAT=%+
TIME_PREFIX=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted

[jinseong]

hello

open the limits.conf and configration maxchars=10240

[somesoni2]

Just want to make sure you're aware that having that many line in a single event will not give you a pleasant Splunk UI experience when viewing the same. Assuming you still want to do it, give this a try

 [tally_nightly_prd]
 SHOULD_LINEMERGE=false
 LINE_BREAKER = ([\r\n]+)(?=(\*){12}\snightlyProcess\sStarted)
 TRUNCATE=0
 MAX_EVENTS=90000
 TIME_FORMAT=%+
 TIME_PREFIX=^\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted

[payal23]

Thanks...Yes, logs are having big xml payload and hence merging in an event will make sense.

I tried the above but now the lines are breaking in single line.

😞

[manishankark04]

you can increase the truncate parameter to 40k or 50k.

[FrankVl]

And what exactly is your question? Is your current config not working as expected? If so: what is the expected outcome and what outcome are you now getting?

Also a bit more context around the data you're ingesting and what you are trying to achieve would probably help 🙂

[payal23]

In between my file start and end points there are number of lines in between which is more than 10,000 and i want all the lines to come under one event. But the breaking is not happening in that way. In mid it is breaking anywhere.

[FrankVl]

And how are you collecting this data? With a HF or a UF and how/where is it then forwarded?

[payal23]

We are collecting from UF

[FrankVl]

And is that UF sending to a single indexer/HF or to a load balanced pool of destinations (e.g. indexer cluster, multiple intermediate forwarders...)?

[payal23]

Sending to indexer cluster