Getting Data In

How can I get more than 10,000 lines into a single event?

Path Finder

I want more than 10,000 lines to merge and show in a single event.

[tally_nightly_prd]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
TRUNCATE=0
disabled=false
BREAK_ONLY_BEFORE=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
MAX_EVENTS=90000
TIME_FORMAT=%+
TIME_PREFIX=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
0 Karma

Path Finder

hello

open the limits.conf and configration maxchars=10240

0 Karma

SplunkTrust
SplunkTrust

Just want to make sure you're aware that having that many line in a single event will not give you a pleasant Splunk UI experience when viewing the same. Assuming you still want to do it, give this a try

 [tally_nightly_prd]
 SHOULD_LINEMERGE=false
 LINE_BREAKER = ([\r\n]+)(?=(\*){12}\snightlyProcess\sStarted)
 TRUNCATE=0
 MAX_EVENTS=90000
 TIME_FORMAT=%+
 TIME_PREFIX=^\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
0 Karma

Path Finder

Thanks...Yes, logs are having big xml payload and hence merging in an event will make sense.

I tried the above but now the lines are breaking in single line.

😞

0 Karma

New Member

you can increase the truncate parameter to 40k or 50k.

0 Karma

Ultra Champion

And what exactly is your question? Is your current config not working as expected? If so: what is the expected outcome and what outcome are you now getting?

Also a bit more context around the data you're ingesting and what you are trying to achieve would probably help 🙂

0 Karma

Path Finder

In between my file start and end points there are number of lines in between which is more than 10,000 and i want all the lines to come under one event. But the breaking is not happening in that way. In mid it is breaking anywhere.

0 Karma

Ultra Champion

And how are you collecting this data? With a HF or a UF and how/where is it then forwarded?

0 Karma

Path Finder

We are collecting from UF

0 Karma

Ultra Champion

And is that UF sending to a single indexer/HF or to a load balanced pool of destinations (e.g. indexer cluster, multiple intermediate forwarders...)?

0 Karma

Path Finder

Sending to indexer cluster

0 Karma