Getting Data In

How can I get more than 10,000 lines into a single event?

payal23
Path Finder

I want more than 10,000 lines to merge and show in a single event.

[tally_nightly_prd]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
TRUNCATE=0
disabled=false
BREAK_ONLY_BEFORE=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
MAX_EVENTS=90000
TIME_FORMAT=%+
TIME_PREFIX=\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
0 Karma

jinseong
Path Finder

hello

open the limits.conf and configration maxchars=10240

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Just want to make sure you're aware that having that many line in a single event will not give you a pleasant Splunk UI experience when viewing the same. Assuming you still want to do it, give this a try

 [tally_nightly_prd]
 SHOULD_LINEMERGE=false
 LINE_BREAKER = ([\r\n]+)(?=(\*){12}\snightlyProcess\sStarted)
 TRUNCATE=0
 MAX_EVENTS=90000
 TIME_FORMAT=%+
 TIME_PREFIX=^\*\*\*\*\*\*\*\*\*\*\*\*\snightlyProcess\sStarted
0 Karma

payal23
Path Finder

Thanks...Yes, logs are having big xml payload and hence merging in an event will make sense.

I tried the above but now the lines are breaking in single line.

😞

0 Karma

manishankark04
New Member

you can increase the truncate parameter to 40k or 50k.

0 Karma

FrankVl
Ultra Champion

And what exactly is your question? Is your current config not working as expected? If so: what is the expected outcome and what outcome are you now getting?

Also a bit more context around the data you're ingesting and what you are trying to achieve would probably help 🙂

0 Karma

payal23
Path Finder

In between my file start and end points there are number of lines in between which is more than 10,000 and i want all the lines to come under one event. But the breaking is not happening in that way. In mid it is breaking anywhere.

0 Karma

FrankVl
Ultra Champion

And how are you collecting this data? With a HF or a UF and how/where is it then forwarded?

0 Karma

payal23
Path Finder

We are collecting from UF

0 Karma

FrankVl
Ultra Champion

And is that UF sending to a single indexer/HF or to a load balanced pool of destinations (e.g. indexer cluster, multiple intermediate forwarders...)?

0 Karma

payal23
Path Finder

Sending to indexer cluster

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...