topic Re: Events chunked into 256 lines in Getting Data In

Events chunked into 256 lines

Nicholas_Key — Tue, 14 Sep 2010 04:16:12 GMT

Hi all,

I have the following output from a Perl script that runs every 5 mins:

09-13-2010 16:21:20 -  Inventory Report 
DATACENTER, CLUSTER, VMHOST, VM
PPD, DNSAS-Cluster1, dnsas-esx1, dnsa-secweb1
PPD, DNSAS-Cluster1, dnsas-esx1, dnsasval1-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, ddist3-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, dmplupe1-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, dnsasext1-dev9
...
...
...
(520 lines)

And this is how my props.conf looks like:

[source::vm_inventory]
SHOULD_LINEMERGE=True
BREAK_ONLY_BEFORE_DATE=True
TIME_FORMAT=%m-%d-%Y %H:%M:%S

However the events got truncated into 256 line chunks. Am I missing anything here? Or should I do it this way?

[source::vm_inventory]
TRUNCATE = 0
LINE_BREAKER = (?!)

Re: Events chunked into 256 lines

dwaddle — Tue, 14 Sep 2010 04:24:52 GMT

In props.conf, change MAX_EVENTS

MAX_EVENTS = <integer>
* Specifies the maximum number of input lines to add to any event. 
* Splunk breaks after the specified number of lines are read.
* Defaults to 256.

Re: Events chunked into 256 lines

gkanapathy — Tue, 14 Sep 2010 06:16:50 GMT

You can either set MAX_EVENTS high, which will take CR-LF delimited lines and merge them up to 256 lines, or you can use the

TRUNCATE = 0
LINE_BREAKER = (?!)
SHOULD_LINEMERGE = false

method, except that the LINE_BREAKER = (?!) will only split an event at the end of input, i.e., it's only going to work when you get a new file, or if it's a scripted input that gets invoked once for each event. SHOULD_LINEMERGE = false potentially has much better performance, but probably if you use it, you should use:

LINE_BREAKER = ([\r\n]+)\d{2}-\d{2}-\d{4}\s+\d{1,2}:\d{2}:\d{2}

FURTHERMORE, I notice that you don't have a time zone specified in your timestamp, I strongly recommend that you include the time zone in your timestamp (unless you are committed to always logging those in UTC, in which case it wouldn't hurt to include Z at the end anyway).