Getting Data In

Events chunked into 256 lines

Splunk Employee
Splunk Employee

Hi all,

I have the following output from a Perl script that runs every 5 mins:

09-13-2010 16:21:20 -  Inventory Report 
DATACENTER, CLUSTER, VMHOST, VM
PPD, DNSAS-Cluster1, dnsas-esx1, dnsa-secweb1
PPD, DNSAS-Cluster1, dnsas-esx1, dnsasval1-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, ddist3-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, dmplupe1-dev9
PPD, DNSAS-Cluster1, dnsas-esx1, dnsasext1-dev9
...
...
...
(520 lines)

And this is how my props.conf looks like:

[source::vm_inventory]
SHOULD_LINEMERGE=True
BREAK_ONLY_BEFORE_DATE=True
TIME_FORMAT=%m-%d-%Y %H:%M:%S

However the events got truncated into 256 line chunks. Am I missing anything here? Or should I do it this way?

[source::vm_inventory]
TRUNCATE = 0
LINE_BREAKER = (?!)
Tags (1)

Splunk Employee
Splunk Employee

You can either set MAX_EVENTS high, which will take CR-LF delimited lines and merge them up to 256 lines, or you can use the

TRUNCATE = 0
LINE_BREAKER = (?!)
SHOULD_LINEMERGE = false

method, except that the LINE_BREAKER = (?!) will only split an event at the end of input, i.e., it's only going to work when you get a new file, or if it's a scripted input that gets invoked once for each event. SHOULD_LINEMERGE = false potentially has much better performance, but probably if you use it, you should use:

LINE_BREAKER = ([\r\n]+)\d{2}-\d{2}-\d{4}\s+\d{1,2}:\d{2}:\d{2}

FURTHERMORE, I notice that you don't have a time zone specified in your timestamp, I strongly recommend that you include the time zone in your timestamp (unless you are committed to always logging those in UTC, in which case it wouldn't hurt to include Z at the end anyway).

SplunkTrust
SplunkTrust

In props.conf, change MAX_EVENTS

MAX_EVENTS = <integer>
* Specifies the maximum number of input lines to add to any event. 
* Splunk breaks after the specified number of lines are read.
* Defaults to 256.
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!