Getting Data In

Is there a way to tell Splunk how long to wait for the beginning of the next event?

Champion

Hi,

I have a multi-line feed that appears to be having issues when the "next event" is delayed. Each event starts with a timestamp, and we have the line_breaker configured to break on those lines. It appears that when the feed gets slow, and additional lines are added to the existing event, Splunk is turning them into new events. (Hope this makes sense). Is there a way to tell Splunk how long to wait for the beginning of the next event?

For example: "AP" lines below took their time coming in , they then appeared as separate events. Other multi-line events that use the same format, but that aren't slow, worked fine.

20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnPartyDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnCallDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
-AP[94739]->-16 @20:21:56.8953
-Ap[94739]-<-16 @20:21:56.8958
-AP[98780]->-4 @20:22:01.2976
-Ap[98780]-<-4 @20:22:01.3170
20:22:03.332 <<<=== 'EventAgentNotReady'(76) seq=aa73d6

Our props.conf:

ANNOTATEPUNCT = false
FIELD
HEADERREGEX = ^File:
KV
MODE = auto
LINEBREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}.\d{3}
MAX
TIMESTAMPLOOKAHEAD = 25
NO
BINARYCHECK = 1
SHOULD
LINEMERGE = false
TIMEFORMAT = %H:%M:%S.%3N
TIME
PREFIX = ^
TRUNCATE = 999999

0 Karma

Hi,

looks like your line break pattern somehow also matches your AP lines. For me it works with this LINE_BREAKER:

^\d{2}:\d{2}:\d{2}\.\d{3}

alt text

0 Karma

Champion

Thanks. It worked for me (as did the other one) as well, but in prod it's not working. I actually tailed the file, and in once scenario, it took 15 seconds for the next "AP" line to appear (despite the timestamp on that line). Splunk is making these separate events. I'm guessing that the time between lines on the multi-event is causing the issue.

0 Karma

Champion

Anyone? I've looked through the doc, but nothing stands out.

0 Karma

Legend

If this is a file monitor input, have a look at the "timebeforeclose" directive in inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/inputsconf

0 Karma

Champion

Is that for waiting for the next event, for for determining that the file is closed? My issue is the delay between lines on a multi-line event.

0 Karma

Champion

Anyone?

0 Karma

Champion

Any idea on this?

0 Karma

SplunkTrust
SplunkTrust

Per definition, it says "timebeforeclose" is the time in second that Splunk will wait before considering data saved in file is completed. So, As @Ayn suggested, set this value to 1800 (30 mins) OR 1200 (20 min) based on your requirement, then Splunk will not start reading the data unless the last modification time for a version is atleast that seconds old. There will be delay though with this approach in getting your data to Splunk.

0 Karma

Champion

I think that we are talking two different things. The issue isn't that the file has closed or rolled-over - it's all in the same file. It goes like this (examples above):

First line of multi-line vent comes in:
Next line of multi-line event comes in - some seconds after the above line
Additional line of multi-line event comes in - some seconds after previous line

The first line is it's own event, and the next two are combined as a single multi-line event. If I test this input with my props, it works fine, so I think it's timing related.

0 Karma

Champion

Bump......

0 Karma