Heavy Forwarder line breaking, reading logs too fast

[meenal901]

Hi,

We use Heavy Forwarders in our environment. Recently, I noticed that the events are not breaking up properly. We also use event filtering, so the event with keyword "low information" should get Ignored.

Upon digging into splunkd.log, found the issue:The events are written slower in logs and splunk's reading speed is much more.

example event:
low information 2015/05/08 0:52:49.039 Generic HybridTradeServer "TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = ****"
Indexed events (2):
"TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = ****"

Is there a way to tell splunk to slowly read the logs? Can we "Pause" Splunk for 1 sec after each read so that the complete event gets written. I am seeing this problem with multiple events, esp. stack traces where 1 log line gets broken into 2 or 3 events.

I saw below options on community, will these help:
- crcSalt= <SOURCE>
- time_before_close = 15

Since we use heavy forwarer, maxkbps is 0.

Thanks,
Meenal

[esix_splunk]

Are your events actually multiline events? Or single line events with long messages in it? Your example seems to have a message content enclosed with quotes " ... "

My regex for break_only_before is wrong, should be:

BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\/\d{2}\/\d{2}

[meenal901]

My current props setting are as below:

BREAK_ONLY_BEFORE = \d{4}/\d{2}/\d{2}
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
pulldown_type = 1
TRUNCATE = 0
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000

[esix_splunk]

What you want to do is edit your props.conf for the sourcetype you are using and configure the event breaking. It doesnt matter how "fast" or slow" Splunk will read the the file if the event breaks are set correctly.

It also depends if these are SINGLE or MULTILINE events... Im assuming multiline..

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Propsconf

Refer to the SHOULD_LINEMERGE and BREAK_* options.

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\\\d{2}\\\d{2}