Getting Data In

Heavy Forwarder line breaking, reading logs too fast

Communicator

Hi,

We use Heavy Forwarders in our environment. Recently, I noticed that the events are not breaking up properly. We also use event filtering, so the event with keyword "low information" should get Ignored.

Upon digging into splunkd.log, found the issue:The events are written slower in logs and splunk's reading speed is much more.

example event:
low information 2015/05/08 0:52:49.039 Generic HybridTradeServer "TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = ****"
Indexed events (2):
"TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = *
***"

Is there a way to tell splunk to slowly read the logs? Can we "Pause" Splunk for 1 sec after each read so that the complete event gets written. I am seeing this problem with multiple events, esp. stack traces where 1 log line gets broken into 2 or 3 events.

I saw below options on community, will these help:
- crcSalt= <SOURCE>
- timebeforeclose = 15

Since we use heavy forwarer, maxkbps is 0.

Thanks,
Meenal

0 Karma

Splunk Employee
Splunk Employee

Are your events actually multiline events? Or single line events with long messages in it? Your example seems to have a message content enclosed with quotes " ... "

My regex for breakonlybefore is wrong, should be:

BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\/\d{2}\/\d{2}
0 Karma

Communicator

My current props setting are as below:

BREAKONLYBEFORE = \d{4}/\d{2}/\d{2}
MAXTIMESTAMPLOOKAHEAD = 150
NOBINARYCHECK = 1
pulldowntype = 1
TRUNCATE = 0
SHOULD
LINEMERGE = true
MAX_EVENTS = 10000

0 Karma

Splunk Employee
Splunk Employee

What you want to do is edit your props.conf for the sourcetype you are using and configure the event breaking. It doesnt matter how "fast" or slow" Splunk will read the the file if the event breaks are set correctly.

It also depends if these are SINGLE or MULTILINE events... Im assuming multiline..

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Propsconf

Refer to the SHOULDLINEMERGE and BREAK* options.

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\\\d{2}\\\d{2}
0 Karma