Getting Data In

Heavy Forwarder line breaking, reading logs too fast

meenal901
Communicator

Hi,

We use Heavy Forwarders in our environment. Recently, I noticed that the events are not breaking up properly. We also use event filtering, so the event with keyword "low information" should get Ignored.

Upon digging into splunkd.log, found the issue:The events are written slower in logs and splunk's reading speed is much more.

example event:
low information 2015/05/08 0:52:49.039 Generic HybridTradeServer "TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = ****"
Indexed events (2):
"TradingProductHome:<anonymous>>>> Attempt to enable strategy with no legs ignored, strategy key = *
***"

Is there a way to tell splunk to slowly read the logs? Can we "Pause" Splunk for 1 sec after each read so that the complete event gets written. I am seeing this problem with multiple events, esp. stack traces where 1 log line gets broken into 2 or 3 events.

I saw below options on community, will these help:
- crcSalt= <SOURCE>
- time_before_close = 15

Since we use heavy forwarer, maxkbps is 0.

Thanks,
Meenal

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Are your events actually multiline events? Or single line events with long messages in it? Your example seems to have a message content enclosed with quotes " ... "

My regex for break_only_before is wrong, should be:

BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\/\d{2}\/\d{2}
0 Karma

meenal901
Communicator

My current props setting are as below:

BREAK_ONLY_BEFORE = \d{4}/\d{2}/\d{2}
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
pulldown_type = 1
TRUNCATE = 0
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What you want to do is edit your props.conf for the sourcetype you are using and configure the event breaking. It doesnt matter how "fast" or slow" Splunk will read the the file if the event breaks are set correctly.

It also depends if these are SINGLE or MULTILINE events... Im assuming multiline..

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Propsconf

Refer to the SHOULD_LINEMERGE and BREAK_* options.

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \w+ \w+ \d{4}\\\d{2}\\\d{2}
0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>