All Apps and Add-ons

Rails Log LINEMERGE not working

krussell101
Path Finder

I'm not successfully getting line merging to work in my environment. The problem is with virtually all of my rail logs.

In /opt/splunk/etc/system/local/props.conf on my indexer I have the following:

[source::.../home/myapp/log/production.log]
BREAK_ONLY_BEFORE = Started
SHOULD_LINEMERGE = True

I don't have this on the HWF that are creating the logs. That's right is it not?

I had the above two lines reversed and changed due to a comment in another post. But that didn't fix it.

Also of note, I'm running tail -f production.log in a terminal window and watching that while seeing the real-time events come in on splunk. Pretty sure this is NOT related to log buffering. The events are flowing out of "tail" quite nicely. Certainly no second long delays.

Here are some examples of what is and is not working:

did not get split

Started GET "/merchant/en" for xx.xxx.xxx.xxx at 2012-07-25 19:07:21 +0000

Processing by Partner::TransfersController#index as */*

Parameters: {"locale"=>"en"}

Rendered partner/transfers/_form.html.erb (23.9ms)

Rendered partner/transfers/_merchant_info.erb (1.3ms)

Rendered partner/transfers/_transactions_table.html.erb (0.5ms)

Rendered partner/transfers/_filter_form.html.erb (2.9ms)

Rendered partner/transfers/_total_filter.html.erb (0.7ms)

Rendered partner/transfers/_filter_table.html.erb (1.1ms)

Rendered partner/transfers/index.html.erb within layouts/application (33.3ms)

Rendered layouts/public/_header.html.erb (60.7ms)

Completed 200 OK in 121ms (Views: 95.7ms | ActiveRecord: 8.9ms)

split this entry

Started GET "/merchant/en" for xx.xxx.xxx.xxx at 2012-07-25 19:02:23 +0000

Processing by Partner::TransfersController#index as */*

Parameters: {"locale"=>"en"}

Rendered partner/transfers/_form.html.erb (25.4ms)

Rendered partner/transfers/_merchant_info.erb (0.8ms)

Rendered partner/transfers/_transactions_table.html.erb (0.5ms)

split was here

Rendered partner/transfers/_filter_form.html.erb (67.5ms)

Rendered partner/transfers/_total_filter.html.erb (0.8ms)

Rendered partner/transfers/_filter_table.html.erb (1.4ms)

Rendered partner/transfers/index.html.erb within layouts/application (99.5ms)

Rendered layouts/public/_header.html.erb (1.1ms)

Completed 200 OK in 127ms (Views: 102.0ms | ActiveRecord: 8.8ms)

Split this entry

Started POST "/merchant/en/merchant_employees/sign_in" for xx.xxx.xxx.xxx at 2012-07-25 19:12:23 +0000

Processing by SessionsController#create as */*

Parameters: {"utf8"=>"✓", "authenticity_token"=>"+/lDricYw5cogAGwWZKMfbRFi1UNIip3bBVLXBUB1OM=", "merchant_employee"=>{"login_or_email"=>"healthcheck", "password"=>"[FILTERED]"}, "locale"=>"en"}

split was here

Redirected to https://www.geopay.kg/merchant/en

Completed 302 Found in 95ms (ActiveRecord: 0.0ms)

did not get split

Started POST "/merchant/en/merchant_employees/sign_in" for XX.XXX.XXX.XXX at 2012-07-25 19:17:20 +0000

Processing by SessionsController#create as */*

Parameters: {"utf8"=>"✓", "authenticity_token"=>"8HkKQSzw5TP/Rur0U+mqOm9xkbthOaVAAsAoBZ5ziFE=", "merchant_employee"=>{"login_or_email"=>"healthcheck", "password"=>"[FILTERED]"}, "locale"=>"en"}

Redirected to https://www.geopay.kg/merchant/en

Completed 302 Found in 159ms (ActiveRecord: 0.0ms)

Help Appreciated!

dwaddle
SplunkTrust
SplunkTrust

I see your comment "I don't have this on the HWF that are creating the logs. That's right is it not?"

If you are using the Heavy forwarder, then you need the linebreaking / linemerging configurations set up there. The Heavy forwarder does do event parsing, which is what makes it "heavy".

For a light or universal forwarder, having those configurations only on the indexer is correct.

What I might do to change your configuration is to update it as follows:

BREAK_ONLY_BEFORE = ^Started

Just to make sure that "Started" that appears in the middle of a line is ignored.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...