We are running a rails application and are using splunk to parse our rails logs. We have a search-head and 2 indexers. On the indexers, I have added the following to /opt/splunk/etc/apps/search/local/props.conf to ensure that the logging for each rails request is are parsed as a single event:
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
Each application server runs a forwarder. The rails log for each project is added using the following type of command, so that the source types match the pattern in the above stanza.
/opt/splunkforwarder/bin/splunk add monitor -source '/home/builder/abitlucky/web/luckyonrails/log/production.log' -sourcetype project1-rails
/opt/splunkforwarder/bin/splunk add monitor -source '/home/builder/abitlucky/web/luckyonrails/log/production.log' -sourcetype project2-rails
The issue is that I am seeing every single rails request duplicated in the search application. If I click on 'Show Source' for both of the events for a duplicated request, I get strange results. For some cases, it highlights the original request in the source for one of the events and a completely different request in the source for the duplicate (i.e., the request it shows as the source does not match the request I clicked 'Show Source' for). In other cases, it highlights the original request in the source for one of the events, and then highlights a second listing of that request in the source for the second event. In these cases, I can see the request repeated in the source it is showing me. However, if I go back to the original log, the request only shows up once, so the repeated request in the source Splunk is showing me is a phantom/fake request that was not in the original log for these cases.
I'm not sure if that makes sense without actually seeing what I am referring to, but I explained it as best I could. Has anyone seen this behavior? What could be causing it?
... View more