We have been testing Splunk processing Apache access logs that we have defined using a custom log entry to output key=value pairs. For the last month everything has been working nicely, allowing us to search based on field names and values.
For example, source_type="apachekv" field1 > 30 etc
But what we have recently noticed is that for one event in the Splunk Web UI, we are now getting 2,3 or more or more apache access lines concatenated into one single Splunk event being displayed.When I look at the raw apache logfile, each entry is on a single line and looks ok.
We are using version 4.3.3, build 128297.
Has anyone seen this before. I did see an old Changelog entry (12/5/2011) about a fix for concatenated lines in Apache.
By default, splunk break event with timestamp that splunk can recognize. The setting is "BREAKONLYBEFORE_DATE = true". I do not know your apache log format, but there is possibility that splunk can not recognize timestamp of your apache log so that splunk can not break each events properly.