Why is Splunk combining several log lines into one event, and how do I search within these multiline events to get accurate count results?

[joemadeus]

We're trying to count the number of times a particular call is made to a service. To do that, we're logging a log line for every call, one that contains a well-known string, to a gunicorn file (one file per instance), which is then being picked up by Splunk for analysis.

When I search for these log lines, I find that what's returned are events, not lines -- and events have many lines in them. In some cases, I see four or six of the calls we're interested in logged, but count (when I use timechart) is equal to one. Emitting sum(linecount) doesn't produce the correct result either, since it emits the total number of lines in the event, not just the lines we're interested in. I can't seem to get it to return the correct number (four, six).

What's the way through? It seems like this is something Splunk should be able to do. I'm told it's possible that the parser we're using is wrong, but I don't see where I change that. Any hints would be very, very welcome.

Thanks for your time.

[mreynov_splunk]

The problem is that Splunk does not understand how to break up the lines in your data. To fix that, you need to set the correct LINE_BREAKER: http://docs.splunk.com/Documentation/Splunk/6.2.8/Data/Indexmulti-lineevents

Do you have a sourcetype for your data or did you use an existing one?

[joemadeus]

@mreynov, thanks. I did see this and ([\r\n]+) (the default) should work. I think SHOULD_LINEMERGE=false is even more useful -- but the docs only tell us how to set this for the Enterprise version. Is there a way to do this for the Cloud offering?

And, we used the existing sourcetype -- I didn't change it and I don't think anyone else did.

[mreynov_splunk]

You should be able to play around with sourcetype settings after clicking "Add Data" and using a file.
To get the file, you should be able to export some of your search results in raw format.
Then save the sourcetype and you will be able to use that when defining your actual input.