topic Re: Multi line log issue in Getting Data In

Multi line log issue

castle1126 — Tue, 26 Oct 2010 01:28:34 GMT

I have a Windows system with 4.1.5 forwarding to my Splunk indexer, that puts out logs in this format:

begin error

lines of interesting log entries

end error

I've been noodling around with different options within the PROPS.CONF on the Forwarder system. So far no luck. My goal is to be have the forwarder sear the data correctly then transfer to the Indexing server.

Any tips or ideas I'm missing?

Re: Multi line log issue

ftk — Tue, 26 Oct 2010 01:58:05 GMT

So do you want multiline events or not? Can't quite tell from your question. Can you please clarify?

Re: Multi line log issue

southeringtonp — Tue, 26 Oct 2010 02:27:35 GMT

Are these Windows Event Log events or text-based? What do you want the indexed events to look like?

If all events follow the format you describe, then it should be enough to do:

#props.conf
[yoursourcetype]
LINE_BREAKER=([\r\n]+)(### begin error)

Not sure if the hash marks would need to be escaped.

Re: Multi line log issue

castle1126 — Tue, 26 Oct 2010 03:01:09 GMT

The logs I'm going after are not Event logs, they're output from a custom program. I tried as you noted and no luck. The first event shows the header line (===== begin error =====) and the next line from the file (ASP error on page: http://server/page.asp).
The next event begins with the 3rd line of text (At 10/25/2010 3:55:09pm) and shows the remaining lines of text up to the last line (===== end error =====).

By the way, I tested the = sign in REGEX Buddy and it recognized the = sign as a character.

Any other ideas?

Re: Multi line log issue

castle1126 — Tue, 26 Oct 2010 03:06:14 GMT

A question - in my setup the forwarder system has the props.conf with the LINE_BREAKER entry in it. Should this props.conf be moved to the indexing server or left on the forwarding system?

Re: Multi line log issue

southeringtonp — Tue, 26 Oct 2010 04:22:23 GMT

Equals signs would not need to be escaped. In your original question, you had hashes, which might have been interpreted as the beginning of a comment in the config file. If you are using a lightweight forwarder, then do this at the indexer. For heavy-weight forwarders, do it at the forwarder.

Re: Multi line log issue

castle1126 — Tue, 26 Oct 2010 08:01:32 GMT

Sorry for the posting glitch, my bad with the # versus = sign. In this case the forwarder being used is the light forwarder. Thanks for the tip, I'll make the entry in the indexer's props.conf and see how that works.

Re: Multi line log issue

castle1126 — Tue, 26 Oct 2010 22:14:43 GMT

Hey SoutheringtonP,

I changed the client from running LightForwarder to Forwarder. Your change to the props.conf worked perfect! Thanks for the insight!!