Multiline event query - Sharepoint logs

[neilamoran]

I am trying to index logs from my Sharepoint farm into Splunk. I have managed to get Splunk to correctly deal with multiline events

The difficulty I have is in correctly dealing with the fields within the events that are created when the Sharepoint Message field has a larger number of characters.

As a simplified example, I might see lines like this:

Timestamp,1,2,3,The 
Timestamp*,1,2,3,sky
Timestamp*,1,2,3,is
Timestamp*,1,2,3,blue

Sharepoint uses the * after the timestamp to indicate a multiline event, duplicates all fields other than the Message field across all the lines in the event, and then splits the Message field across all the lines. In reality it further indicates the multiline event by using ellipses like so

Timestamp,1,2,3,The...
Timestamp*,1,2,3,...sky...
Timestamp*,1,2,3,...is...
Timestamp*,1,2,3,...blue

I'd like to get Splunk to index the example above as a single event like:

Timestamp,1,2,3,The sky is blue

...so (now I'm doing it! 😄 ) that I could search for things like Message="*sky*blue*" and return the single consolidated event, instead of nothing.

Any suggestions? I'm not sure what would be the best approach here.

UPDATE: Just realised I hadn't explained what currently seems to happen with the fields set in my props.conf - basically the fields get read correctly out of the first line (and for all single line events) but then Splunk hits the second line and lumps it and all the rest of the event into an undefined field so it can't be searched. So what I'd get from the example above is an event that contains:

_time=Timestamp
field1=1
field2=2
field3=3
Message=The...

and the raw event wound contain the entire text from all lines. So this is the behaviour I'm trying to avoid.

Cheers!

[gkanapathy]

I would consider using SEDCMD, something like:

SEDCMD-spmsg = s/(?s-m)/(?:[\r\n])+Timestamp\*,(?:[^,]*,){3}\s*\.\.\.([\V]*)\.\.\.(?=\v|$)/\1/g

or

SEDCMD-spmsg = s/(?m-s)/^Timestamp\*,(?:[^,]*,){3}\s*\.\.\.(.*)\.\.\.$/\1/g

might do it.

[neilamoran]

Wow - that looks complicated. Sorry to be a painfully ignorant newbie, but could you break down for me what that little lot is actaully doing? Looks like I'll need to modify it slightly to use in the real world (unlike my simplified example) and I'm not clear how it works, so don't know what to change, or how much.

Also, as far as I can understand, SEDCMD s/ should do string replacement - will this handle the string correctly as it is built up over multiple lines?

Please excuse my ignorance - just trying to understand.