topic Re: extract multi lines fields in Splunk Search

extract multi lines fields

lgmnemesis — Wed, 11 Dec 2013 17:49:02 GMT

We are logging the following application network statistics. I want to be able to index the data into splunk so we can generate reports on it.

The First line consists of the following fields:
timestamp, site name, remote server name , local server name

Other lines of the same record consists of the following fields:
statistic name : message type : origin Node : statistic Value

This is the actual log:

1386704158913 SITE-A,remoteServer1,localhost
receivedMessages:AAA:NODE1:10
receivedMessages:BBB:NODE1:10
sentMessages:CCC:NODE2:10
discMessages:AAA:NODE1:1
discMessages:BBB:NODE2:1

1386704158913 SITE-A,remoteServer2,localhost2
receivedMessages:FFF:NODE1:10
receivedMessages:GGG:NODE1:10
sentMessages:HHH:NODE2:10
discMessages:FFF:NODE1:1
discMessages:III:NODE2:1

Is there a way to extract all the fields above from that log format?

Thanks a lot.

Re: extract multi lines fields

kristian_kolb — Wed, 11 Dec 2013 17:59:46 GMT

Any problems with breaking the log into indvidual events? Or is it only regarding the field extraction?

Re: extract multi lines fields

lgmnemesis — Wed, 11 Dec 2013 18:35:16 GMT

If i break the log/lines into individual events, wont i loos the correlation between the first line (which consists of the event time stamp and other shared fields) and the other sub lines?

Re: extract multi lines fields

kristian_kolb — Wed, 11 Dec 2013 21:07:40 GMT

No, you should not break them into single-line events, for the exact reasons that you mention. My question was if you had succeeded in creating the (multi-line) events correctly in splunk.

Re: extract multi lines fields

asimagu — Thu, 12 Dec 2013 02:14:44 GMT

You can easily do that by adjusting the line breaking in props.conf

Have a play with regular expressions and the options under "Attributes that are available only when SHOULD_LINEMERGE is set to true" in

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Indexmulti-lineevents

Re: extract multi lines fields

linu1988 — Thu, 12 Dec 2013 06:30:43 GMT

Use Regext to extract the time, then MVEXPAND then you will be able to have the correlation. Then make the extractions.