topic Re: How do I create key/value pairs from a _raw field with only values? in Splunk Search

How do I create key/value pairs from a _raw field with only values?

joshua_hart — Thu, 18 Jul 2013 12:18:04 GMT

I have a Symantec Messaging Gateway syslog input that provides syslog with no keys, only values. For example:

2013-07-11T13:13:16-04:00 appliance-name ecelerity: 1373562795|d6038c16-b7fe96d000000710-2d-51dee7aae3dd|SENDER|some-email-address@domain.tld

Right now this entire event is contained within the "_raw" field. The log data is everything after "ecelerity:" and each value is delimited by a pipe character. What I'd like to do is create fields for those values and then index the event so I can search on those fields. The example above would have five fields:

Epoch Time
Unique ID
Action
Sender Address

It's important to note that the above example is but one among many. Some of the other events have more values and the keys for those values will differ based on the type of event (though everything up to and including the 'Action' field would be consistent across events).

What I need is the means to parse these events and then create rules for each event to add keys to the values. How can I do this? I'm thinking something in the props/transforms, but I'm not exactly sure how.

Re: How do I create key/value pairs from a _raw field with only values?

sdaniels — Thu, 18 Jul 2013 14:49:14 GMT

You could start with the Splunk Interactive Field Extractor (IFX) to parse out your fields for you. It will generate the appropriate regex for you. Sometimes it may need to be tweaked though.

http://www.splunk.com/view/SP-CAAADUY
http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/ExtractfieldsinteractivelywithIFX

And yes you are correct that you'll use props.conf and transforms.conf to manually extract out fields. The IFX will write out data to those config files so you'll see the examples it creates. You should see those additions under $SPLUNK_HOME/etc/users.

http://docs.splunk.com/Documentation/Splunk/5.0.3/Knowledge/Addfieldsatsearchtime

There is also a way to extract fields on the fly in a search if it's something less common and you don't already have a field:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Search/Extractfieldswithsearchcommands

Re: How do I create key/value pairs from a _raw field with only values?

the_wolverine — Thu, 18 Jul 2013 20:18:04 GMT

If your data originates from a file that contains a header, I would use automatic header-based fields: http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Extractfieldsfromfileheadersatindextime

Re: How do I create key/value pairs from a _raw field with only values?

joshua_hart — Sat, 10 Aug 2013 00:01:21 GMT

Using the IFX seemed to work for now. I wasn't able to extract all the fields I was looking for, but I was able to get at what I needed for our purposes.

Ideally, if I had the option to format the data before being sent to syslog, I'd be happy. In fact, if Symantec didn't send Brightmail mail audit logs to syslog as separate events (each aspect of a single record is sent as a separate syslog event) I'd have a much easier time extracting fields.

Thanks for the tips, BTW.