topic Extracting multiple fields in Splunk Search

Extracting multiple fields

johntobin — Wed, 02 Apr 2014 18:29:35 GMT

Hi all,

My logs have strings like the following:

Mon Mar 31 2014 10:41:48 [info] wsgw(parlayx-all-interfaces): tid(2658703090)[response][96.1.1.1]: attachment-size:{0}
Mon Mar 31 2014 10:41:48 [info] wsgw(parlayx-all-interfaces): tid(2658703090)[response][96.1.1.1]: applicationid:{}
Mon Mar 31 2014 10:41:48 [info] wsgw(parlayx-all-interfaces): tid(2658703090)[response][96.1.1.1]: operation:{getWirelessNetworkSubscriberProfileResponse} BillingType:{postpaid} EquipmentId:{} nil:{true} SimId:{} nil:{true} Imsi:{302220007118231} IpAddress:{} GatewayId:{MMS-22990121_wap2.company.com} SubscriberId:{22167135} TechnologyType:{HSPA} uri:{14185648339} PreferredLanguage:{fr-ca} ServiceProviderId:{COMPANY} UserStatus:{active}
Mon Mar 31 2014 10:41:48 [info] wsgw(parlayx-all-interfaces): tid(3159363380)[response][96.1.1.1]: attachment-size:{0}

I would like to extract all the fields such as "operation:{getWirelessNetworkSubscriberProfileResponse}" and "TechnologyType:{HSPA}", each as a field, with the value inside the {}'s. This command works well:

source=sdf | extract extract kvdelim=":" pairdelim=" "

...except that the curly brackets are left behind in the field value. Is there a way to tell extract that the value in the key/value pair has quotes (a delimiter) around it? Should I execute an additional transform? I'd love to have all these fields automatically extracted whenever I search this source, but I can't see how to do this easily. I'm sure it's possible, but I have no idea how to do that.

Hope you can help, thanks!

Re: Extracting multiple fields

martin_mueller — Wed, 02 Apr 2014 18:52:53 GMT

This should be doable with a transforms.conf based field extraction, something like this:

[yourstanza]
REGEX = \s(?<_KEY_1>\w+):\{(?<_VAL_1>[^}]+)\}

Re: Extracting multiple fields

johntobin — Thu, 03 Apr 2014 14:19:12 GMT

Thank you so much for your quick reply! This worked perfectly. I also created the following in 'props.conf' to leverage the stanza, as per the documentation (not sure if I had to do this, but it's working). I thought I'd mention this for anyone else who is trying to get this to work:

[mysourcetype]
REPORT-myattributes = yourstanza

Relevant documentation for the version I'm using: http://docs.splunk.com/Documentation/Splunk/6.0.2/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles

Thanks again!

Re: Extracting multiple fields

martin_mueller — Thu, 03 Apr 2014 15:10:19 GMT

Yeah, the REPORT-foo = bar setting in props.conf is necessary - else Splunk wouldn't know it needs to apply the transforms.conf stanza.