topic Re: heavy forwarder lookup in Splunk Search

heavy forwarder lookup

tmarlette — Mon, 23 Jun 2014 15:30:16 GMT

I was wondering if it is possible to have a heavy forwarder perform a lookup on a field before it sends data to the indexer?

For instance, I have a series of KV pairs that are numeric in nature, and so are their values, so splunk doesn't recognize them as fields. below is an example of some of the data I am capturing:

1015=USD  9053=0 20064=329915 20200=TESTTR 20401=100 20403=100,101 20404=ef2508bb-5fc-0n5i-3 20409=3 20677=Purf 20687=ef2508bb-5fc-0n5i 23054=14:9:35 23065=119 23153=5646521 23249=1532 23610=12 23955=1

Take for instance "1015=USD". This is the field that determines the currency. I am looking for the heavy forwarder to perform a lookup on "1015" and then forward to the indexer as 'currency'.

Is this possible?

Re: heavy forwarder lookup

lguinn2 — Tue, 24 Jun 2014 00:47:49 GMT

Sorry but no. However, on the indexer (or search head), you could extract the field on the left of the equal sign with a field name like "fieldDefn" and extract the data on the right side of the equal sign with the name "fieldValue".

You could then use the fieldDefn field to do a lookup and come up with the string representation of the field name...

But what you would do after that depends on the purpose of your search or report.

Re: heavy forwarder lookup

s2_splunk — Tue, 24 Jun 2014 03:35:57 GMT

Is this a FIX log format, by any chance...?
If so, have you seen this: http://apps.splunk.com/app/431/

Re: heavy forwarder lookup

tmarlette — Tue, 24 Jun 2014 15:34:31 GMT

Negative, this is a proprietary applications format, and while FIX tags are also included, this is not explicit FIX.

Re: heavy forwarder lookup

tmarlette — Tue, 24 Jun 2014 15:40:56 GMT

yeah but when I try that it doesn't work.

here is my RegEx for the capture:
(?\d+)=[^\s]+

This works in regexr, but not in splunk for some reason. Splunk only captures 2 of those fields with this extraction.