I have a vendor log file that has numeric codes for the field names (i.e. E-1, E-710, etc). The vendor also provides a lookup table that translates the codes into meaningful names (i.e. Asset Name, CPU Time, etc).
I am currently adding the translation via FIELDALIAS in the props.conf, but I'd rather provide the translation via a lookup table for convenience and maintainability. Also I'd prefer to replace the numeric code with the meaningful name rather than create duplicates.
Any ideas on how I should proceed?
Thanks in advance,
Chris
This should be possible if your key/value pairs are present in the raw data of your events.
Let's take a hypothetical example, assuming that this is the raw data of one of your events :
20 Jan 2011 19:34:22 E-1=100 Purgamentum init, exit purgamentum
Let's say that field name "E-1" needs to be translated to "Error message" and field value "100" needs to be translated to "Banjo infestation".
The first thing we want to do is to extract the "E-1" and "100" as a field each, not as a key/value pair. Let's do this with a rex :
| rex "\d2:\d2:\d2 (?<field_name_code>[^\=]*)\=(?<field_value_code>[^\s]*)\s
So now we have two new fields : field_name_code = E-1
and field_value_code = 100
.
This is where your lookup comes in (let's say it's defined in transforms.conf as "errorlookup"] to output two new fields : "field_name" which will be looked up based on "field_name_code" and "field_value" which will be looked up based on "field_value_code".
| lookup errorlookup field_name_code field_value_code OUTPUT field_name field_value
So we'll now have a new field named "field_name" with a value of "Error message" and another one called "field_value" with a value of "Banjo infestation".
Now you want to create a new field that concatenates "field_name" and "field_value" with a known delimiter (let's say "=") :
| eval error_kv=field_name."=".field_value
We'll now have a new field called "error_kv" with a value of "Error message=Banjo infestation".
The final step is to use a delimiter-based search-time field extraction scoped to the "error_kv" field in order to create a new field named "field_name" with a value of "field_value". The options of the kv command do not allow for this to be done inline, so we'll need to define our field extraction in $SPLUNK_HOME/etc/system/local/transforms.conf (or anywhere else that is adequate to your configuration) :
[extract_error_kv]
SOURCE_KEY = error_kv
DELIMS = "="
And now, we can call this custom extraction inline using kv :
| kv extract_error_kv
In our example, this should result in the creation of the field "Error message" with a value of "Banjo infestation"!
Part of this is theory, so I'm quite curious to know if it works in your environment. Please let me know!
+1 for banjo infestation
It's okay, I appreciate you taking the time to respond.
Ahh, I see - you're talking about the names of the fields themselves, not their values? I don't have any better solution than the one you're already going with, sorry...
The lookup is a CSV, but the lookup only maps the values, not the fields. For example,
E-1=100, could use the lookup table to map to E-1=Error Message
However, I haven't been able to map the E-1 to a better presentation value, expect via the FIELDALIAS option.
Seeing as you're provided with a lookup table from the vendor, you should absolutely be using that, especially if it's in CSV format - just put it in the lookups directory, configure the lookup in props.conf / transforms.conf, and you're all set!