Mapping Fields to Meaningful Names

[cgl]

I have a vendor log file that has numeric codes for the field names (i.e. E-1, E-710, etc). The vendor also provides a lookup table that translates the codes into meaningful names (i.e. Asset Name, CPU Time, etc).

I am currently adding the translation via FIELDALIAS in the props.conf, but I'd rather provide the translation via a lookup table for convenience and maintainability. Also I'd prefer to replace the numeric code with the meaningful name rather than create duplicates.

Any ideas on how I should proceed?

Thanks in advance,

Chris

[hexx]

This should be possible if your key/value pairs are present in the raw data of your events.

Let's take a hypothetical example, assuming that this is the raw data of one of your events :

20 Jan 2011 19:34:22 E-1=100 Purgamentum init, exit purgamentum

Let's say that field name "E-1" needs to be translated to "Error message" and field value "100" needs to be translated to "Banjo infestation".

The first thing we want to do is to extract the "E-1" and "100" as a field each, not as a key/value pair. Let's do this with a rex :

| rex "\d2:\d2:\d2 (?<field_name_code>[^\=]*)\=(?<field_value_code>[^\s]*)\s

So now we have two new fields : field_name_code = E-1 and field_value_code = 100.

This is where your lookup comes in (let's say it's defined in transforms.conf as "errorlookup"] to output two new fields : "field_name" which will be looked up based on "field_name_code" and "field_value" which will be looked up based on "field_value_code".

| lookup errorlookup field_name_code field_value_code OUTPUT field_name field_value

So we'll now have a new field named "field_name" with a value of "Error message" and another one called "field_value" with a value of "Banjo infestation".

Now you want to create a new field that concatenates "field_name" and "field_value" with a known delimiter (let's say "=") :

| eval error_kv=field_name."=".field_value

We'll now have a new field called "error_kv" with a value of "Error message=Banjo infestation".

The final step is to use a delimiter-based search-time field extraction scoped to the "error_kv" field in order to create a new field named "field_name" with a value of "field_value". The options of the kv command do not allow for this to be done inline, so we'll need to define our field extraction in $SPLUNK_HOME/etc/system/local/transforms.conf (or anywhere else that is adequate to your configuration) :

[extract_error_kv]
SOURCE_KEY = error_kv
DELIMS = "="

And now, we can call this custom extraction inline using kv :

| kv extract_error_kv

In our example, this should result in the creation of the field "Error message" with a value of "Banjo infestation"!

Part of this is theory, so I'm quite curious to know if it works in your environment. Please let me know!

[piebob]

+1 for banjo infestation

[cgl]

It's okay, I appreciate you taking the time to respond.

[Ayn]

Ahh, I see - you're talking about the names of the fields themselves, not their values? I don't have any better solution than the one you're already going with, sorry...

[cgl]

The lookup is a CSV, but the lookup only maps the values, not the fields. For example,

E-1=100, could use the lookup table to map to E-1=Error Message

However, I haven't been able to map the E-1 to a better presentation value, expect via the FIELDALIAS option.

[Ayn]

Seeing as you're provided with a lookup table from the vendor, you should absolutely be using that, especially if it's in CSV format - just put it in the lookups directory, configure the lookup in props.conf / transforms.conf, and you're all set!