topic Re: adding static field value using props transforms based on source in Getting Data In

adding static field value using props transforms based on source

sonicZ — Thu, 26 Jan 2012 19:03:31 GMT

Hey,

I am looking to add a static field "instance=testdrive" to all results from a source input with td-idp-manager in the path.
I was able to extract a value from the source like so, but would prefer to not have to extract the "td-idp-manager" path in source and just use the word "testdrive" instead. Not sure the syntax to use...

What i am currently using:
props.conf

[source::.../td-idp-manager/*]
REPORT-instance = testdrive_transform

transforms.conf

[testdrive_transform]
SOURCE_KEY = source
REGEX = (?i)[\/A-Za-z]+\/(?<instance>td-idp-manager+)/.*

fields.conf

[instance]
INDEXED_VALUE = false

Re: adding static field value using props transforms based on source

lguinn2 — Sun, 29 Jan 2012 11:33:19 GMT

There are several possibilities that I can think of - choose your favorite:

1 - Add a field to each event

In props.conf

[source::.../td-idp-manager/*]
TRANSFORM = testdrive_transform
EXTRACT-tim1 = \sinstance:(?<instance>testdrive)$

In transforms.conf

[testdrive_transform]
REGEX=(.*)
FORMAT=$1 instance:testdrive
DEST_KEY=raw

This will add the string " instance:testdrive" to each event, which is then extracted into the field. Now you can use instance=testdrive in your searches.

2 - Use Tags

Delete the transform that you have now. Set up a tag named testdrive that corresponds to

source=*td-idp-manager*

Then you can search by entering tag=testdrive

3 - Use Lookups

Keep your current transform, but change the name of the field extracted - make it instance_raw instead. Then set up a lookup table with two columns:

instance_raw,instance
td-idp-manager,testdrive

Set this as an automatic lookup, and then you will have a field named instance with the value testdrive. So you will be able to search instance=testdrive. If you have multiple values that you want to map, just add additional entries to the lookup table.

Summary

Option 1 adds a small amount of overhead at indexing time, and a few bytes to each event.
Option 2 is all done at search time; this is the most simple option.
Option 3 is the only one that uses your existing transform, but it then uses a lookup to map the values.

Re: adding static field value using props transforms based on source

sonicZ — Tue, 31 Jan 2012 07:04:19 GMT

Thanks lguinn, i like option #3 the best, will go with lookups. Do i need to do any changes in props.conf such as
EXTRACT-instance = testdrive_transform or call the lookup in props.conf?

Re: adding static field value using props transforms based on source

lguinn2 — Tue, 31 Jan 2012 09:26:31 GMT

For option 3, keep your transform. Just change <instance> to <instance_raw> in your REGEX.

You can create the lookup very easily from the Splunk Manager UI:

Build a CSV file on your desktop that contains the data you need. The first row MUST be a header; the column names will be the field names in your lookup.
Go to Manager » Lookups in the Splunk UI
Add a new Lookup Table File. This is where you will upload the CSV file from your desktop into Splunk.
Add a new Lookup Definition. This is where you tell Splunk that you want to do a file-based lookup, using your Lookup Table file from the previous step.
Add a new Automatic Lookup. Here you tell Splunk how to use your Lookup Definition automatically, and tell it which fields to retrieve, etc.
Be sure to set permissions on each of the items: the table, the definition, and the automatic lookup - if you want them to be used by others.

Once the lookup is created, you can see that it adds additional entries to props.conf and transforms.conf. If you want to edit them directly, take a look at Setup a fields lookup based on a static file