topic Re: How do I index a value thats not in the message? in Splunk Search

How do I index a value thats not in the message?

Marinus — Wed, 17 Nov 2010 20:13:47 GMT

I'm currently collecting logs on a lightweight forwarder. I'm adding a special field to the messages which I'd like to extract index it for later use at search time to enrich events with a lookup.

I don't want to add the field into the message. Is there s simple way to get my field to the indexer?

My first plan was to concatenate it to the host meta key on the forwarder and then extract and index it on the indexer. The host key can then be cleaned up.

[forwarder]

host=xyz,special_key

[indexer]

host=xyz

special_field=special_key

[seach_time]

enrich using special_field

Re: How do I index a value thats not in the message?

Stephen_Sorkin — Wed, 17 Nov 2010 20:19:57 GMT

http://answers.splunk.com/questions/5571/way-to-insert-create-field-based-on-source

Re: How do I index a value thats not in the message?

Marinus — Wed, 17 Nov 2010 22:13:13 GMT

Thanks Stephen, I'm just not sure that I'm following your logic here. Step 1, would be to locate my special value V and index it using a new meta key K::V
Step 2, would be to remove the value from the host
Am I missing something from your post?

Re: How do I index a value thats not in the message?

dwaddle — Fri, 19 Nov 2010 01:44:45 GMT

That is pretty much it - your data that is not in "_raw" becomes a new indexed field that is included in the index data when the events are indexed. The caveat is these are index-time and cannot be changed/updated/removed without dumping and reindexing data. We use this in my shop today to tag data as coming from "test" versus "prod" for files on the same source path.

Re: How do I index a value thats not in the message?

Marinus — Thu, 13 Jan 2011 22:11:25 GMT

It's fine to do it for a source, I'm doing it at the event level. Each event can have it's own unique key.