Splunk Search
Highlighted

FEATURE REQUEST: Generate new Splunk fields based on enventtypes / tags

Motivator

I am hoping this will be added to a future release.
Please UPVOTE if you agree so Splunk will prioritize this!

Overall, I feel associating metadata in Splunk is overcomplicated. I'm proposing this new feature to simplify creation of metadata which i feel requires too much work presently.

LOOKUPS
Pros of using lookups: able to generate new fields which are much simpler to search and report on than tags
Cons of using lookups: manual to maintain and/or scripting required to generate and maintain, possible performance impact

EVENT TYPES/TAGS
Pros of using tags: easy to define, better performance
Cons of using tags: too much overlapping of tag names, long tag names to avoid conflicts, requires filtering to do reporting by tags

Example: Say I have a host naming which uses prefix standards prlapacm001
pr tells me this is a prod host
the l tells me this is a linux host
ap tells me this is an app server
acm tells me this is the acme app

This should be easy to classify in splunk and generate new fields, however it is not.
I could create a lookup table of all my hosts including prlapacm001 and output new fields, however I need an entry for each host and to maintain as new hosts are added. The only good thing about this approach is that I could generate new fields like the following.
mycohostappid=ac

myco
hostappname=acme
mycohostenv=prod
mycohostos_type=linux

I could also use tags which are simple to create but are not able to generate new fields and If I want to report by tag, I get other tags, so it's not easy to use commands like | top tag without having to first pre-filter the tags with a stats command, etc.
host=prl: tag=mycohostostypelinux
host=pr
: tag=mycohostenv_prod

I'm really wishing I could have tags also generate new fields for matches.
so when the conditions, host=pr* are met, I could create a field called mycohostenv and set the value to prod (mycohostenv=prod)

0 Karma
Highlighted

Re: FEATURE REQUEST: Generate new Splunk fields based on enventtypes / tags

SplunkTrust
SplunkTrust

As a workaround until Splunk delivers the feature (if ever), try putting this regex in transforms.conf.

(?<host>(?<myco_host_env>\w\w)(?<myco_host_os_type>\w)(?<myco_host_app_id>\w\w)(?<myco_host_app_name>\w{3})(?<srvrNum>\d{3}))

You also extract fields from host at search time this way

... | rex field=host "(?<myco_host_env>\w\w)(?<myco_host_os_type>\w)(?<myco_host_app_id>\w\w)(?<myco_host_app_name>\w{3})(?<srvrNum>\d{3})" | ...
---
If this reply helps you, an upvote would be appreciated.
Highlighted

Re: FEATURE REQUEST: Generate new Splunk fields based on enventtypes / tags

In addition to the fields extracted above you could further enrich the data with lookups or eval statements to create additional fields. For example converting the "pr" to "prod" and "l" to "linux"

0 Karma
Highlighted

Re: FEATURE REQUEST: Generate new Splunk fields based on enventtypes / tags

Motivator

@richgalloway - Just curious how did u manage to type your tags, < host> < mycohostenv>, without a space in them in answer 🙂

0 Karma
Highlighted

Re: FEATURE REQUEST: Generate new Splunk fields based on enventtypes / tags

Motivator

Thanks for your input everyone. I do understand that I could create field extractions, then use a lookup table to say map additional fields such as mycohostdatacenter, etc, however, I think this task is more complicated than it should be and I would have to worry about performance impact of lookups vs tags. It seems very logical to me that I should be able to create an event type like the following

host=pr* ipfield3=150

and have it generate new fields for that I specify such as
mycodatacenter=newyork
mycodctype=primary

etc, etc...

My general argument is that Splunk should make this much simpler for such an everyday metadata creation task.

0 Karma