Metrics props and transforms

[coreyf311]

In our non metrics indexes we use props to create a field at search time. This field is used to create role based security. If you have access to "servergroupA" then your role is restricted to servergroup=servergroupA. How can we apply the same logic to a metric store? Applying this in props.conf does not create the servergroup field. My understanding is that we need to create dimensions for everything NOT a metric in transforms.conf using REGEX. Which we are currently doing.

LOOKUP-HOST-LIST = hostlist.csv host OUTPUTNEW servergroup as servergroup

[xpac · Accepted solution]

Splunk does not (yet?) support automatic lookups on metrics data. I guess this is because you never get to see distinct events, but only statistical aggregations of that data, so you can't do index=yourmetricsindex and see all the events in there.
You can however do a manual lookup after your first metrics command, like this

| mstats WHERE index=mymetricdata AND metric_name=cpu_util BY host
| lookup hostlist.csv host OUTPUT servergroup

But - you need to do this manually in your searches, it can't be automated. I'd recommend putting this into a macro and using that macro in your searches, for easier management.

[coreyf311]

At index time is what i am looking for in this case. The source is gmetad but we dont have access to the source at the host level. Only to the aggregate points of the data. There are multiple aggregation points each handling a couple thousand hosts. Managing at the source would be a nightmare, there are thousands of hosts per aggregation. We maintain this hostlist.csv (in splunk) which contains all the hosts. It would be easier from a management perspective to do it at index time as this would be an aggregation point of all the hosts.

I realize this might be a slow down of indexing this data, we are testing right now. Maybe its worth it, maybe its not.

[xpac]

Yeah, that sounds difficult (I've no experience on gmetad). If you can get that data written to disc, you could have a python script preprocess and then ingest it. However, you might actually be better off with a simple macro that does the lookup on search time, because you're more flexible and can do more of it on the Splunk side of things, and need to do less dirty hacks on data-index-time-level 😉

If my answer helped you, would appreciate an upvote/accept, maybe others can profit from this too 🙂

[coreyf311]

thanks, that's what I am doing now to get by. Is there anything that could be done with external_cmd in transforms to accomplish this?

[xpac]

No, I fear not, it's just another type of lookup, but you still wouldn't be able to make automatic.

Well, you could try to insert that information at index time, but then it's fixed, i.e. it's written to the index and can't be changed later on (without running a lookup again to overwrite that value).
Where are you getting your data from, what is your data source? Maybe you can preprocess it before indexing to add that information there?

[xpac]

Can you please add some details to your question? I don't really get what are trying to do, so maybe adding screenshots, example data etc. makes it easier to grasp it.

[coreyf311]

I cant provide screen shots or example data. The basics are, I am trying to create a dimension called "servergroup" in my metric store using a lookup to compare the host field to. Each host in this "hostlist.csv" has a field called "servergroup". So a host is member of a "servergroup". Lets say there are 4 groups, servergroupA, servergroupB, etc. Using the provided example, lookup the host in the hostlist.csv and output the hosts "servergroup" into the "servergroup" dimension in our metrics store. This is easy in a regular index. How do I do this for a Metrics Store?