topic Re: Applying Field Extractions across similarly named servers in Splunk Search

Applying Field Extractions across similarly named servers

mgranger1 — Tue, 29 Sep 2020 16:05:13 GMT

Hey Gang,

Here are the basics: We are running Splunk Enterprise 6.5.1. I have a distributed architecture that has two separate search heads, 4 indexers with AutoLB (but no clustering) and a deployment server (all 6.5.1 running on RedHat).

Now for the actual question. We have 70 or more websphere servers that are all similarly named (i.e. prdwas01, prdwas02, prdwas03...tstwas01, tstwas02, tstwas03....stgwas01, stgwas02, stgwas03....etc.etc.etc.). I have a series of extracted fields that I pull from the "source" value (see below):

EXTRACT-sourcefields = (?<WAS_Cluster_All>(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?<WAS_JVM_name>[a-zA-Z0-9]++)_(?[a-zA-Z0-9]++) in source

Basically, I'm pulling out some cluster and JVM characteristics from the file path of the source. Now, I would like to apply this across all 70 websphere servers. As near as I can tell, you can only specify field extractions for host, source or sourcetype. Well, I want to be able to pull these values across 70 or more different servers without having to enter in over 70 separate stanzas. If I could apply it based on index I'd be fine, but that's not an option.

I have seen some articles on answers that reference using regex as part of the stanza title for these extractions in source and sourcetype, and I have attempted some of those, and not gotten them to work. Ideally, I would like to have a stanza that said something along the lines of:

[host::.{3}was\d\d]

That would represent any hostname that had 3 characters, then the letters 'was' and then two digits and have it then apply the included field extractions. I spoke with my sales engineer, and he claimed that it wasn't possible to use regex as part of the stanza header, and I wasn't able to get any of the examples from answers to work, so I decided to ask a question that specifically dealt with what I was trying to do.

Any thoughts or information would be very appreciated.

Thanks,
Matthew Granger

Re: Applying Field Extractions across similarly named servers

darrenfuller — Tue, 03 Oct 2017 20:55:09 GMT

Hi Matthew,

I think you are asking for a way to include the hostname in the field names... I would think there's a better way to do that on the fly rather than trying to extract a tonne of fields with the hostname hard coded...that makes future homework as servers are added/decommissioned... who wants to miss monitoring on the new server because poor overworked Matthew forgot to add the new server to the list...

What about something like this:

start with this as a set of base data to illustrate the point:

| makeresults | eval data="1507063392.123,was01,17,18,19,20|1507063394.345,was01,21,22,23,24|1507063396.567,was02,25,26,27,28"
| makemv delim="|" data
| mvexpand data
| rex field=data "^(?<_time>[^\,]+)\,(?<host>[^\,]+)\,(?<datapoint1>\d+)\,(?<datapoint2>\d+)\,(?<datapoint3>\d+)\,(?<datapoint4>\d+)"
| table _time host datapoint*

so i have multiple data points for each host... like so:

_time                         host      datapoint1  datapoint2  datapoint3  datapoint4 
2017-10-03 16:43:12.123     was01       17          18          19          20         
2017-10-03 16:43:14.345     was01       21          22          23          24         
2017-10-03 16:43:16.567     was02       25          26          27          28

From your question, I read that you want to get each datapoint1 value to be hostname-datapoint1

So to make that happen, do this:

| makeresults | eval data="1507063392.123,was01,17,18,19,20|1507063394.345,was01,21,22,23,24|1507063396.567,was02,25,26,27,28"
| makemv delim="|" data
| mvexpand data
| rex field=data "^(?<_time>[^\,]+)\,(?<host>[^\,]+)\,(?<datapoint1>\d+)\,(?<datapoint2>\d+)\,(?<datapoint3>\d+)\,(?<datapoint4>\d+)"
| table _time host datapoint*
| eval measurename=host+"-datapoint1"
| fields + _time measurename datapoint1
| chart avg(datapoint1) by measurename

Which outputs...:

measurename         AVG
----------------       ---
was01-datapoint1       19
was02-datapoint1       25

Re: Applying Field Extractions across similarly named servers

mgranger1 — Wed, 04 Oct 2017 12:50:38 GMT

That's not at all what I'm trying to do. I have all the host names. The host names are not the issue. The issue is that I want to apply the SAME field extraction to multiple hosts without having to create multiple field extraction entries in props.conf. Right now, in props.conf I have to have the following stanzas:

[host::prdwas05]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source

[host::prdwas06]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source

The extractions work fine. I have no problem with the extractions at all. What I'm asking is, right now I have to have a separate stanza for EVERY websphere server, and I have over 70 websphere servers. So, instead of having one stanza that is [host::prdwas05] and ANOTHER stanza for [host::prdwas06] and ANOTHER and ANOTHER and ANOTHER.....(for up to 70 of them), I would like to have a single stanza that pattern matches the name of the server, and then applies the field extractions. Something like:

[host::.{3}was\d\d]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source

However, that doesn't work.

Re: Applying Field Extractions across similarly named servers

mgranger1 — Wed, 04 Oct 2017 12:56:04 GMT

Okay, HTML encoding screwed up my sample extractions (darn less than and greater than symbols), but the point stands, My extractions are working when I apply them specifically to one server. I'm trying to find a way to apply the same field extractions to Multiple servers based on a hostname pattern match.