topic Re: Combining fields pre-search in Getting Data In

Combining fields pre-search

foomanjee — Wed, 15 May 2013 15:06:04 GMT

Hello, I have what may or may not be a bit of a unique issue regarding extracted fields.

We've got a few webservers and we use a relatively unique custom log format, so I've had to manually extract fields. One of the fields is the HTTP header hostname that Apache sees.

The issue is this: We have several sites with multiple hostnames that all serve the same content. For example:
www.domain.com
origin-www.domain.com

The CustomLog in Apache shows both hostnames in their respective log lines, so Splunk does as well.

This results in our Splunk generated graphs showing both fields separately, rather than combined, which results in incorrect numbers (hits/sec, etc).

What I need is to combine those two. In other words, I want splunk to combine the origin-www.domain.com field with the www.domain.com field.

Any ideas on how this can be done?

Re: Combining fields pre-search

sideview — Wed, 15 May 2013 18:01:42 GMT

One simple way is that before you transform the data with stats/chart/timechart, just clean it up a little with eval.

| eval host=replace(host,"origin-www.","www.")

If there are many different ways in which duplicates exist, but the list is relatively stable over time another tool you might use is a lookup, to map each duplicate hostname to the main one.

Re: Combining fields pre-search

sideview — Wed, 15 May 2013 18:01:43 GMT

One simple way is that before you transform the data with stats/chart/timechart, just clean it up a little with eval.

| eval host=replace(host,"origin-www.","www.")

If there are many different ways in which duplicates exist, but the list is relatively stable over time another tool you might use is a lookup, to map each duplicate hostname to the main one.

Re: Combining fields pre-search

foomanjee — Thu, 16 May 2013 18:22:56 GMT

Thanks. I'm having issues getting this to work as an automatic lookup. I've got the CSV file up, doing something simple like:

vhost,old
www.domain.com,origin-www.domain.com
www.domain2.com,origin-www.domain2.com
etc

I've got the appropriate definitions configured, and my lookups are basic:
input:
vhost=vhost

output:
old=old

I'm not seeing either new field show up in the search results, and there are no replacements being made.

Any ideas?

Re: Combining fields pre-search

foomanjee — Thu, 16 May 2013 18:49:45 GMT

I should add that I've already got a field called 'vhost'. What I'm trying to do is search for vhost=www.domain.com, and have it show both www.domain.com and origin-www.domain.com entries, without having to use "replace". I have about 50 domains, so using "replace" would be rather ugly.