topic Re: Field extraction from information in field=source in Getting Data In

Field extraction from information in field=source

zindain24 — Wed, 07 Aug 2013 14:39:01 GMT

My webserver logs are sent to my indexers through a Universal Forwarder.

*Snippet from inputs.conf on the Universal Forwarder

[monitor:///path/to/apache/2.2/web/.../logs/*access_log]

disabled = false

sourcetype = access_combined

index = internet

followTail=0

With this configuration, we properly set the following fields
index = internet,
host = unixservername,
sourcetype = access_combined

The problem is, we need a field with the webserver name in segment 6 of the source:
/path/to/apache/2.2/web/.../logs/*access_log

We tried adding host_segment = 6 to the forwarder stanzas, but then we lose our true "host = unixservername" which is also necessary. Unfortunately, this information is NOT available anywhere but the source field.

So....

We can easily create a search time |rex for Splunk to process to pull the information:
|rex field=source "\/path\/to\/apache\/[0-9].[0-9]\/\w+\/(?.*?)\/"

This works well... however, I don't want my users to have to run this every time they search.

I would like the ability to add this as a Index time or Search time extraction through props and transforms -- preferably at the forwarder or indexer level. Any suggestions? Thanks for your help, ideas, and input! I'm stuck...

Jeremy

Re: Field extraction from information in field=source

Ayn — Wed, 07 Aug 2013 16:45:48 GMT

Just set up a field extraction as usual - are you familiar with how to do this in props.conf / transforms.conf? If so, it's just like a regular field extraction but you specify SOURCE_KEY = (yourfieldhere) (for REPORT style extractions that reference a transforms.conf entry) or EXTRACT = <yourregex> in <yourfieldhere> (for EXTRACT style extractions directly in props.conf).

So for the first case it'd be something like...props.conf:

[yoursourcetype]
REPORT-getfieldfromsource = getfieldfromsource

transforms.conf:

[getfieldfromsource]
SOURCE_KEY = source
REGEX = /path/to/apache/[0-9]\.[0-9]/\w+/(.*?)/
FORMAT = webserver::$1

Or, in the second case, just throw your rex statement in almost unaltered into an EXTRACT extraction in props.conf:

[yoursourcetype]
EXTRACT-getfieldfromsource = /path/to/apache/[0-9]\.[0-9]/\w+/(?<webserver>.*?)/ in source

Re: Field extraction from information in field=source

zindain24 — Wed, 07 Aug 2013 19:09:16 GMT

Fantastic Ayn, Thank you for the detailed response!