topic Re: Applying Field Extractions to Several Sourcetypes in Splunk Search

Applying Field Extractions to Several Sourcetypes

niall_munnelly — Wed, 02 Apr 2014 21:16:13 GMT

Hi,
Per a policy I've inherited, we're separating our business groups' web server logs into separate sourcetypes. It seems it would be easier if I could create a transform and use tags, but the dude running the show is averse to these solutions.

So we have maybe 20 sourcetypes, named

foo_apache_access
bar_apache_access
baz_apache_access
quux_apache_access

...and so on. This has made creating a standard collection of extracted fields tricky; it seems like I have to create the extractions within every sourcetype, which is manual and slow. I'm also not getting consistent results for some of the extracted fields in one sourcetype vs another, so I may have to break my combined regex patterns into several individual extractions, making the whole process way slower and sloppier, still.

Wildcard characters appear to be parsed literally, so *_apache_access didn't work as hoped. Have I missed something? Is there a wildcard or UI command I can use to effect this? Thanks.

Re: Applying Field Extractions to Several Sourcetypes

jhowkins — Wed, 02 Apr 2014 23:08:54 GMT

I ran into a similar issue where I have 60 different sourcetypes all belonging to the same host.

Depending on the contents of your files, you could create a transform and/or extraction that captures similar things contained within each file. When you create your regex pattern, just make sure it qualifies something within the data that distinguishes it from everything else.

If you can provide an example of the file contents and your intentions, I can provide a better answer.

Re: Applying Field Extractions to Several Sourcetypes

jhowkins — Tue, 08 Apr 2014 00:50:50 GMT

To answer your question, yes there is a way to use wildcards to use a single extraction against multiple sourcetypes. See my answer below.

Re: Applying Field Extractions to Several Sourcetypes

jhowkins — Tue, 08 Apr 2014 02:15:02 GMT

If you would rather create an extraction by sourcetype and using a wildcard, you can do it but it requires a little regex trickery...

To capture the sourcetypes you provided, you can use the following as your sourcetype;

(?::){0}*_apache_access

Re: Applying Field Extractions to Several Sourcetypes

niall_munnelly — Tue, 08 Apr 2014 16:04:35 GMT

Thanks for your replies, JHowkins; I'll see how i can make this fit. With some 20-odd sourcetypes and a projected 5 or 6 extractions per sourcetype, anything I can do to avoid creating them manually will be blessing.

Re: Applying Field Extractions to Several Sourcetypes

niall_munnelly — Wed, 23 Apr 2014 15:35:33 GMT

This totally worked! Thanks!

Re: Applying Field Extractions to Several Sourcetypes

UCOP — Thu, 26 May 2016 13:40:19 GMT

I have the same problem. I need a regular expression to identify several sourcetypes. Examples of the source types are as follows:

application-ucop-topcop-pub:default-2
application-ucop-bft-nais-sub:default
application-ucop-bft-pub:default

I know that "application-ucop" and ":default" will remain constant, but in between them will be different one word or two words separated by a "-". Also some of them have nothing after :default and some have a "-" followed by a number.

I have never worked with regular expressions and I am trying to simplify my field extractions. Thank you.

Re: Applying Field Extractions to Several Sourcetypes

ss026381 — Mon, 26 Jun 2017 17:01:45 GMT

have you ever find out the answer for this?

Re: Applying Field Extractions to Several Sourcetypes

HattrickNZ — Sun, 03 Sep 2017 22:47:25 GMT

What if the soucetype is not in the text where you want to do the extraction?

2 source types
XBSN
QSN

need 2 extractions on each sourctype:
^[^/\n]/\w+:(?P[^"]+)
^[^/\n]/\w+:(?P[^"]+)

Where do I specify the sourcetype?