topic Re: Is it possible to use wildcards in sourcetype props.conf stanzas in Getting Data In

Is it possible to use wildcards in sourcetype props.conf stanzas

Glenn — Fri, 29 Oct 2010 20:47:15 GMT

We have a large number of logs deserve a different sourcetype, but are effectively from the same application, and have the same format. I use a prefix on the sourcetype to indicate what app they came from, so they can be linked in searches using sourcetype=TOM* for example

eg. TOMEnvoyPayments, TOMMarketPermissions, TOMpaymentsbpmadmin

I want to extract the same field from the same part of the log entries, using the same regex. Obviously I could do this by specifying a stanza for each sourcetype specifically (three stanzas, in this example), but is it possible to use a wildcard to condense this into a single stanza, eg.

[TOM*]
REPORT-TOMgeneral = misc_MsgType_startofline

I know wildcards are possible in [source::blah] stanzas, I am interested in sourcetype stanzas specifically.

Cheers,

Glenn

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

gkanapathy — Fri, 29 Oct 2010 20:57:35 GMT

Yes, you should be able to use:

[(?:::){0}TOM*]

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Glenn — Mon, 28 Sep 2020 09:20:06 GMT

Hi, I tested this, but it doesn't seem to work?

I add this to props.conf:

[TOMConfirms]
REPORT-TOMall = misc_MsgType_startofline

... and "run |extract reload=t" and I get the "MsgType" field extracted correctly, it shows in the "pick fields" list.

I add this to props.conf:

[::TOM*]
REPORT-TOMall = misc_MsgType_startofline

And run "|extract reload=t" and the MsgType field is not being extracted for ANY TOM* sourcetypes.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

gkanapathy — Sat, 20 Nov 2010 13:39:59 GMT

My mistake. The regex stanza need to be [(::){0}TOM*]. Updated original

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

klee310 — Tue, 20 Sep 2011 02:52:00 GMT

Thanks gkanapathy for the solution. Glenn please mark this post as answered...

My application has sourcetypes coming in from (ie. resp101, resp201, resp202, resp203, etc..)

and in my props.conf, all I needed to define for the was:
[(?:::){0}resp*]

Works wonderfully. Someone from Splunk should put this somewhere in the props.conf documentation.

klee

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

richnavis — Thu, 12 Jan 2012 19:49:49 GMT

Also tried this, [(?:::){0}httperr-*] and it worked perfectly for my http error files which had 10 different source types

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

jrodman — Thu, 19 Jan 2012 10:44:12 GMT

I don't think we should be recommending this hack very much. If this is a common need, then make Splunk add proper support for wildcarding sourcetypes. This above thing relies on a very internal interaction between the regex expression and the way the matching is accomplished and it could easily change in the future.

It does work currently, but I would definitely recommend against leaning on it heavily -- leaving it installed for other parties at customer sites, putting it in apps, leaving an employer as a splunk admin where critical functionality relies on this without specifically passing on that information, etc, are all things that may do others a disservice in the longer run.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

cmeo — Thu, 16 Feb 2012 04:21:59 GMT

It seems that official support for wildcarding sourcetypes in this context is something that comes up pretty often. Enhancement coming soon? 🙂

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Ricapar — Tue, 11 Jun 2013 14:19:20 GMT

Interesting little hack there. It seems to be working, but I'm a 'tad uncomfortable using it without fully understanding what it is doing.

Could someone shed some more light as to why that stanza header works?

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

jrodman — Fri, 31 Oct 2014 16:52:09 GMT

Please send your use cases into support to be passed along to product management.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Sriram — Mon, 28 Sep 2020 18:49:53 GMT

Does this work for prefix wild card as well ? We have source types like UIT1_XXX_APPLICATION, UIT2_XXX_APPLICATION etc. The above command does'nt seem to be working.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

mparks11 — Fri, 29 May 2015 02:19:43 GMT

Hey I'm curious about the use of this "hack", because it still seems to work (in 6.1.4). Was it implemented as a feature? Is it still just a hack that may or may not go away at some point? Thanks.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Masa — Fri, 12 Feb 2016 21:34:13 GMT

Agree with Jrodman. This "hack" is not supported. Actually this caused unexpected behavior in an example user tied.
Splunk doc should mention this hack is not supported.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

bandit — Thu, 28 Apr 2016 15:46:01 GMT

I think folks have been trying to do this for 5+ years. Much easier to use a simple pattern then add 100 rules. Why not add it officially?

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

alanos — Mon, 22 Oct 2018 13:28:24 GMT

Tested with Splunk 7.1.3., still works.

Hesitant to use it though, looking for supported alternative.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

ddrillic — Mon, 22 Oct 2018 13:39:57 GMT

It doesn't seem that a supported alternative exists.

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Graham_Hanningt — Tue, 26 Nov 2019 00:57:57 GMT

Splunk, please add official support for wildcards in sourcetype stanzas in props.conf: for example, [TOM*].

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Graham_Hanningt — Tue, 26 Nov 2019 01:43:02 GMT

Masa, your view contrasts strongly with that of Jason Conger, who in July 2014 published the Splunk blog post "Quick Tip: Wildcard Sourcetypes in Props.conf".

Jason writes "Here is a quick one [tip] I use often".

Different Splunk staff are sending (or have sent) different signals about this technique.

Splunk, could you please clarify the support for this technique?

Re: Is it possible to use wildcards in sourcetype props.conf stanzas

Esky73 — Mon, 26 Apr 2021 23:45:57 GMT

This is still working in 8.1x

But also please vote for an official method for wildcarding sourcertypes :

https://ideas.splunk.com/ideas/E-I-11