Getting Data In

Is it possible to use wildcards in sourcetype props.conf stanzas

Glenn
Builder

We have a large number of logs deserve a different sourcetype, but are effectively from the same application, and have the same format. I use a prefix on the sourcetype to indicate what app they came from, so they can be linked in searches using sourcetype=TOM* for example

eg. TOMEnvoyPayments, TOMMarketPermissions, TOMpaymentsbpmadmin

I want to extract the same field from the same part of the log entries, using the same regex. Obviously I could do this by specifying a stanza for each sourcetype specifically (three stanzas, in this example), but is it possible to use a wildcard to condense this into a single stanza, eg.

[TOM*]
REPORT-TOMgeneral = misc_MsgType_startofline

I know wildcards are possible in [source::blah] stanzas, I am interested in sourcetype stanzas specifically.

Cheers,

Glenn

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Yes, you should be able to use:

[(?:::){0}TOM*]

View solution in original post

Esky73
Builder

This is still working in 8.1x

But also please vote for an official method for wildcarding sourcertypes :

https://ideas.splunk.com/ideas/E-I-11

0 Karma

Graham_Hanningt
Builder

Splunk, please add official support for wildcards in sourcetype stanzas in props.conf: for example, [TOM*].

0 Karma

jrodman
Splunk Employee
Splunk Employee

I don't think we should be recommending this hack very much. If this is a common need, then make Splunk add proper support for wildcarding sourcetypes. This above thing relies on a very internal interaction between the regex expression and the way the matching is accomplished and it could easily change in the future.

It does work currently, but I would definitely recommend against leaning on it heavily -- leaving it installed for other parties at customer sites, putting it in apps, leaving an employer as a splunk admin where critical functionality relies on this without specifically passing on that information, etc, are all things that may do others a disservice in the longer run.

Masa
Splunk Employee
Splunk Employee

Agree with Jrodman. This "hack" is not supported. Actually this caused unexpected behavior in an example user tied.
Splunk doc should mention this hack is not supported.

0 Karma

Graham_Hanningt
Builder

Masa, your view contrasts strongly with that of Jason Conger, who in July 2014 published the Splunk blog post "Quick Tip: Wildcard Sourcetypes in Props.conf".

Jason writes "Here is a quick one [tip] I use often".

Different Splunk staff are sending (or have sent) different signals about this technique.

Splunk, could you please clarify the support for this technique?

0 Karma

bandit
Motivator

I think folks have been trying to do this for 5+ years. Much easier to use a simple pattern then add 100 rules. Why not add it officially?

mparks11
Path Finder

Hey I'm curious about the use of this "hack", because it still seems to work (in 6.1.4). Was it implemented as a feature? Is it still just a hack that may or may not go away at some point? Thanks.

0 Karma

alanos
Engager

Tested with Splunk 7.1.3., still works.

Hesitant to use it though, looking for supported alternative.

0 Karma

ddrillic
Ultra Champion

It doesn't seem that a supported alternative exists.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Yes, you should be able to use:

[(?:::){0}TOM*]

View solution in original post

Sriram
Communicator

Does this work for prefix wild card as well ? We have source types like UIT1_XXX_APPLICATION, UIT2_XXX_APPLICATION etc. The above command does'nt seem to be working.

0 Karma

Ricapar
Communicator

Interesting little hack there. It seems to be working, but I'm a 'tad uncomfortable using it without fully understanding what it is doing.

Could someone shed some more light as to why that stanza header works?

cmeo
Contributor

It seems that official support for wildcarding sourcetypes in this context is something that comes up pretty often. Enhancement coming soon? 🙂

0 Karma

jrodman
Splunk Employee
Splunk Employee

Please send your use cases into support to be passed along to product management.

0 Karma

richnavis
Contributor

Also tried this, [(?:::){0}httperr-*] and it worked perfectly for my http error files which had 10 different source types

0 Karma

klee310
Communicator

Thanks gkanapathy for the solution. Glenn please mark this post as answered...

My application has sourcetypes coming in from (ie. resp101, resp201, resp202, resp203, etc..)

and in my props.conf, all I needed to define for the was:
[(?:::){0}resp*]

Works wonderfully. Someone from Splunk should put this somewhere in the props.conf documentation.

klee

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

My mistake. The regex stanza need to be [(::){0}TOM*]. Updated original

Glenn
Builder

Hi, I tested this, but it doesn't seem to work?

I add this to props.conf:

[TOMConfirms]
REPORT-TOMall = misc_MsgType_startofline

... and "run |extract reload=t" and I get the "MsgType" field extracted correctly, it shows in the "pick fields" list.

I add this to props.conf:

[::TOM*]
REPORT-TOMall = misc_MsgType_startofline

And run "|extract reload=t" and the MsgType field is not being extracted for ANY TOM* sourcetypes.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!