Getting Data In

Is it possible to use wildcards in sourcetype props.conf stanzas

Builder

We have a large number of logs deserve a different sourcetype, but are effectively from the same application, and have the same format. I use a prefix on the sourcetype to indicate what app they came from, so they can be linked in searches using sourcetype=TOM* for example

eg. TOMEnvoyPayments, TOMMarketPermissions, TOMpaymentsbpmadmin

I want to extract the same field from the same part of the log entries, using the same regex. Obviously I could do this by specifying a stanza for each sourcetype specifically (three stanzas, in this example), but is it possible to use a wildcard to condense this into a single stanza, eg.

[TOM*]
REPORT-TOMgeneral = misc_MsgType_startofline

I know wildcards are possible in [source::blah] stanzas, I am interested in sourcetype stanzas specifically.

Cheers,

Glenn

Tags (2)
1 Solution

Splunk Employee
Splunk Employee

Yes, you should be able to use:

[(?:::){0}TOM*]

View solution in original post

Splunk, please add official support for wildcards in sourcetype stanzas in props.conf: for example, [TOM*].

0 Karma

Splunk Employee
Splunk Employee

I don't think we should be recommending this hack very much. If this is a common need, then make Splunk add proper support for wildcarding sourcetypes. This above thing relies on a very internal interaction between the regex expression and the way the matching is accomplished and it could easily change in the future.

It does work currently, but I would definitely recommend against leaning on it heavily -- leaving it installed for other parties at customer sites, putting it in apps, leaving an employer as a splunk admin where critical functionality relies on this without specifically passing on that information, etc, are all things that may do others a disservice in the longer run.

Splunk Employee
Splunk Employee

Agree with Jrodman. This "hack" is not supported. Actually this caused unexpected behavior in an example user tied.
Splunk doc should mention this hack is not supported.

0 Karma

Masa, your view contrasts strongly with that of Jason Conger, who in July 2014 published the Splunk blog post "Quick Tip: Wildcard Sourcetypes in Props.conf".

Jason writes "Here is a quick one [tip] I use often".

Different Splunk staff are sending (or have sent) different signals about this technique.

Splunk, could you please clarify the support for this technique?

0 Karma

Motivator

I think folks have been trying to do this for 5+ years. Much easier to use a simple pattern then add 100 rules. Why not add it officially?

Path Finder

Hey I'm curious about the use of this "hack", because it still seems to work (in 6.1.4). Was it implemented as a feature? Is it still just a hack that may or may not go away at some point? Thanks.

0 Karma

Engager

Tested with Splunk 7.1.3., still works.

Hesitant to use it though, looking for supported alternative.

0 Karma

Ultra Champion

It doesn't seem that a supported alternative exists.

0 Karma

Splunk Employee
Splunk Employee

Yes, you should be able to use:

[(?:::){0}TOM*]

View solution in original post

Communicator

Does this work for prefix wild card as well ? We have source types like UIT1XXXAPPLICATION, UIT2XXXAPPLICATION etc. The above command does'nt seem to be working.

0 Karma

Communicator

Interesting little hack there. It seems to be working, but I'm a 'tad uncomfortable using it without fully understanding what it is doing.

Could someone shed some more light as to why that stanza header works?

Contributor

It seems that official support for wildcarding sourcetypes in this context is something that comes up pretty often. Enhancement coming soon? 🙂

0 Karma

Splunk Employee
Splunk Employee

Please send your use cases into support to be passed along to product management.

0 Karma

Contributor

Also tried this, [(?:::){0}httperr-*] and it worked perfectly for my http error files which had 10 different source types

0 Karma

Communicator

Thanks gkanapathy for the solution. Glenn please mark this post as answered...

My application has sourcetypes coming in from (ie. resp101, resp201, resp202, resp203, etc..)

and in my props.conf, all I needed to define for the was:
[(?:::){0}resp*]

Works wonderfully. Someone from Splunk should put this somewhere in the props.conf documentation.

klee

0 Karma

Splunk Employee
Splunk Employee

My mistake. The regex stanza need to be [(::){0}TOM*]. Updated original

Builder

Hi, I tested this, but it doesn't seem to work?

I add this to props.conf:

[TOMConfirms]
REPORT-TOMall = miscMsgTypestartofline

... and "run |extract reload=t" and I get the "MsgType" field extracted correctly, it shows in the "pick fields" list.

I add this to props.conf:

[::TOM*]
REPORT-TOMall = miscMsgTypestartofline

And run "|extract reload=t" and the MsgType field is not being extracted for ANY TOM* sourcetypes.

0 Karma