Splunk Search

Can you regex or wildcard props Sourcetype stanzas?

Builder

Still haven't seen an official answer to this. Source and host can use regex patterns, but sourcetypes cannot. Even a splunk blog recommends a way that is well... not recommended: http://blogs.splunk.com/2014/07/31/quick-tip-wildcard-sourcetypes-in-props-conf/

I want to do what the acme: example he is doing in that post, but that doesn't seem to be supported.

1 Solution

Builder

Here is a test performed by Splunk support Case: 219804, Props stanza not honoring a wildcard source:

Where...
Splunk Version is 6.0.5
source=/User/TEST/LOG/SMP1234.log
sourcetype=test1

#[(?:::){0}SMP.+]                          # does not work
#[source::/Users/TEST/LOGS/SMP...]       # works
#[::SMP*]                                  # does not work
#[source::.../SMP*]                        # works
#[source::.../SMP...*]                     # works
#[::.../SMP*]                              # does not work
#[*::.../SMP*]                             # does not work
#[...::.../SMP*]                           # does not work
#[(?:::){0}SMP.*]                          # does not work
#[source::\/Users\/TEST\/LOGS\/SMP...]   # works
#[source::\/Users\/TEST\/LOGS\/SMP...*]  # works
#[source::/Users/TEST/LOGS/SMP1234.log]  # works
#[test1]                                   # works
#[test1*]                                  # does not work
#[sourcetype::test1]                       # does not work

View solution in original post

Splunk Employee
Splunk Employee

Sourcetypes stanza in props.conf do not support regexes matching.
The answer above is an hack, that is not supported and non maintained by splunk codebase.

So do not use and if it works by sheer luck, do not expect it to work in all versions and all situations. ( e.g. do not rely on that hack for any critical business usage)

Builder

Here is a test performed by Splunk support Case: 219804, Props stanza not honoring a wildcard source:

Where...
Splunk Version is 6.0.5
source=/User/TEST/LOG/SMP1234.log
sourcetype=test1

#[(?:::){0}SMP.+]                          # does not work
#[source::/Users/TEST/LOGS/SMP...]       # works
#[::SMP*]                                  # does not work
#[source::.../SMP*]                        # works
#[source::.../SMP...*]                     # works
#[::.../SMP*]                              # does not work
#[*::.../SMP*]                             # does not work
#[...::.../SMP*]                           # does not work
#[(?:::){0}SMP.*]                          # does not work
#[source::\/Users\/TEST\/LOGS\/SMP...]   # works
#[source::\/Users\/TEST\/LOGS\/SMP...*]  # works
#[source::/Users/TEST/LOGS/SMP1234.log]  # works
#[test1]                                   # works
#[test1*]                                  # does not work
#[sourcetype::test1]                       # does not work

View solution in original post

Builder

Thank you for this. Although it doesn't give a solution for pure [sourcetype] stanzas, I think I should be able to work out a [source::] stanza since that supports wildcards... in some fashion or other as I can see by your post.

You didn't happen to test something like [source::.../LOG/...] did you? This is my actual use case.

0 Karma

Builder

No problem. Source is the way to go since those stanzas will overwrite clashes between sourcetype/host configurations.

Potential enhancement request to the product to allow wildcards on source types? I do not feel the need for this if source accepts wildcards.

0 Karma

Builder

So I just tested this (using the example data above):

 [source::.../LOG/...]

It does not work. I switched out [default] for the above. [Default] of course works.

An example set of log files I have is:

/User/TEST/LOG/SMP.log
/User/TEST/LOG/PPC.log
/User/TEST/LOG/ABC.log
/User/TEST/LOG/LMO.log
0 Karma

Builder
* matches non-directory separators

Try adding the star after the trailing 3 periods.

0 Karma

Builder

My apologies, my test config had a duplicate extract class (with the incorrect one overriding the correct one).

[source::.../LOG/...] does indeed work! Thanks again.

0 Karma

Influencer

I think you're misunderstanding the blog post. This is completely supported:

[(?::){0}acme:*]

The "CAUTION – THIS DOES NOT WORK" refers to the first example:

[acme:*]

Builder

Thanks for the post, however If you read further into the comments, or google other splunk answers, you can see that it is unsupported and causes issues:

I agree with Joshua Rodman here.
Recently we found unexpected behavior
using this. There is no official
documentation for this functionality.
At this point, introducing this
functionality to users without letting
user know “not supported
functionality” is not recommended. At
the same time, requesting Splunk to
support sourcetuype regex is very
reasonable.

Another comment eludes to how this is more of a workaround/regex injection than a real solution:

Seconding. Is there a reason that there has to be a zero-length match at the start?

0 Karma

Path Finder

I implemented the "[(?::){0}acme:*]"-method without any issues to match many sourcetypes at once.
Any Splunk user that notices any strange behaviors using this method, is hereby invited to post more details about when these issues occur.

Without details, I regard the zero-length match as a reliable solution.

0 Karma

Influencer

I guess the problem is that as its not officially supported, it could disappear in a new release. So I would be cautious when using this.

0 Karma

Influencer

From props.conf.spec

[source::] and [host::] stanza match language:

Match expressions must match the entire name, not just a substring. If you are familiar with regular expressions, match expressions are based on a full implementation of PCRE with the translation of ..., * and . Thus . matches a period, * matches non-directory separators, and ... matches any number of any characters.

"full implementation of PCRE". So I don't see how this is unsupported. Can you provide a link to other splunk answers that give examples of the issues encountered? I don't mind if I'm wrong, but it would be useful for other readers

0 Karma

Builder

Yes you are correct there, however, this is not supported for Sourcetype stanzas. Testing confirms this as well.

spec can be:
1. sourcetype, the source type of an event.
2. host::, where is the host, or host-matching pattern, for an
event.
3. source::, where is the source, or source-matching
pattern, for an event.

My apologies for not specifying Sourcetype was what I was looking for.

0 Karma