Splunk Search

Can you regex or wildcard props Sourcetype stanzas?

thisissplunk
Builder

Still haven't seen an official answer to this. Source and host can use regex patterns, but sourcetypes cannot. Even a splunk blog recommends a way that is well... not recommended: http://blogs.splunk.com/2014/07/31/quick-tip-wildcard-sourcetypes-in-props-conf/

I want to do what the acme: example he is doing in that post, but that doesn't seem to be supported.

1 Solution

ben_leung
Builder

Here is a test performed by Splunk support Case: 219804, Props stanza not honoring a wildcard source:

Where...
Splunk Version is 6.0.5
source=/User/TEST/LOG/SMP1234.log
sourcetype=test1

#[(?:::){0}SMP.+]                          # does not work
#[source::/Users/TEST/LOGS/SMP...]       # works
#[::SMP*]                                  # does not work
#[source::.../SMP*]                        # works
#[source::.../SMP...*]                     # works
#[::.../SMP*]                              # does not work
#[*::.../SMP*]                             # does not work
#[...::.../SMP*]                           # does not work
#[(?:::){0}SMP.*]                          # does not work
#[source::\/Users\/TEST\/LOGS\/SMP...]   # works
#[source::\/Users\/TEST\/LOGS\/SMP...*]  # works
#[source::/Users/TEST/LOGS/SMP1234.log]  # works
#[test1]                                   # works
#[test1*]                                  # does not work
#[sourcetype::test1]                       # does not work

View solution in original post

yannK
Splunk Employee
Splunk Employee

Sourcetypes stanza in props.conf do not support regexes matching.
The answer above is an hack, that is not supported and non maintained by splunk codebase.

So do not use and if it works by sheer luck, do not expect it to work in all versions and all situations. ( e.g. do not rely on that hack for any critical business usage)

ben_leung
Builder

Here is a test performed by Splunk support Case: 219804, Props stanza not honoring a wildcard source:

Where...
Splunk Version is 6.0.5
source=/User/TEST/LOG/SMP1234.log
sourcetype=test1

#[(?:::){0}SMP.+]                          # does not work
#[source::/Users/TEST/LOGS/SMP...]       # works
#[::SMP*]                                  # does not work
#[source::.../SMP*]                        # works
#[source::.../SMP...*]                     # works
#[::.../SMP*]                              # does not work
#[*::.../SMP*]                             # does not work
#[...::.../SMP*]                           # does not work
#[(?:::){0}SMP.*]                          # does not work
#[source::\/Users\/TEST\/LOGS\/SMP...]   # works
#[source::\/Users\/TEST\/LOGS\/SMP...*]  # works
#[source::/Users/TEST/LOGS/SMP1234.log]  # works
#[test1]                                   # works
#[test1*]                                  # does not work
#[sourcetype::test1]                       # does not work

thisissplunk
Builder

Thank you for this. Although it doesn't give a solution for pure [sourcetype] stanzas, I think I should be able to work out a [source::] stanza since that supports wildcards... in some fashion or other as I can see by your post.

You didn't happen to test something like [source::.../LOG/...] did you? This is my actual use case.

0 Karma

ben_leung
Builder

No problem. Source is the way to go since those stanzas will overwrite clashes between sourcetype/host configurations.

Potential enhancement request to the product to allow wildcards on source types? I do not feel the need for this if source accepts wildcards.

0 Karma

thisissplunk
Builder

So I just tested this (using the example data above):

 [source::.../LOG/...]

It does not work. I switched out [default] for the above. [Default] of course works.

An example set of log files I have is:

/User/TEST/LOG/SMP.log
/User/TEST/LOG/PPC.log
/User/TEST/LOG/ABC.log
/User/TEST/LOG/LMO.log
0 Karma

ben_leung
Builder
* matches non-directory separators

Try adding the star after the trailing 3 periods.

0 Karma

thisissplunk
Builder

My apologies, my test config had a duplicate extract class (with the incorrect one overriding the correct one).

[source::.../LOG/...] does indeed work! Thanks again.

0 Karma

jplumsdaine22
Influencer

I think you're misunderstanding the blog post. This is completely supported:

[(?::){0}acme:*]

The "CAUTION – THIS DOES NOT WORK" refers to the first example:

[acme:*]

makelovenotwar
Path Finder

Can you explain the regex? it looks like (?::) looks for any field identifier... but what about {0}? Is that for zero of that field identifier? I am asking because I am trying the stanza below and it doesn't seem to be working:

[source::(top)|(ps)|(bandwidth)]

Context: sending ta_nix data from UF to HF, then using RFS to send to flat file. The stanza aboves simply routes the data to the destination using ingest actions. 

Thanks!

0 Karma

thisissplunk
Builder

Thanks for the post, however If you read further into the comments, or google other splunk answers, you can see that it is unsupported and causes issues:

I agree with Joshua Rodman here.
Recently we found unexpected behavior
using this. There is no official
documentation for this functionality.
At this point, introducing this
functionality to users without letting
user know “not supported
functionality” is not recommended. At
the same time, requesting Splunk to
support sourcetuype regex is very
reasonable.

Another comment eludes to how this is more of a workaround/regex injection than a real solution:

Seconding. Is there a reason that there has to be a zero-length match at the start?

0 Karma

machiel
Path Finder

I implemented the "[(?::){0}acme:*]"-method without any issues to match many sourcetypes at once.
Any Splunk user that notices any strange behaviors using this method, is hereby invited to post more details about when these issues occur.

Without details, I regard the zero-length match as a reliable solution.

0 Karma

jplumsdaine22
Influencer

I guess the problem is that as its not officially supported, it could disappear in a new release. So I would be cautious when using this.

0 Karma

jplumsdaine22
Influencer

From props.conf.spec

[source::] and [host::] stanza match language:

Match expressions must match the entire name, not just a substring. If you are familiar with regular expressions, match expressions are based on a full implementation of PCRE with the translation of ..., * and . Thus . matches a period, * matches non-directory separators, and ... matches any number of any characters.

"full implementation of PCRE". So I don't see how this is unsupported. Can you provide a link to other splunk answers that give examples of the issues encountered? I don't mind if I'm wrong, but it would be useful for other readers

0 Karma

thisissplunk
Builder

Yes you are correct there, however, this is not supported for Sourcetype stanzas. Testing confirms this as well.

spec can be:
1. sourcetype, the source type of an event.
2. host::, where is the host, or host-matching pattern, for an
event.
3. source::, where is the source, or source-matching
pattern, for an event.

My apologies for not specifying Sourcetype was what I was looking for.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...