How to set sourcetype using regex on source?

[Fonzie2k]

Hello fellow Splunkers.

I am trying to set the sourcetype name using a part of the source path. I've read the answers from the same question on the community, but i just cant get it working, so ill give it a shot and ask here.

The goal:
Set sourcetype name from the third folder in the source path.
For example, automaticly set sourcetype to "foobar" from logs collected from source C:\data\Logs\foobar\logfile.log

What i have tried:

(on the universal forwarder)
inputs.conf:
[monitor://C:\data\Logs\...\*]
index = main

(on the indexer receiving from the universal forwarder)
props.conf:
[source::C:\data\Logs\*]
TRANSFORMS-changesourcetype = changesourcetype

transforms.conf:
[changesourcetype]
SOURCE_KEY = MetaData:Source
REGEX = C:\\data\\[^\\]+\\([^\\]+)\\
#REGEX = C:\\data\\Logs\\(\w+)\\ (Commented out: for debugging purposes)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

I've also tried (for debugging-reasons) solving it differently, by tagging a temporary sourcetype at the UFW:
inputs.conf:
[monitor://C:\data\Logs\...\*]
index = main
sourcetype = changemeplease

props.conf:
[changemeplease]
TRANSFORMS-changesourcetype = changesourcetype

transforms.conf:
[changesourcetype]
SOURCE_KEY = MetaData:Source
REGEX = C:\\data\\[^\\]+\\([^\\]+)\\
#REGEX = C:\\data\\Logs\\(\w+)\\ (Commented out: for debugging purposes)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

The data gets forwarded and indexed, but the transforms seem to not hit, what so ever.
Any suggestions on what i am doing wrong here?

[Fonzie2k]

Thanks for the reply.

I am pretty sure it has nothing to do with the regex.
For debugging purposes I tried using a static value instead of an regex, but the transforms still doesnt rewrite the sourcetype:

transforms.conf
[changesourcetype]
SOURCE_KEY = MetaData:Source
FORMAT = sourcetype::test
DEST_KEY = MetaData:Sourcetype

One thing i am unsure of:
Does this only work if the data passes trough an heavy forwarder before it reaches the indexer?
I am trying to achieve this on a single node Splunk Enterprise instance.

[isoutamo]

Data must go trough that HF to apply these props and transforms. If you have these on separate HF and source data is on separate node then it didn't work. You must send events to that HF to apply these configurations to it.

Also you must remember that 

[<spec>]
* This stanza enables properties for a given <spec>.
* A props.conf file can contain multiple stanzas for any number of
  different <spec>.
* Follow this stanza name with any number of the following setting/value
  pairs, as appropriate for what you want to do.
* If you do not set a setting for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an
                 event.
3. source::<source>, where <source> is the source, or source-matching
                     pattern, for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type
                     classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed
                            source type classification rule.
                            These are only considered as a last resort
                            before generating a new source type based on the
                            source seen.

**[<spec>] stanza precedence:**

For settings that are specified in multiple categories of matching [<spec>]
stanzas, [host::<host>] settings override [<sourcetype>] settings.
Additionally, [source::<source>] settings override both [host::<host>]
and [<sourcetype>] settings.

This means that precedence for those configurations are above.

[Fonzie2k]

Uhm.. I do not have a separate HF.
I only have a single Splunk Enterprise instance:
Splunk Universal Forwarder -> Splunk Enterprise

Are you saying that i need another instance of Splunk Enterprise, configured as a HF, to achieve this?
Splunk Universal Forwarder -> Splunk Enterprise (HF) -> Splunk Enterprise (SH and Indexer)

[isoutamo]

UF -> Indexer / all in one instance is enough, no need for separate HF. Usually you should send all events directly from UF to indexers. Just install those configuration on indexer (Your splunk enter. server). 

Are you sure that you haven't both source, sourcetype and/or host configuration on props.conf? Only one of those can be effective and the selection between those are done by precedence (see my previous post).

[Fonzie2k]

Yes, i am sure i do not have two configurations at the same time in props.conf.

But i've tried with both [source::<source>] and [sourcetype] at different times, just to try to figure out why this props.conf does not "hit", because that seems to be the problem.

To summary what ive tried regarding props.conf:
* Set a temporary sourcetype on the UniversalForwarder, and target that sourcetype in props.conf
* Do not set any sourcetype on the UniversalForwarder, and instead target the source in props.conf

[changemeplease]
TRANSFORMS-changesourcetype = changesourcetype

[source::C:\data\Logs\*]
TRANSFORMS-changesourcetype = changesourcetype

[isoutamo]

On props.conf is example

When you specify Windows-based file paths as part of a [source::<source>]
stanza, you must escape any backslashes contained within the specified file
path.

Example: [source::c:\\path_to\\file.txt]

and you have

[source::C:\data\Logs\*]

I suppose that you must use \\ instead of \ on source:: stanza. 

[isoutamo]

I suppose that the issue is wrong amount of escape characters?

You should try it with

REGEX = C:\\\data\\\[^\\\]+\\\([^\\\]+)\\\\

Or if not needed drop last 4 \ away.

You could test it with

| makeresults 
| eval source="C:\data\Logs\foobar\logfile.log"
| rex field=source "C:\\\data\\\[^\\\]+\\\(?<BAR>[^\\\]+)\\\\"
| table source BAR

 The rule of thumb with rex and \ is "just add one more \ and check if it work" 😉

r. Ismo

[Fonzie2k · Accepted solution]

Ok, so i found the problem. I took a look at the transforms from the web gui afterwards, and the error was clear:

So the config files is not correctly readed by Splunk. Its like it doesnt register the "new line" afte the regex key value. Does anyone have any idea what causes this?
(editing it from the web gui, made a simular file in the apps local folder, and fixed the problem).

Config file: