Why is my sourcetype configuration in props.conf not being used for the sourcetype defined by transforms.conf?

[michael_sleep]

Hey there,

We have a distributed Splunk environment... so, we have universal forwarders sending data to a heavy forwarder, sending data to an indexer, etc, etc. We have a couple hundred server boxes where the log directories we want to consume are of an unknown name... example:

E:\Tomcat-PVSP80BA\logs\ 
E:\Tomcat-PVSP80BB\logs\ 
E:\Tomcat-PVSP80BC\logs\ 
E:\Tomcat-PVSP80BD\logs\ 

These folders are different on every server, the only thing that's the same is:

E:\Tomcat-PVSP*\logs\ 

We point a general monitor at the logs folder and ingest every log that's there under the sourcetype "tomcat-appl". This is something like a dozen unique log formats. Because we don't necessarily know what logs will be there and whether new or unexpected log names may appear (but want them all to have unique sourcetypes), we have a props/transform set up on our heavy forwarder that basically catches any logs being ingested from this folder and changes the sourcetype to something based on the filename... example:

E:\Tomcat-PVSP80BD\logs\server_frontend.log is monitored with a sourcetype of "tomcat-appl" but gets transformed to a sourcetype of "server_frontend" by a props/transform on the heavy forwarder.

E:\Tomcat-PVSP80BD\logs\catalina-2016-04-06.log is monitored with a sourcetype of "tomcat-appl" but gets transformed to a sourcetype of "catalina" that strips the date information... we do this with a few other log file name formats as well, we convert them into something simpler and more uniform.

This process is working fine, except we noticed that if you transform the sourcetype that it doesn't use any configuration set for that sourcetype... what I mean is that I for example have a configuration stanza set up for [server_frontend] in props.conf on the heavy forwarder but those settings aren't being used. I am guessing because the sourcetype was set through a transform that it doesn't retroactively go back and check for configurations for the newly set sourcetype. If I (as a test) defined the input itself as a sourcetype of "server_frontend" it will use the settings fine... but I'm not able to define the sourcetype at the input level and have to figure them out and set them at parse/index time.

I have been trying to get around this by defining a source stanza in the props.conf file (because I have enough information to know what the file names are likely to be or what format they will be in) but have not had much luck getting any kind of wild card to work in order to configure my incoming data... examples of things I've tried:

[source::E:\Tomcat-PVSP.*?\logs\server_frontend.log] 
[source::E:\Tomcat-PVSP.*?\\logs\\server_frontend.log] 
[source::E:\Tomcat-PVSP*\logs\server_frontend.log] 
[source::E:\Tomcat-PVSP.*?\logs\server_frontend.*?/.*] 
[source::E:\Tomcat-PVSP.*?\\logs\\server_frontend.*?/.*] 
[source::E:\Tomcat-PVSP.*?\logs\server_frontend.*?.*] 
[source::E:\Tomcat-PVSP.*\logs\server_frontend.log] 
[source::E:\Tomcat-PVSP.*\logs\server_frontend*.log] 
[source::E:\Tomcat-PVSP.*\logs\server_frontend*.*] 

And several others in addition to that but none of them work... I do know that the source stanza would work as setting it to a specific source applies the correct configuration to the data... example:

[source::E:\Tomcat-PVSP80BA\logs\server_frontend.log] 

But obviously without the wildcard, I'm only applying those settings to one specific log from a specific server.

My question is:
What is the proper format to target server_frontend.log files in the wildcarded folders I'm monitoring when using a source stanza in props.conf?
Another question might be is, is there a way for my transformed sourcetype not to ignore the sourcetype configuration that has been set in the props.conf file?

[somesoni2]

Give this a try

HF props.conf

[source::E:\\Tomcat-\PVSP*\\logs\\server_frontend.log] 
...your event processing attributes
TRANSFORM-sourcetype = set_sourcetype