Getting Data In

Altering sourcetype based on source in props.conf

Explorer

Hi,

I'm trying to figure out where I'm going wrong with this. My setup consists of an indexer and several universal forwarders, all sending data to the same tcp port on the indexer. I would like to change sourcetypes for application data based on the source file it originates from, but I've read conflicting or unspecific information on whether I should alter props.conf on the indexer or each individual forwarder. I'm leaning towards the indexer because as far as I understand the sourcetype is not set until index-time and the forwarders are not processing the information in any significant way.

Regardless, I've tried both altering props.conf on both indexer and forwarders as follows:

An example source log file looks like:

/var/log/company/application1/application1.log
/var/log/company/application2/application2.log
...etc

So I've entered the following in props.conf for each application log:

[source::.../var/log/company/application#/application#.log*]
sourcetype = application#

Ive also tried without the wildcards, e.g:

[source::/var/log/company/application#/application#.log]
sourcetype = application#

But neither methods work - Splunk still uses the default sourcetypes. As far as I can tell both patterns should match a specific log file. Any ideas on where I'm going wrong or an easier method of achieving this would be appreciated!

Path Finder

Hi,

Have you found the solutions to this, only with props.conf?
I am also facing similar issue.

0 Karma

Explorer

So I figure I should update in case anyone else is having similar issues. As per the wiki link above I added the sourcetypes to the monitor entries in inputs.conf on each forwarder, and my sourcetypes are now being set as I want them. Moral of the story - RTFM...

0 Karma

Explorer

Ok, so I've carried on trying to get this to work in $SPLUNK_BASE/etc/default/local/props.conf on the forwarder but it is still not happening. Would it be a bad idea to use transforms on the indexer to change sourcetype based on source? I know this isn't the recommended way but I feel I'm making no progress and at least I can get transforms to work. Is this likely to impact performance on the indexer considering I will probably index up to 500Mb or so per day?

0 Karma

Explorer

you might have another pattern matching (e.g. [source::....log]) look at ALL your patterns to make sure one isn't conflicting.

Explorer

Ok, just found this very-useful-looking page, and it seems to confirm that these entries should be placed on the forwarders, which is what I didn't really want but hey:

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

I will give this another go and update if I get it working.

0 Karma

Explorer

Also I had a look using "splunk cmd btool..." and I can't see any other conflicting entries in config anywhere.

0 Karma

Explorer

Thanks, but I don't see anything like that. I haven't specified any other stanza's referencing any ".log" yet. Also if I replace the application log name after the last slash with a wildcard (not best practice but just to test), then it still doesn't work.

I'm confused as I do have other source-based stanza's working on the indexer which are just a word followed by a wildcard. These use transform attributes rather than sourcetype, so maybe i'm trying to change sourcetype in the wrong place. I guess I could use transforms but this seems a bit inefficient.

0 Karma

Explorer

Sorry, the "#" is just a subsitute for a number in a made-up application name, ie. application1, application2 etc... In the actual file I'm using names of various application logs.

0 Karma

Legend

What are the # characters doing there?

0 Karma