Getting Data In

Altering sourcetype based on source in props.conf

mmcoltman
Explorer

Hi,

I'm trying to figure out where I'm going wrong with this. My setup consists of an indexer and several universal forwarders, all sending data to the same tcp port on the indexer. I would like to change sourcetypes for application data based on the source file it originates from, but I've read conflicting or unspecific information on whether I should alter props.conf on the indexer or each individual forwarder. I'm leaning towards the indexer because as far as I understand the sourcetype is not set until index-time and the forwarders are not processing the information in any significant way.

Regardless, I've tried both altering props.conf on both indexer and forwarders as follows:

An example source log file looks like:

/var/log/company/application1/application1.log
/var/log/company/application2/application2.log
...etc

So I've entered the following in props.conf for each application log:

[source::.../var/log/company/application#/application#.log*]
sourcetype = application#

Ive also tried without the wildcards, e.g:

[source::/var/log/company/application#/application#.log]
sourcetype = application#

But neither methods work - Splunk still uses the default sourcetypes. As far as I can tell both patterns should match a specific log file. Any ideas on where I'm going wrong or an easier method of achieving this would be appreciated!

ankireddy007
Path Finder

Hi,

Have you found the solutions to this, only with props.conf?
I am also facing similar issue.

0 Karma

mmcoltman
Explorer

So I figure I should update in case anyone else is having similar issues. As per the wiki link above I added the sourcetypes to the monitor entries in inputs.conf on each forwarder, and my sourcetypes are now being set as I want them. Moral of the story - RTFM...

0 Karma

mmcoltman
Explorer

Ok, so I've carried on trying to get this to work in $SPLUNK_BASE/etc/default/local/props.conf on the forwarder but it is still not happening. Would it be a bad idea to use transforms on the indexer to change sourcetype based on source? I know this isn't the recommended way but I feel I'm making no progress and at least I can get transforms to work. Is this likely to impact performance on the indexer considering I will probably index up to 500Mb or so per day?

0 Karma

amritogreen
Explorer

you might have another pattern matching (e.g. [source::....log]) look at ALL your patterns to make sure one isn't conflicting.

mmcoltman
Explorer

Ok, just found this very-useful-looking page, and it seems to confirm that these entries should be placed on the forwarders, which is what I didn't really want but hey:

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

I will give this another go and update if I get it working.

0 Karma

mmcoltman
Explorer

Also I had a look using "splunk cmd btool..." and I can't see any other conflicting entries in config anywhere.

0 Karma

mmcoltman
Explorer

Thanks, but I don't see anything like that. I haven't specified any other stanza's referencing any ".log" yet. Also if I replace the application log name after the last slash with a wildcard (not best practice but just to test), then it still doesn't work.

I'm confused as I do have other source-based stanza's working on the indexer which are just a word followed by a wildcard. These use transform attributes rather than sourcetype, so maybe i'm trying to change sourcetype in the wrong place. I guess I could use transforms but this seems a bit inefficient.

0 Karma

mmcoltman
Explorer

Sorry, the "#" is just a subsitute for a number in a made-up application name, ie. application1, application2 etc... In the actual file I'm using names of various application logs.

0 Karma

Ayn
Legend

What are the # characters doing there?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...