topic Re: Two different sourcetypes in the same folder in Getting Data In

Two different sourcetypes in the same folder

Hazel — Mon, 19 Apr 2010 16:51:01 GMT

Hello,

I am trying to pick up to files in specific directories under different sourectypes.

[monitor:///app/ems-store-uat/uat/.../config/queues.conf]
sourcetype = ems_queues
disabled = false

[monitor:///app/ems-store-uat/uat/.../config/topics.conf]
sourcetype = ems_topics
disabled = false

The files exist in multiple paths such as /app/ems-store-uat/uat/U1_LN_DERIV_TEST/config/queues.conf & /app/ems-store-uat-uat/U1_LN_DERIV_TEST/config/topics.conf.

I want them under separate sourcetypes, because I want to group them by different type of config, but it seems that the first one is blocking the second one - the topics.conf get blacklisted, perhaps by the first?

04-19-2010 10:43:09.212 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_STAGING_DESFOCASH/config/topics.conf to ignore list.
04-19-2010 10:43:09.492 DEBUG TailingProcessor - Ignoring non-whitelisted file: /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf
04-19-2010 10:43:09.492 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf to ignore list.

Is there a way that I can do this?

Re: Two different sourcetypes in the same folder

the_wolverine — Mon, 19 Apr 2010 18:45:49 GMT

The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.

In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:

inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$

props.conf:
[source::.../topics.conf]
sourcetype=ems_topics

[source::.../queues.conf]
sourcetype=ems_queues

Re: Two different sourcetypes in the same folder

gkanapathy — Mon, 19 Apr 2010 22:24:38 GMT

Please let us know the version of your forwarder/monitor, as there were significant changes made as of 4.1.

Re: Two different sourcetypes in the same folder

gkanapathy — Mon, 19 Apr 2010 22:25:52 GMT

should also whitelist (?:topics.conf|queues.conf)$ if there might be other files in the directory you don't want.

Re: Two different sourcetypes in the same folder

Hazel — Mon, 19 Apr 2010 22:48:00 GMT

The forwarder is currently on version: Splunk 4.0.7 (build 72459). Should I upgrade to 4.1 to fix the issues?

Re: Two different sourcetypes in the same folder

Hazel — Tue, 20 Apr 2010 00:53:24 GMT

Thanks I will try this in the meantime. See comment above for current version.

Re: Two different sourcetypes in the same folder

the_wolverine — Tue, 20 Apr 2010 09:24:56 GMT

Yes - good point GK. I've updated my example now. Thanks.

Re: Two different sourcetypes in the same folder

gkanapathy — Tue, 20 Apr 2010 09:47:18 GMT

4.1 will work the way you have configured above, but 4.0 and below will require tina_p's method below to work reliably.

Re: Two different sourcetypes in the same folder

Hazel — Tue, 20 Apr 2010 15:03:28 GMT

Thankyou all for your comments, I will upgrade and implement this in the meantime.

Re: Two different sourcetypes in the same folder

Hazel — Tue, 20 Apr 2010 17:45:14 GMT

I have upgraded and can confirm that this is working. Thanks for your help!

Re: Two different sourcetypes in the same folder

tpsplunk — Sat, 05 Mar 2011 02:04:36 GMT

are you sure multiple sourcetypes in inputs.conf should work as expected in 4.1? I'm trying something very similar in 4.1.6 and it doesn't seem to work.

looking through the guides I found this statement: "Note: Monitor input stanzas may not overlap. That is, monitoring /a/path while also monitoring /a/path/subdir will produce unreliable results. Similarly, monitor input stanzas that watch the same directory with different whitelists, blacklists, and wildcard components are not supported."

from here: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories

which seems to imply that you can't define multiple sourcetypes in inputs.conf.