Two different sourcetypes in the same folder

[Hazel]

Hello,

I am trying to pick up to files in specific directories under different sourectypes.

[monitor:///app/ems-store-uat/uat/.../config/queues.conf]
sourcetype = ems_queues
disabled = false

[monitor:///app/ems-store-uat/uat/.../config/topics.conf]
sourcetype = ems_topics
disabled = false

The files exist in multiple paths such as /app/ems-store-uat/uat/U1_LN_DERIV_TEST/config/queues.conf & /app/ems-store-uat-uat/U1_LN_DERIV_TEST/config/topics.conf.

I want them under separate sourcetypes, because I want to group them by different type of config, but it seems that the first one is blocking the second one - the topics.conf get blacklisted, perhaps by the first?

04-19-2010 10:43:09.212 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_STAGING_DESFOCASH/config/topics.conf to ignore list.
04-19-2010 10:43:09.492 DEBUG TailingProcessor - Ignoring non-whitelisted file: /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf
04-19-2010 10:43:09.492 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf to ignore list.

Is there a way that I can do this?

[tpsplunk]

are you sure multiple sourcetypes in inputs.conf should work as expected in 4.1? I'm trying something very similar in 4.1.6 and it doesn't seem to work.

looking through the guides I found this statement: "Note: Monitor input stanzas may not overlap. That is, monitoring /a/path while also monitoring /a/path/subdir will produce unreliable results. Similarly, monitor input stanzas that watch the same directory with different whitelists, blacklists, and wildcard components are not supported."

from here: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories

which seems to imply that you can't define multiple sourcetypes in inputs.conf.

[Hazel]

• I have upgraded and can confirm that this is working. Thanks for your help!

[gkanapathy]

4.1 will work the way you have configured above, but 4.0 and below will require tina_p's method below to work reliably.

[Hazel]

The forwarder is currently on version: Splunk 4.0.7 (build 72459). Should I upgrade to 4.1 to fix the issues?

[gkanapathy]

Please let us know the version of your forwarder/monitor, as there were significant changes made as of 4.1.

[the_wolverine · Accepted solution]

The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.

In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:

inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$

props.conf:
[source::.../topics.conf]
sourcetype=ems_topics

[source::.../queues.conf]
sourcetype=ems_queues

[Hazel]

Thankyou all for your comments, I will upgrade and implement this in the meantime.

[the_wolverine]

Yes - good point GK. I've updated my example now. Thanks.

[Hazel]

Thanks I will try this in the meantime. See comment above for current version.

[gkanapathy]

should also whitelist (?:topics.conf|queues.conf)$ if there might be other files in the directory you don't want.