Hello,
I am trying to pick up to files in specific directories under different sourectypes.
[monitor:///app/ems-store-uat/uat/.../config/queues.conf] sourcetype = ems_queues disabled = false [monitor:///app/ems-store-uat/uat/.../config/topics.conf] sourcetype = ems_topics disabled = false
The files exist in multiple paths such as /app/ems-store-uat/uat/U1_LN_DERIV_TEST/config/queues.conf & /app/ems-store-uat-uat/U1_LN_DERIV_TEST/config/topics.conf.
I want them under separate sourcetypes, because I want to group them by different type of config, but it seems that the first one is blocking the second one - the topics.conf get blacklisted, perhaps by the first?
04-19-2010 10:43:09.212 INFO TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_STAGING_DESFOCASH/config/topics.conf to ignore list. 04-19-2010 10:43:09.492 DEBUG TailingProcessor - Ignoring non-whitelisted file: /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf 04-19-2010 10:43:09.492 INFO TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf to ignore list.
Is there a way that I can do this?
The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.
In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:
inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$
props.conf:
[source::.../topics.conf]
sourcetype=ems_topics
[source::.../queues.conf]
sourcetype=ems_queues
are you sure multiple sourcetypes in inputs.conf should work as expected in 4.1? I'm trying something very similar in 4.1.6 and it doesn't seem to work.
looking through the guides I found this statement: "Note: Monitor input stanzas may not overlap. That is, monitoring /a/path while also monitoring /a/path/subdir will produce unreliable results. Similarly, monitor input stanzas that watch the same directory with different whitelists, blacklists, and wildcard components are not supported."
from here: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories
which seems to imply that you can't define multiple sourcetypes in inputs.conf.
4.1 will work the way you have configured above, but 4.0 and below will require tina_p's method below to work reliably.
The forwarder is currently on version: Splunk 4.0.7 (build 72459). Should I upgrade to 4.1 to fix the issues?
Please let us know the version of your forwarder/monitor, as there were significant changes made as of 4.1.
The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.
In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:
inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$
props.conf:
[source::.../topics.conf]
sourcetype=ems_topics
[source::.../queues.conf]
sourcetype=ems_queues
Thankyou all for your comments, I will upgrade and implement this in the meantime.
Yes - good point GK. I've updated my example now. Thanks.
Thanks I will try this in the meantime. See comment above for current version.
should also whitelist (?:topics.conf|queues.conf)$ if there might be other files in the directory you don't want.