Getting Data In
Highlighted

Two different sourcetypes in the same folder

Communicator

Hello,

I am trying to pick up to files in specific directories under different sourectypes.

[monitor:///app/ems-store-uat/uat/.../config/queues.conf]
sourcetype = ems_queues
disabled = false

[monitor:///app/ems-store-uat/uat/.../config/topics.conf]
sourcetype = ems_topics
disabled = false

The files exist in multiple paths such as /app/ems-store-uat/uat/U1_LN_DERIV_TEST/config/queues.conf & /app/ems-store-uat-uat/U1_LN_DERIV_TEST/config/topics.conf.

I want them under separate sourcetypes, because I want to group them by different type of config, but it seems that the first one is blocking the second one - the topics.conf get blacklisted, perhaps by the first?

04-19-2010 10:43:09.212 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_STAGING_DESFOCASH/config/topics.conf to ignore list.
04-19-2010 10:43:09.492 DEBUG TailingProcessor - Ignoring non-whitelisted file: /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf
04-19-2010 10:43:09.492 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf to ignore list.

Is there a way that I can do this?

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Champion

The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.

In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:

inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$

props.conf:
[source::.../topics.conf]
sourcetype=ems_topics

[source::.../queues.conf]
sourcetype=ems_queues

View solution in original post

Highlighted

Re: Two different sourcetypes in the same folder

Splunk Employee
Splunk Employee

should also whitelist (?:topics.conf|queues.conf)$ if there might be other files in the directory you don't want.

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Communicator

Thanks I will try this in the meantime. See comment above for current version.

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Champion

Yes - good point GK. I've updated my example now. Thanks.

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Communicator

Thankyou all for your comments, I will upgrade and implement this in the meantime.

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Splunk Employee
Splunk Employee

Please let us know the version of your forwarder/monitor, as there were significant changes made as of 4.1.

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Communicator

The forwarder is currently on version: Splunk 4.0.7 (build 72459). Should I upgrade to 4.1 to fix the issues?

0 Karma
Highlighted

Re: Two different sourcetypes in the same folder

Splunk Employee
Splunk Employee

4.1 will work the way you have configured above, but 4.0 and below will require tina_p's method below to work reliably.

Highlighted

Re: Two different sourcetypes in the same folder

Communicator
  • I have upgraded and can confirm that this is working. Thanks for your help!
0 Karma