Getting Data In

Two different sourcetypes in the same folder

Communicator

Hello,

I am trying to pick up to files in specific directories under different sourectypes.

[monitor:///app/ems-store-uat/uat/.../config/queues.conf]
sourcetype = ems_queues
disabled = false

[monitor:///app/ems-store-uat/uat/.../config/topics.conf]
sourcetype = ems_topics
disabled = false

The files exist in multiple paths such as /app/ems-store-uat/uat/U1_LN_DERIV_TEST/config/queues.conf & /app/ems-store-uat-uat/U1_LN_DERIV_TEST/config/topics.conf.

I want them under separate sourcetypes, because I want to group them by different type of config, but it seems that the first one is blocking the second one - the topics.conf get blacklisted, perhaps by the first?

04-19-2010 10:43:09.212 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_STAGING_DESFOCASH/config/topics.conf to ignore list.
04-19-2010 10:43:09.492 DEBUG TailingProcessor - Ignoring non-whitelisted file: /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf
04-19-2010 10:43:09.492 INFO  TailingProcessor - Adding /app/ems-store-uat/uat/U1_LN_DERIV_AIRLOCK/config/topics.conf to ignore list.

Is there a way that I can do this?

0 Karma
1 Solution

Champion

The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.

In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:

inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$

props.conf:
[source::.../topics.conf]
sourcetype=ems_topics

[source::.../queues.conf]
sourcetype=ems_queues

View solution in original post

Communicator

are you sure multiple sourcetypes in inputs.conf should work as expected in 4.1? I'm trying something very similar in 4.1.6 and it doesn't seem to work.

looking through the guides I found this statement: "Note: Monitor input stanzas may not overlap. That is, monitoring /a/path while also monitoring /a/path/subdir will produce unreliable results. Similarly, monitor input stanzas that watch the same directory with different whitelists, blacklists, and wildcard components are not supported."

from here: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories

which seems to imply that you can't define multiple sourcetypes in inputs.conf.

0 Karma

Communicator
  • I have upgraded and can confirm that this is working. Thanks for your help!
0 Karma

Splunk Employee
Splunk Employee

4.1 will work the way you have configured above, but 4.0 and below will require tina_p's method below to work reliably.

Communicator

The forwarder is currently on version: Splunk 4.0.7 (build 72459). Should I upgrade to 4.1 to fix the issues?

0 Karma

Splunk Employee
Splunk Employee

Please let us know the version of your forwarder/monitor, as there were significant changes made as of 4.1.

0 Karma

Champion

The behavior you're describing sounds like a bug. You've specified a whitelist by naming the log file in your monitor input. Please file a support ticket.

In the meantime, you should be able to use a single monitor input in conjunction with props.conf to get this to work:

inputs.conf:
[monitor:///app/ems-store-uat/uat/.../config]
_whitelist = (topics\.conf|queues\.conf)$

props.conf:
[source::.../topics.conf]
sourcetype=ems_topics

[source::.../queues.conf]
sourcetype=ems_queues

View solution in original post

Communicator

Thankyou all for your comments, I will upgrade and implement this in the meantime.

0 Karma

Champion

Yes - good point GK. I've updated my example now. Thanks.

0 Karma

Communicator

Thanks I will try this in the meantime. See comment above for current version.

0 Karma

Splunk Employee
Splunk Employee

should also whitelist (?:topics.conf|queues.conf)$ if there might be other files in the directory you don't want.

0 Karma