Multiple sourcetypes in the same directory

jheilman — Tue, 24 May 2011 16:07:32 GMT

I know this question has been asked numerous times before, because I've read most of the questions and answers. I still can't seem to get it right, no matter what I try. We have several Windows servers running JBoss. The folder structure is similar to the following...

D:\jboss\server\<ContainerName>\log\
    access.2011-05-24.log
    app.log
    boot.log
    stderr.log
    stdout.log

So, the goal is to pull access.YYYY-MM-DD.log as sourcetype=access_common and everything else as sourcetype=log4j. Ideally, I'd like to be able to create a JBoss server class and push a generic configuration out to all of our JBoss servers to pull the logs.

I've tried several different things, but nothing seems to work as expected. I've tried using simple regular expressions in the [monitor] stanzas as suggested in one answer and I've tried a very general [monitor:] stanza pointing at the directory with accompanying [source::] stanzas to filter the file names and specify sourcetypes in props.conf. I've tried more than that, but those two seemed to be the most promising. I've used https://servername:8089/services/admin/inputstatus/TailingProcessor:FileStatus to verify the files are being read, but they don't seem to be getting indexed, or they don't have the expected sourcetype if they are.

I know things have changed from version to version in Splunk and maybe the problem is that I'm trying things that don't work anymore. Can someone set me straight?

My current configuration is as follows...

inputs.conf:

#
# JBoss - Common Log Files
#
[monitor://D:\jboss\server\*\log\*.log]
index = fod-web

props.conf:

[source::...\\access\.\d{4}-\d{2}-\d{2}\.log$]
sourcetype = access_common

[source::...\\(?!access)[\w-_.]+\.log$]
sourcetype = log4j

Re: Multiple sourcetypes in the same directory

jbsplunk — Tue, 24 May 2011 17:00:50 GMT

Could you paste the configuration you are using to try and do your sourcetyping? Also, you may want to review the following, there are some pretty good example configurations:

http://blogs.splunk.com/2010/02/11/sourcetypes-gone-wild/

I think you should be using:

[source::…\access.\d{4}-\d{2}-\d{2}.log$]

and

[source::...\(?!access)[\w+.log$]

Re: Multiple sourcetypes in the same directory

jheilman — Wed, 25 May 2011 14:19:05 GMT

I updated the original question with the configuration I'm using currently.

topic Multiple sourcetypes in the same directory in Getting Data In

Multiple sourcetypes in the same directory

Re: Multiple sourcetypes in the same directory

Re: Multiple sourcetypes in the same directory