Getting Data In

How do I configure inputs.conf to define sourcetypes for multiple types of log files in the same folder?

sramiz
Path Finder

Hello,

I have a question about indexing multiple types of logs file in same folder. How would go about defining sourcetypes correctly to these files using inputs.conf

for an example:
../local/application/logs/ contains
server.log (sourcetype=log4j)
audit.log (sourcetype=log4j)
_problems.log (sourcetype=log4j)
errors.log (sourcetype=log4j)
YYYY_MM_DD.request.log (sourcetype=access_combined)

Thanks
Ramiz

Tags (2)
0 Karma
1 Solution

fdi01
Motivator

You have 2 options: use blacklist and whitelist configurations in your inputs.conf file (that is what I would do) OR, have splunk monitor a different directly and run a cron job to create links in that directory that point back to the files in the original directory but only for the files that you would like to forward.

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/inputsconf

ex:

 [monitor:///../local/application/logs/*.log  ]
   disabled = false
   index = your_index_name
   sourcetype = log4j
   blacklist = /../local/application/logs/request.log

 [monitor:///../local/application/logs/request.log ]
   disabled = false
   index = your_index_name
   sourcetype = access_combined

View solution in original post

fdi01
Motivator

You have 2 options: use blacklist and whitelist configurations in your inputs.conf file (that is what I would do) OR, have splunk monitor a different directly and run a cron job to create links in that directory that point back to the files in the original directory but only for the files that you would like to forward.

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/inputsconf

ex:

 [monitor:///../local/application/logs/*.log  ]
   disabled = false
   index = your_index_name
   sourcetype = log4j
   blacklist = /../local/application/logs/request.log

 [monitor:///../local/application/logs/request.log ]
   disabled = false
   index = your_index_name
   sourcetype = access_combined

lguinn2
Legend

This works too, especially the example that you have given. Just be careful that you don't create identical monitor stanzas! For example, this may cause problems:

[monitor:///server/default/logs/]
sourcetype = log4j
whitelist=.*problems\.log.*

[monitor:///server/default/logs/]
sourcetype = access_combined
whitelist=.*request\.log.*

sramiz
Path Finder

worked... thanks a lot Iguinn and fdi01... I got it working with the combination of your suggested solution.... thanks again..

cheers
SR

lguinn2
Legend

The most efficient way to do this is to define a separate stanza in inputs.conf for each of the 5 inputs. For example
inputs.conf

[monitor://../local/application/logs/server.log]
sourcetype=log4j

[monitor://../local/application/logs/audit.log]
sourcetype=log4j

etc...

But that technique is not helpful if you want to capture all the files in the logs folder - and the files might change over time. In that case, I would use a combination of inputs.conf and props.conf - both of these files are used during the input phase. Put both config files on the same machine as the ../local/application/logs/ directory (probably on a forwarder).

inputs.conf

[monitor://../local/application/logs/]

props.conf

[source::../local/application/logs/(server|audit|_problems|errors|request).log]
sourcetype=log4j

[source::../local/application/logs/request.log]
sourcetype=access_combined

Of course, if new files are added to the directory, they will be indexed, but their sourcetype will be automatically determined by Splunk. You would have to add the new files to props.conf to override the default sourcetypes.

BTW, be sure that your path is correct in these configuration files, the .. makes me uncomfortable. It should really be an absolute path.

0 Karma

sramiz
Path Finder

thanks for you suggestion.... it worked partially as i can see some files in spunk but not all of the them...

my inputs.conf looks like

[monitor:///server/default/logs/_problems_.log*]
disabled = false
sourcetype = log4j
index = test

[monitor:///server/default/logs/_full_.log*]
disabled = false
sourcetype = log4j
index = test

[monitor:///server/default/logs/schema.log*]
disabled = false
sourcetype = log4j
index = test

[monitor:///server/default/logs/errors.log*]
disabled = false
sourcetype = log4j
index = test

also what if files have date added to their names... an example

-rw-rw-r-- 1 app1 grp1    30154 Jun 15 20:10 _problems_.log.2015-06-15
-rw-rw-r-- 1 app1 grp1   665785 Jun 15 20:10 _full_.log.2015-06-15
-rw-rw-r-- 1 app1 grp1        0 Jun 16 00:00 caf.log.2015-06-16
-rw-rw-r-- 1 app1 grp1    24335 Jun 16 22:45 _problems_.log.2015-06-16
-rw-rw-r-- 1 app1 grp1   622448 Jun 16 22:45 _full_.log.2015-06-16
-rw-rw-r-- 1 app1 grp1        0 Jun 17 00:00 caf.log.2015-06-17
-rw-rw-r-- 1 app1 grp1   682876 Jun 17 00:13 2015_06_16.request.log
-rw-rw-r-- 1 app1 grp1  5511792 Jun 17 17:02 2015_06_17.request.log
-rw-rw-r-- 1 app1 grp1    26474 Jun 17 22:20 _problems_.log.2015-06-17
-rw-rw-r-- 1 app1 grp1   870503 Jun 17 22:20 _full_.log.2015-06-17
-rw-rw-r-- 1 app1 grp1        0 Jun 18 00:00 caf.log.2015-06-18
0 Karma

lguinn2
Legend
caf.log.2015-06-16         - does not match any stanza
 _problems_.log.2015-06-16 - matches /server/default/logs/_problems_.log*
_full_.log.2015-06-16      - matches /server/default/logs/_full_.log*
2015_06_16.request.log     - does not match any stanza

If the date precedes the "request.log", then you need to update the stanza so that the * wildcard appears at the beginning of the file name. You have no stanza that matches the "caf.log*".

The wildcard will pick up the files regardless of what characters appear, whether dates or otherwise

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...