Getting Data In

We are thinking of using directory recursion to collect logs from various servers, but what's the overhead on the system?

gopal20
New Member

In a large enterprise with thousands of IIS, apache and tomcat servers, with each server having multiple web or app instances and with each instance having different log file locations, any suggestions on an efficient way to collect these logs?

We are thinking of using directory recursion, but what’s the overhead on the system?

0 Karma

thomrs
Communicator

I gather lots of logs from our syslog server recursively and no issues. The host_regex in the inputs.conf helps keeps all the server names in order. If you use log rotation to move logs make sure you have a good whitelist/blacklist strategy.

We also had to up the bandwidth the UF uses to keep up with the amount of data in limits.conf

http://docs.splunk.com/Documentation/Splunk/6.2.2/admin/inputsconf

http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Limitsconf

TO keep an eye on things take a look at the SOS and Deployment Monitor apps.

https://apps.splunk.com/app/748/

https://apps.splunk.com/app/1294/

0 Karma

gopal20
New Member

Using directory recursion generated 30% increase in CPU usage by Splunkd. Not a good option for our environment.

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...