using splunk for debug or trace logs

[aviadr1]

I have a complex system which sometimes needs to be debugged or troubleshooted by using verbose trace logs. the challenges in my scenario are:

1. the verbose logs grow fast, in the order to 5Mb - 20mb per minute.
2. the machines producing the logs are also cpu constrained, as they run cpu intensive tasks
3. the machines producing the logs are located at a remote client location (we have many clients), these are far(bandwidth is limited) from the location where we host the main splunk indexer

what would be a good topology in this case?

[Genti · Accepted solution]

This can be one way of dealing with this; i must admit, you seem quite limited in resources but:

• i dont think 1 would be an issue if 2 and 3 were not around. (i would guess that these logs will only grow so large at very rare occasions and not constantly and all time)
• From 2, it seems like you should run LightWeightForwarders which have the lowest footprint and CPU usage
• From 3, it seems like you are being bandwith limited and cant really send the data to the main indexer which is in a different location.(that would be the ideal option)

If it is possible to have an extra box dedicated to splunk in the remote location, then you can install splunk as an indexer (at this remote location), collect the remote logs into this indexer and then allow the main splunk indexer to search the remote one, (basically adding search peers).

[aviadr1]

I'm thinking splunk isn't built to handle debug level logging