I am in the process of setting up a Universal Forwarder that will be running on EC2. I am looking for information on hardware recommendations for on the forwarder. There is great information on the Splunk servers themselves, but nothing on the forwarders. Is there a place in the documentation that I missed?
If not, if anyone has any rules of thumb for a Universal Forwarder, that would be great.
Have a look at;
This has some best practices and recommendations when planning your first deployment.
The UF itself is designed to use as little resource as possible and is rate limited to 256kbps (by default, can be changed) when first installed.
Your best shout is to install and get started and come back with any specific issues. If its a first setup its always a great idea to test against a test index on the indexer, saves having to clear everything and start again if you go wrong.
If you do use a test index you can set the destination index via inputs.conf on the UF;
and when you have finished your test the idea is that you can delete the test index on the indexer, set the UF to forward to your production index and reset the UF so it sends everything again.
The UF uses something called a fishbucket to record what files it has forwarded, you can clear this to re-send everything with (in the splunk/bin directory) ./splunk clean all
If you have set a password on the UF this will also reset that, I don't believe it resets the configs but it never hurts to take a backup first. Remember to only use that on the UF (Universal forwarder)
I should have described my setup a little better, as don't believe that I can use a test index. My setup is a number of servers sending to a single forwarder which forwards to storm. I am doing tests on the servers that will roughly simulate a load base (which would get multiplied by number of users).
From what I understand, the test index requires splunk instances that forward to the UF which gets forwarded to the target instance.
My current plan is to write a test application that sends a large number of messages to the forwarder, but I don't know how to measure if the forwarder is overloaded. Would the fishbucket work for that?
Have a look at;
Splunk uses queues for indexing/forwarding and these become blocked when overloaded. To troubleshoot you can have a look at;
Finally, install the splunk SoS app! It can help measure throughput and blocked queues all in a few dashboards which would allow you to performance test / record your network setup.
By a "test index" I just mean not to use the main index when you start, send it all to a test index, delete it and start again when you are happy with your setup.