I am in the process of setting up a Universal Forwarder that will be running on EC2. I am looking for information on hardware recommendations for on the forwarder. There is great information on the Splunk servers themselves, but nothing on the forwarders. Is there a place in the documentation that I missed?
If not, if anyone has any rules of thumb for a Universal Forwarder, that would be great.
I should have described my setup a little better, as don't believe that I can use a test index. My setup is a number of servers sending to a single forwarder which forwards to storm. I am doing tests on the servers that will roughly simulate a load base (which would get multiplied by number of users).
From what I understand, the test index requires splunk instances that forward to the UF which gets forwarded to the target instance.
My current plan is to write a test application that sends a large number of messages to the forwarder, but I don't know how to measure if the forwarder is overloaded. Would the fishbucket work for that?
This has some best practices and recommendations when planning your first deployment.
The UF itself is designed to use as little resource as possible and is rate limited to 256kbps (by default, can be changed) when first installed.
Your best shout is to install and get started and come back with any specific issues. If its a first setup its always a great idea to test against a test index on the indexer, saves having to clear everything and start again if you go wrong.
and when you have finished your test the idea is that you can delete the test index on the indexer, set the UF to forward to your production index and reset the UF so it sends everything again.
The UF uses something called a fishbucket to record what files it has forwarded, you can clear this to re-send everything with (in the splunk/bin directory) ./splunk clean all
If you have set a password on the UF this will also reset that, I don't believe it resets the configs but it never hurts to take a backup first. Remember to only use that on the UF (Universal forwarder)