I'm trying to figure out the sizing of a Splunk environment that will only be used for a very short time but by a substantial amount of users (20-40 workshop participants). All these users will be running searches simultaneously against the same index.
The idea of the workshop is to build a dashboard to visualize some previously indexed data. In a way it's very similar to the official Splunk4Rookies workshop, just with different data.
My concern is that the user experience will be terrible due to too many searches being attempted at same time. This raises the following questions:
- how many concurrent searches need to be possible to support 30 users simultaneously building dashboards?
- how far can I increase the maxsearchesper_cpu parameter in limits.conf? what are the downsides?
- assuming all data resides in one index (and all searches being run on this index) is this enough or should one index be replicated by implementing indexer clustering? how many searchable copies would be necessary?
I'm hoping to be able to use an all-in-one Splunk instance (so no indexer clustering) but I have no means to realistically test the search performance/experience with 20-40 simultaneous users before the actual workshop.
Has anyone have any experience with such a setup or does anyone know how Splunk does this for their Splunk4Rookies workshops?
I believe Splunk do this with an ec2 instance per user.
Its a good way to make sure all users get the same (decent) experience, and you don't have to fiddle with limits or worry too much about optimising.
At a cost of somewhere between $0.20 - $0.40 per hour (depending on the spec you choose) its pretty cost effective per 'seat'
If your indexing requirements are modest you can possibly make do with the free licence, or if needs be point your ec2 instances at your existing licence master.
Deploy the first one, add your data and customise it as required - maybe some scripts to regenerate sample data every hour etc.
Create an AMI of it, and deploy it to your 20-40 temporary instances for "fresh" classroom in about 5 mins.
at max $16 an hour (for 40), its probably way less cost/effort than spending days on optimisation and the architecture.
You can have each participant go to splunk.com and click the green "Free Splunk" button in the upper right. They will need to create a free splunk account, but once thats done, they can select cloud trial. It will spin up an cloud instance for them, good for 15 days and 5gb/day. As part of your workshop, have them upload and index the demo data that they will use.