I'm currently looking at increasing the performance of our Splunk Search Head. I'm running a number of Apps at the request of my network engineer. However I'm noticing a number of things:
Should mention that I'm currently running Splunk Indexer and Splunk Search Head (seperate servers) in Azure. Things seem descent in Azure. And am increasing the instance. But some other things I'm thinking of doing:
Before I invest in these, I'd love to get the Splunk Communities input on all of this. I admit, Splunk is becoming very App-Heavy. Which I'm not pleased about. So any ways of increasing performance is appreciated.
Aw, one last thing. I'm still fairly new to data modeling. Though I've worked with the CIM I haven't tagged everything. I'm wondering if limiting the tags to specific Data Models would be of great benefit to performance, or just harm it.
Hi!
If you want to solve your performance problems you should start by adopting the hardware requirements for Splunk. There are pages dedicated to that in docs. For Azure deployments, there is this tech brief that sheds some light on some good practices and even advice on which is the appropriate set of instances you should use depending on the size of your deployment.
Regarding CIM datamodels, you should enable acceleration only on the datamodels you are actually using and restrict the indexes each datamodel accelerates data from. The CIM app has a macro for each datamodel where you can place the specific indexes to look for tagged data. This will reduce the amount of data the acceleration searches will have to look into, lowering the run time and the chance you'll endup with skipped searches.
Regarding the "hour behind", i would make sure all your systems have configured and using the same NTP and the time zone they are set into. I've ran into customers not having NTP configured at all making it impossible to properly correlate data or having diferent time zones setup.