What are some of the recommended steps for general daily optimization/maintenance on splunk?

[quahfamili]

Hi all,

I had been using splunk for a period of time. However, I notice that the performance started to degrade as more indexes are added.

Do anyone have any recommended script or steps that i can do daily to improved performance.

I had search through this forum, there are many recommendation but mostly are specific to an issue. I am asking for some sort of general maintenance for splunk.

Thanks in advance.

[renjith_nair · Accepted solution]

Hi @quahfamili ,
You might not find a single click solution to optimize the entire infrastructure because the performance issues might be of different reasons. If the increase in indexes is the main reason of performance degradation, you should re-look at the resource capacity. In a more general way,

• Start with "Monitoring Console" (http://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/DMCoverview) if you haven't looked into it. It gives you almost all information you need for finding the bottleneck
• Identify the phase where you have issues - https://docs.splunk.com/Documentation/Splunk/7.1.1/Indexer/Pipelinesets
• Optimization starts from the data ingestion phase and continue until the search is executed. So define your data inputs in such a way that the indexing is proper and hence the data extraction.
• Fine tune your data retention configurations according to your organization's requirements rather than depending on the default configuration. This will help you to reduce the amount of indexed data/storage.
• Use the inbuilt performance improvements splunk is providing, for e.g.parallelization - http://docs.splunk.com/Documentation/Splunk/7.1.1/Capacity/Parallelization
• Periodically re-look at your infrastructure capacity and upgrade the resources if necessary - http://docs.splunk.com/Documentation/Splunk/7.1.1/Capacity/Referencehardware
• Monitor the user's activity - especially the long running searches and see if you could optimize them
• Consider the suggestions from splunk https://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/OptimizeSplunkforpeakperformance http://docs.splunk.com/Documentation/Splunk/7.1.1/Troubleshooting/Troubleshootindexingperformance

Once you could identify the area of improvement and if it's recurring, you could automate