Splunk as a tool for capacity and performance management

[draganmarkov]

I am wondering if anyone has any experience or suggestions for using Splunk as a tool for Capacity and Performance management (in addition to using it as IT ops and Security tool)

Ultimately i would like to be able to report capacity and performance stats for different domains such as VM's , Network, Telephony, Storage, etc.

The way i see it right now I'll have 3 types of data sources:
1. Systems that Splunk has apps for and logs to monitor (vSphere, CISCO, etc) - this one should be straight forward

1. Systems that can be scripted to produce a daily, weekly or monthly reports (storage system, etc)- i think i should be able to monitor report directory and index the data sources such as .CSV ?

2. Systems that don't log or have ability to report capacity/performance related stat - someone will collect couple of KPI's once a month - what is the best place to store the "manual" data inputs? A CSV file that gets ingested into Splunk?

[draganmarkov]

Thanks everyone. CSV for this type of data source/input will likely be a way to go.

[ramgnisiv]

The real challenge lies in the mapping of your data to your organisational structure. If you do not have proper Configuration Management for all your CI's, you might want to consider using something like a KV store to map the data you are gathering to your organisational structure. Once in place, maintaining the CM(DB) will be one of the challenges you'll face when wanting to report on Capacity & Performance management across your organisation. Just my 2 cents.

[adonio · Accepted solution]

This is a pretty large question as the opportunities are almost endless...
Many large organizations are using Splunk for that purpose, among other use cases.

as for your questions, yes you can index CSV data or use it as a lookup, however, ther great value splunk can bring is on data that is constantly flowing in. it will allow you to create advance statistics, collect many data points for ML and usage predictions and other

Start and looking for published use cases and documents / conf presentations regarding it. there are tons out there
take a look at this one for example:
https://conf.splunk.com/files/2019/slides/FN1137.pdf