Setting up SAI is a multistep process. Start here https://docs.splunk.com/Documentation/InfraApp/2.1.0/Install/About
Read the installation instructions https://docs.splunk.com/Documentation/InfraApp/2.1.0/Install/DistributedDeployment
1. Install the Splunk App for Infrastructure on search heads
2. Install the Splunk Add-on for Infrastructure on indexers. This sets everything up for inputs there.When you install the add-on, it creates the em_metrics and infra_alerts indexes, and handles props and transforms for all data types.
3. Configure inputs.conf for the indexing tier. Do not forget HEC - When you configure an HEC token, set the source type to em_metrics, and specify the metrics index you want to use. By default, the metrics index is em_metrics. For more information about configuring an HEC token, see Create an Event Collector token in the Getting Data In guide.
4. Push the indexer cluster master node's configuration bundle to the indexer cluster (if you use indexer clusters)
5. Configure data collection using the App for Infrastructures "add data" , select os and customize. It will generate a custom installation script to deploy the config (along with the UF if there is not one)
6. Deploy the custom script file created in step 5 to your endpoints like you might do other software. I used PowerShell to deploy to a list of windows system and bash for Linux. You can easily deploy the configuration files without using the "add data" wizard. It will not redeploy the UF if there is one installed.
Problems/Solutions
I could not get the App to work on the search head without adding the Splunk Add-on for Infrastructure to the search head. I think I had one system showing up until I added the TA on the search head.
Had one Windows system that would not report into the dashboard. It ended up being a time-sync issue. The system clock on the Windows Server was not setup for NTP.
... View more