i wrote several blogposts about monitoring splunk using nagios on batchworks.de: one thing you would like to monitor is e.g. license usage on your licenseserver. a cornercase is to monitor for old ufs..
out-of-the-box there good checks are included in the Managementconsole. Those alerts should trigger your nagios.
in the end i would suggest an "end-to-end" test using a technical user and doing a search in you splunk instance - by using rest api for example. At you could be quite sure that everthing is running smooth.
those test including basic os montoring should be a good start.