Create KVStore for Server Status

[stevec]

We've got an alert set up on the Monitoring Console to let us know when a machine is down, but sometimes--such as for hardware maintenance--we know that machine is going to be taken out for some amount of time and we don't want the alert to fire during that time.

What I'd like to do is create a KVStore to track these servers so that we could run a "remove from service" script to add the server to this KVStore. Then I could modify the alert to just ignore any results that are in that lookup. (The script would also handle things like taking offline a clustered indexer.)

I don't have much (any) experience around doing this, so I'd appreciate any help. My impression is that since the alert is on the MonCon, I'd want to stick the KVStore there as well.

# SPLUNK_HOME/etc/apps/splunk_monitoring_console/local/collections.conf

[server_down]
enforceTypes = true
fields.timestamp = time
fields.servername = string
fields.note = string

I think I remember reading that you can set the timestamp to auto-fill on adding to the KVStore, but I can't remember how. Servername is the server, and the "note" field I'm planning to use to track information like why it was taken down. I figure that way if a server is added to the KVStore because there's a bad hard drive, for example, and then is added again because a group of servers that it's part of is undergoing upgrades, I can make sure to remove the upgrade item once the upgrades are done while still making sure it stays out of service while it needs hardware maintenance.

Do I necessarily need a transforms.conf entry? Or will the KVStore defined in collections.conf provide everything I need?

If I want to test what's in my KVStore, I should be able to use this, correct?

| outputlookup server_down

And if I want to test adding something to the KVStore, would this also work?

servername="myserver" note="hardware maintenance"
| inputlookup server_down

And to verify my REST queries, they should look roughly like this, correct?

# Show servers in KVStore
curl -k -u admin:yourpassword \
https://<monitoringconsole>:8089/servicesNS/nobody/splunk_monitoring_console/storage/collections/data/server_down

# Add a server
curl -k -u admin:password \
https://<monitoringconsole>:8089/servicesNS/nobody/splunk_monitoring_console/storage/collections/data/server_down/ \
-H 'Content-Type: application/json' \
    -d '{"servername": "myserver", "note": "hardware maintenance"}'

# Show all servers under hardware maintenance
curl -k -u admin:yourpassword \
https://<monitoringconsole>:8089/servicesNS/nobody/splunk_monitoring_console/storage/collections/data/server_down?note=hardware%20maintenance

# Remove a server
??? 

Not really sure on removing a server. I think I need to somehow get the key, but I'm not sure the best way to do that. With the timestamp maybe?

Thanks for any help! I've tried to include what I think I know from my reading and searching so far, but I know there's a bit more to hammer out still and this seems to be the best place to go.