- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Search for peers with status=down
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What search can I do to find peers with status=down. Looking to form an alert when this happens but can't find it within a search.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk Monitoring console (formally known as DMC) has alert "DMC Alert - Search Peer Not Responding" which does the same thing. It basically runs following search:
| rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, status
| rename peerName as Instance, status as Status
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Anyone coming here should know that in 9.2.0.1 this does not work any more.
Look at dmc_instances_view_default_search macro for how the monitoring console does it now.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk Monitoring console (formally known as DMC) has alert "DMC Alert - Search Peer Not Responding" which does the same thing. It basically runs following search:
| rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, status
| rename peerName as Instance, status as Status
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yup, found this preexisting alert and was able to edit the Actions on it to integrate with the Slack Notifications add-on. Runs every 5 minutes to check if the server is down so this works perfectly for me. Thanks for the help!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you looking for hosts with forwarders installed that havent reported to Splunk in some time?
You can use the Monitoring Console to view that. To view the missing hosts, you can click on the Forwarders tab and then Forwarders: Deployment.
For an alert, go to the Monitoring Console -> Settings -> Alerts Setup. There is an alert named DMC Alert - Missing Forwarders.
Note: A forwarder shows a status of "missing" if it has not connected to indexers within 15 minutes
Troubleshooting the OpenTelemetry Collector
Adoption of Infrastructure Monitoring at Splunk
Modern way of developing distributed application using OTel
