Alerting

Help with Custom Search Please

vwilson3
Path Finder

I would really appreciate any help as I am not very experienced with SPL. I am learning every day, literally.

I need to find a way to show the host name and the tag::host the event occurred on that tripped the alert, the date/time, Alert Manager (AM) incident_id, alert name, Splunk/AM user, AM action, AM comment.

Here is what I have so far... there may be better ways to do this... and this doesn't show the host and host tag for the system where the event occurred. I have tried several different things, but had no success getting the host names.

sourcetype=alert_metadata
| join incident_id 
    [search sourcetype=incident_change]
| table _time incident_id alert user action comment

Any suggestions? Any assistance is greatly appreciated.

0 Karma

vwilson3
Path Finder

In my haste to post my question, I forgot to say that the events that tripped the alert are in a different index and are different sourcetype. I was thinking I could try using the time field as a common field to try to find the host, but this particular alert is scheduled to run every hour, so the time it shows in AM is not the time of the actual event that tripped the alert. Still stumped.

0 Karma

nickhills
Ultra Champion

Are you able to share some of the event data for both sourcetypes?
If at all possible you want to avoid 'join', and seeing the format of the data may help identify a more efficient approach.

If my comment helps, please give it a thumbs up!
0 Karma

vwilson3
Path Finder

I had to make it as generic as possible. I hope it is still helpful.

sourcetype alert_metadata:
{"alert": "myalertname", "alert_time": "2019-03-19T17:15:11.892+00:00", "app": "search", "entry": [{"content": {"earliestTime": "2019-03-19T16:15:00.000+00:00", "eventCount": 1, "eventSearch": "search (myalertsearchstring)", "keywords": "index::myappindex myapp_msgarea::RRR myapp_msgname::SSS", "latestTime": "2019-03-19T17:15:00.000+00:00" remote_mysplunkserver_scheduler_wilsonvsearchRMD54ae8a1459b8e343f_at_1553015700_10843. "optimizedSearch": "| search (mysearchstring", "resultCount": 1, "searchEarliestTime": 1553012100, "searchLatestTime": 1553015700, "searchProviders": ["mySplunkservers]}, "links": {"alternate": "/services/search/jobs/schedulerwilsonvsearchRMD54ae8a1459b8e343f_at_1553015700_10843"}, "name": "search mysearchstring}], "impact": "high", "incident_id": "23464663-4af1-4073-bf84-6edebfb6c837", "job_id": "schedulerwilsonvsearch_RMD54ae8a1459b8e343f_at_1553015700_10843", "name": "myalertname", "owner": "unassigned", "priority": "critical", "result_id": "0", "title": "myalertname", "ttl": 86400, "urgency": "high"}

sourcetype incident_change:
time=2019-03-19T17:15:12.245662 event_id="21794f2b4d134d36359eca1115a478b7" severity=INFO origin="alert_handler" user="splunk-system-user" action="create" alert="myalertname" incident_id="23464663-4af1-4073-bf84-6edebfb6c837" job_id="scheduler_wilsonvsearch_RMD54ae8a1459b8e343f_at_1553015700_10843" result_id="0" owner="unassigned" status="new" urgency="high" ttl="86400" alert_time="2019-03-19T17:15:11.892+00:00"

example of the actual event that tripped the alert:
2019-03-19 16:44:43 myapp_logversion="2" myapp_msgarea="AU" myapp_msgname="L" myapp_pid="37301" myapp_taskno="00031" myapp_slgttyp="CC" myapp_termname="myterm" myapp_username="userhere"

Splunk fields: host, index, sourcetype, source, tag::host

I'm trying to figure out how to get the host name where the event occurred, based on the info in Alert Manager (the top two examples).

0 Karma

shaileshmali
Path Finder

try

sourcetype=alert_metadata | fields incident_id ( select fields in main search incident_id date/time, Alert Manager (AM) incident_id, alert name, Splunk/AM user, AM action, AM comment)
| join incident_id
[search sourcetype=incident_change | table incident_id ( select additional fields in subsearch ) ]
| table _time incident_id alert user action comment

0 Karma

vwilson3
Path Finder

Thank you for the quick response. I will keep this info for a different report.

In my haste to post my question, I forgot to say that the events that tripped the alert are in a different index and are different sourcetype. I was thinking I could try using the time field as a common field to try to find the host, but this particular alert is scheduled to run every hour, so the time it shows in AM is not the time of the actual event that tripped the alert. Still stumped.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.