Damn, thats slick, thank you much! Crazy it takes that kind of work. I was going under the assumption Splunk would treat the lookup as a pseudo index by default since its a csv. ... View more

Ahh, clever. I see now. Basically you are making sure that the key you want the value for is first in the list, then reset its displayed KEY. This solved the issue in an unexpected way. If I wanted to set 2 tokens instead of just one, would this be possible? Like setting what you want with a 1.KEY, 2.KEY, etc. The problem I see then is assigning it to a token. since that is $result.VALUE$ it would be cool to do $result.1.VALUE$ $result.2.VALUE$ etc ... View more

I can get the xml to work, the issue I am having is that I cant reference the transposed values. so activity_id is set to the literal string $result.ActivityId$ because a field ActivityId doesn't exist after the transpose ... View more

I initially had them hard coded, and now am using a search inline when the dashboard is invoked. It is a simple top 0|table X |sort by X to populate a dropdown with a list of dependent systems. ... View more

Sorry I have been swamped. I am going to try this now and see if I can make things work. Thanks! ... View more

I figured this out. in order for these variables to be used from the resultset I had to declare these variables in the alert_actions.conf, then restart. I then could add the information by calling the variable value with setting.get from the action script. alert_actions.conf param.results_team = $result.team$ action.py team = settings.get('results_team') ... View more

these are all defined in the alert. here is the stanza. For whatever reason, the results_file="$results.file$" results_link="$results.link$" portions work fine, but the results_Percent="$result.Percent$" part never gets a value even though the email action easily pics this value up. [TEST AWS SPLUNK EMF ALERT PROD] action.email = 1 action.email.include.results_link = 0 action.email.include.trigger = 1 action.email.include.trigger_time = 1 action.email.include.view_link = 0 action.email.inline = 1 action.email.message.alert = The alert condition for '$name$' was triggered at $result.Percent$ %. action.email.priority = 5 action.email.sendresults = 1 action.email.to = [email protected] action.email.useNSSubject = 1 action.emf = 1 action.emf.command = sendalert $action_name$ results_file="$results.file$" results_link="$results.link$" results_Percent="$result.Percent$" action.emf.param.auth_token_override = xxx action.emf.param.severity = 2 alert.severity = 5 alert.suppress = 1 alert.suppress.period = 5m alert.track = 1 alert_condition = search Percent> .5 auto_summarize.dispatch.earliest_time = -1d@h counttype = custom cron_schedule = * * * * * dispatch.earliest_time = -15m@m dispatch.latest_time = -m@m enableSched = 1 request.ui_dispatch_app = app request.ui_dispatch_view = search schedule_window = 3 search = *base search*|timechart partial=f span=5m avg(pct) as Percent ... View more

basically, we are just creating a json payload and parsing it in the simple script below, the other info comes in fine, but the extradetails where I am testing this, always comes back empty def post_event(payload): settings = payload.get('configuration') api_endpoint = settings.get('api_endpoint').rstrip('/') auth_token = settings.get('auth_token_override') if not auth_token: auth_token = settings.get('auth_token') #FIXME: ideally, this should not be the splunk_server that triggers the alert, # but the host where the data are generated from. host = payload.get('server_host') severity = settings.get('severity') search_name = payload.get('search_name') result_link = payload.get('results_link') result_file = payload.get('results_file'), results_Percent = payload.get('results_Percent') data = { 'source': 'Splunk', 'host': host, 'eventtype': search_name, 'severity': severity, 'summary': search_name, 'extradetails': results_Percent, 'searchpath': result_file } ... View more

Thanks for you very detailed solution, I will definitely be able to implement this in places, however the panels I am using are displaying trend data over time, so using the stats, while it would work (I tried that earlier), it removes the trendline from the panel. Another reason I wanted to use the range color, is that the dashboard will be showing many similar panels for endpoints, however the range for panels all will be different but assigned the same green/yellow/red colors. So basically the checkboxes driving this filter would be 1)show healthy endpoints 2)show endpoints in warning 3)show endpoints in critical ... View more

It sounds like you are understanding my issue. I used the "progress" example, as it was taken from a working condition when it was keying off of the result count condition. I want to instead be able to toggle panels on and off based on the rangeColor applied to the result. ... View more

I was still attempting to use the rolling time range within the alert, rather than scheduling all the alert searches every minute or 5. Iam changing back to your suggestion and vetting. I am still interested in the original issue though, of being able to use splunk's rolling alert on a calculated field, which appears to be the issue. ... View more

It appears this does not work as I would think. removing the timespan from the alert triggers way to aggressively. I get a hit for 33% errors using an alert rolling window searching for pct>33, however timecharting the same alert query for 5 minute buckets, doesn't return any 5 minute segment above 6. ... View more

Thats what I was thinking as well. I also wanted to port these alert status to a dashboard and was running into issues with too many results to display ui warnings, but it became obvious, the alerts can run in the rolling window taking account any indexing latency, and the dashboards can run with a set schedule. It wont be "exactly" the same, but should be close. Ill give it a trial after my meeting this morning and see how it goes, Thanks! ... View more

So far I have arrived at ditching the bin _time part, and converting the stats command to a straight timechart with fixedrange=false and span=1m for rolling time window of 15m with latest -2m. The latest = rt-2m was initially being used to try and not trigger on incomplete time buckets, but may be able to be removed. Still looking for any pointers on how anyone else was working with this. Triggering on error counts is easy, however when you have traffic patterns that record error counts per time period in a sine wave, you need to use pct for this. ... View more

Ha, this is exactly what I was thinking of doing as well, but the syntax was throwing me. This did in fact work, thank you! ... View more