I discovered the mistake yesterday and you also nailed it on the head... It's an issue with my search query. I did not include TestCall="*" in my search. So when alert does not see TestCall because its not in the search. I thought that by defining it as an extracted field was enough, but I gotta put it in my search query. Now my other issue is when I put TestCall="*", a few things happen.. The unique identifier "GUID" is attached to the request and response. This allows me to pipe it into TRANSACTION and group them together and measure the response time for each web service call. So there were 10 events which were grouped together that had a response time longer than 5 seconds today Query 1: index=uvtrans | transaction GUID startswith="*request" endswith="*response" | WHERE duration>5 This returns back 10 events which had response times greater than 5 seconds. Query 2: index=uvtrans TestCall="*" | transaction GUID startswith="*request" endswith="*response" This returns back results but the duration is not longer than 5 seconds like the original search. Query 3: index=uvtrans TestCall"*" | transaction GUID startswith="*request" endswith="*response" | WHERE duration>5 Now introducing the WHERE duration>5 returns back 0 results ... View more

Thanks for the suggestion, I applied to my alert. Any ideas of how to get this token working so I can see which web service is running slow in the email subject line? ... View more

I went ahead and did another field extraction to replace Call5 to make sure that wasn't the issue. So I have a new field extraction called 'TestCall'. I did a search for all web service calls that were greater than 5 seconds then did a field extraction for all the request names (TestCall). I only had about 10 calls greater than 5 seconds and my TestCall extraction picked up all the request names as expected. I then changed my alert to 'Splunk Alert: $result.TestCall$ Resp > 5 Sec' So now I have to wait for a request that takes longer than 5 seconds to see if this works.. I also included my savedsearches.conf entry above. Please advise on what else I should try if this doesn't work ... View more

Here's my entry from savedsearches.conf. Thanks for the help so far [Response Time > 5 Sec] action.email = 1 action.email.include.trigger = 1 action.email.reportServerEnabled = 0 action.email.subject.alert = Splunk Alert: $result.Call5$ Resp > 5 Sec action.email.to = DOTCOM_PERFORMANCE_MONITORING_ALERTS@xxxxxxx.com action.email.useNSSubject = 1 alert.digest_mode = 0 alert.suppress = 0 alert.track = 1 counttype = number of events cron_schedule = * * * * * description = This alert goes off when a web service call takes longer than 5 seconds dispatch.earliest_time = -1m dispatch.latest_time = now display.events.fields = ["host","TYPE","CLASSPATH","splunk_server","Status","City","Code","GUID","req","resp","duration","CLASS","linecount","GUID1"] display.page.search.mode = verbose display.visualizations.chartHeight = 279 display.visualizations.charting.chart = line enableSched = 1 quantity = 0 relation = greater than request.ui_dispatch_app = search request.ui_dispatch_view = search search = index=uvtrans | transaction GUID startswith="*request" endswith="*response" | WHERE duration>5 ... View more

I'm running 6.2 I'm pretty sure I tried both $result.Call5$ and $results.Call5$ with no luck. Looks like my only option left is to call an external script to reference which web service is running slow. Do you know if its possible for me to use Javascript to do so? ... View more

My extracted field is named 'Call5'. I went into my alert and put the email subject as Splunk Alert: $results.Call5$ Response > 5 Seconds Now the only thing appearing in the subject line is Splunk Alert: ... View more

Thanks for the help.. I did a field extraction in my search to pull all the calls which is called 'Call5' and that works as expected. Previously I had in the Subject 'Splunk Alert: $name$'.. I then took your suggestion and changed it to 'Splunk Alert: $result.Call5$' and nothing is showing up now. Any other suggestion? ... View more

This is exactly what I was looking for. My last question is where would I define 'GetDeliveryScheduleRequest' as a token? I have 7 other web service calls, do I have to define each of them? ... View more

This kind of worked but not exactly what I was looking for. Thanks for taking the time to help though! ... View more

I see what you did there but unfortunately did not work, I was finally able to get it and will post my solution below. Thanks for your help and time! I appreciate it ... View more

The unique identifier (GUID) is tied to BOTH the request and response. So when a SOAP request is sent, it has a timestamp and on the response it also has a timestamp. The Transaction command groups these together so I have an output in the 'Statistics' tab which shows the GUID and total duration (time for each SOAP request to respond). Duration GUID 9.3387 dsf234-sdf-234-dsf-344 11.234 jhhbj-767-sdfds-sdfds 10.333 dfds3-h33-hbh3-23jjkk So I currently have this output with my first query, but now I want to make a timechart visually showing the duration. I've tried using count(duration), list(duration), and values(duration) with no luck ... View more

Also I think you may be correct with the list() function.. maybe I should be using a different function, any suggestions? ... View more

Thanks for the response! I took out there where clause and still have the same problem. My original search (the first one listed in my question) will return back exactly what I'm looking for. But when I add timechart, nothing will appear so I'm not sure if it has anything to do with WHERE. Any other suggestions? ... View more

I was able to solve it by editing the props.conf file. Thanks for inquiring though! ... View more

That doesn't work since the Status is enclosed in pipes. Any idea how I could get rid of the most frequent 200 call? ... View more