The use case am working on:
Here is a historic search which could do this easily using append for the past 15 min which I am trying to convert to realtime search.
index=i1 sourcetype=s1 uri_path="api1" | eval uri_path=replace(uri_path, "\w{8}-\w{4}-\w{4}-\w{4}-\w{12}", "{id}") | eval SLA=1000 | stats count as count, count(eval(responsetime>SLA)) as violations,first(SLA) as SLA by uri_path | append [ search index=i1 sourcetype=s1 uri_path="api2" | eval uri_path=replace(uri_path, "\w{8}-\w{4}-\w{4}-\w{4}-\w{12}", "{id}") | eval SLA=750 | stats count as count, count(eval(responsetime>SLA)) as violations,first(SLA) as SLA by uri_path ]
But I need to do this in realtime. Append doesnt work for this. I also believe there could be some issues with using multiple real time queries for real time searches like I used to do for historic searches which am not fully sure.
Here is one which I tried.
index=i1 sourcetype=s1 uri_path="api1" OR uri_path="api2" | eval uri_path=replace(uri_path, "\w{8}-\w{4}-\w{4}-\w{4}-\w{12}", "{id}") | eval SLA=750 | stats count as count, count(eval(responsetime>SLA)) as violations,first(SLA) as SLA by uri_path
The issue is SLA value is not the same for all API's. Its different for each API.
Perhaps there is limitations with using multiple queries in real time. Perhaps a single query should do this when converting the above use case from historic to real time search. Since append doesnt work, am not sure if map, join etc can work also because they involve 2 queries in conjunction.
I need help with this use case for real time searches in terms of writing splunk query for it.
In case there are limitations for this use case, an alternative way is to use scheduled historic search every minute. It should also be that the query should run fast and finish quickly within the minute and query performance acceleration could be a consideration.
Any suggestions for this as well?
Any help will be appreciated.
The best approach (one which I am using as well) will be to create a lookup table file for the SLA value and then reference it in your query using lookup command.
Lookup table: api_sla.csv
Lookup Fields: api_name, sla
Updated (sample query, assuming api name=uri_path, if not use the field which contains api name in the lookup command):
index=i1 sourcetype=s1 uri_path="api1" OR uri_path="api2" | eval uri_path=replace(uri_path, "w{8}-w{4}-w{4}-w{4}-w{12}", "{id}") | lookup api_sla.csv api_name as uri_path OUTPUT sla as SLA | stats count as count, count(eval(responsetime>SLA)) as violations,first(SLA) as SLA by uri_path
The best approach (one which I am using as well) will be to create a lookup table file for the SLA value and then reference it in your query using lookup command.
Lookup table: api_sla.csv
Lookup Fields: api_name, sla
Updated (sample query, assuming api name=uri_path, if not use the field which contains api name in the lookup command):
index=i1 sourcetype=s1 uri_path="api1" OR uri_path="api2" | eval uri_path=replace(uri_path, "w{8}-w{4}-w{4}-w{4}-w{12}", "{id}") | lookup api_sla.csv api_name as uri_path OUTPUT sla as SLA | stats count as count, count(eval(responsetime>SLA)) as violations,first(SLA) as SLA by uri_path
http://docs.splunk.com/Documentation/Splunk/6.0.2/SearchReference/Streamstats
It looks like streamstats is more appropriate from streaming perspective. How to explain this to unfamiliar audience. What about the choice of using stats over streamstats. It looks like streamstats is more appropriate for moving average of last 5 events in the entire collection of events in the real time window whereas stats works on the entire collection of events in the real time window.Are we loosing anything by not using streamstats when real time streaming is used?
Great. please accept the answer if there are no followup questions.
Thanks. Works as expected !