I commonly need to find patterns within relation to a certain event. For instance I want to view all error logs after a code deploy, or I have a bug that causes a server crash and I want to look at the 30 minutes worth of logs before the crash.
In other tools I've used (Graphite, and New Relic) you can send special events for a code deploy and then those will be displayed in the graphs as a vertical line.
This feature in graphite is similar to what I want: graphite.readthedocs.io/en/latest/events.html
I tried using append with two queries:
This query finds the application startup: host="server-01.internal" WFLYSRV0025
This query finds shows me everything I'm looking for host="server-01.internal" ERROR
This was what I tried in append:
host="server-01.internal" WFLYSRV0025 | append [search host="server-01.internal" ERROR]
However I can't differentiate between the regular search results and the event I want to highlight.
I've tried quite a few things and can't seem to figure out how to show the events I'm looking for?
@safetytrick, I think your use case is for Event Annotation
which is one of the new features of Splunk Enterprise 7 , Refer to the following answer and Splunk Documentation:
https://answers.splunk.com/answers/600749/chart-with-trendline-in-splunk.html
https://docs.splunk.com/Documentation/Splunk/latest/Viz/ChartEventAnnotations
@safetytrick, I think your use case is for Event Annotation
which is one of the new features of Splunk Enterprise 7 , Refer to the following answer and Splunk Documentation:
https://answers.splunk.com/answers/600749/chart-with-trendline-in-splunk.html
https://docs.splunk.com/Documentation/Splunk/latest/Viz/ChartEventAnnotations
Thank you, yes this is exactly what I need. It would be nice to do this in the search, but this works too.
Can you try
host="server-01.internal" WFLYSRV0025 | append [search host="server-01.internal" ERROR] | highlight "ERROR"
Have a look at highlight
SPL command.
https://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Highlight
let me know if this helps!
This highlights matches in the search results, what I'm hoping for is to show these special events in the timeline.
Try this then
host="server-01.internal" WFLYSRV0025 | append [search host="server-01.internal" ERROR] | eval status=if(like(_raw,”%ERROR%”,”ERROR”,”NORMAL EVENTS”) | timechart span=1h count by status.
Let me know if this helps!