topic Re: create a dashboard that tracks 1 or 2 log feeds in Dashboards & Visualizations

How to create a dashboard that tracks 1 or 2 log feeds?

ngwodo — Tue, 24 May 2022 20:22:43 GMT

you are to create a dashboard that tracks log feeds

so I imagine it would look like a table and have things like

log feed | last seen and it would be colored based on some threshhold (last seen 24 hours red, last seen 10 mins green). It will include:

color for categorizing critical levels
email alerting
can start with small features

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Thu, 17 Sep 2020 18:58:25 GMT

There as always with splunk a number of different ways to do this. For example, you could have a panel for each log feed with a "<single>" element in each. The data in the single could be driven by the number of days since the latest entry in the log. The number displayed could be coloured depending on its value giving you the various levels of threshold you want.

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Thu, 17 Sep 2020 20:03:10 GMT

Please how do you do the coloring? Do you do it by editing XML?

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Thu, 17 Sep 2020 20:04:49 GMT

Please what would be the Splunk query to achieve it?

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Thu, 17 Sep 2020 20:18:34 GMT

Yes, edit the XML Add a couple of options to the single

<option name="rangeValues">[0,10,20,30]</option> <option name="rangeColors">["0x00FF00","0xFFFF00","0x0000FF","0xFF0000"]</option>

You choose appropriate values and colours

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Thu, 17 Sep 2020 21:40:20 GMT

Thanks. What about the splunk queries to monitor these logs in 24 hours and also 10 minutes.

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Thu, 17 Sep 2020 23:20:59 GMT

| eval secondsago=now()-_time

So you can set your ranges to be 10*60 (for 10 minutes) and 24*60*60 (for 24 hours) etc.

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Fri, 18 Sep 2020 01:37:40 GMT

Please do I have to include something like this before the eval command?

index=main source=windowseventlog sourcetype=access_combined_wcookie | eval secondsago=now()-_time

Please how would to include the splunk time range you specified in your last command in the querries?

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Fri, 18 Sep 2020 06:54:03 GMT

The first part is your search. This retrieves (matching) event records from your index. The eval will create an additional field for each event for you to use. It would probably be better to do it this way

index=main source=windowseventlog sourcetype=access_combined_wcookie | head 1 | eval minutesago=round((now()-_time)/60,0)

Since you are interested in when the latest event occurred. Splunk should have put a timestamp on each record in the _time field. This eval calculates the number of minutes ago that the event was based on this timestamp.

In your single, either by editing the XML or by modifying the format of the single in the dashboard editor, your set the range for the different colours you want e.g 0, 10, 30,, 60 etc. for 10 minute, 30 minute, 60 minute thresholds

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Fri, 18 Sep 2020 18:00:16 GMT

Thanks. what splunk queries would you add to the previous query you wrote to specify different colors for 24 hours and then for 10 mins?

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Fri, 18 Sep 2020 21:17:38 GMT

The query is OK - you configure the colours by editing the dashboard and formatting the single

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Sun, 20 Sep 2020 02:21:22 GMT

I got warning message when I added those couple of lines to the XML. Why do I get the warning error message when I edit XML for the colors?

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Sun, 20 Sep 2020 05:37:21 GMT

Thanks. I wrote this query:

| tstats latest(_time) as latest where index=* earliest=-24h by host
| eval recent = if(latest > relative_time(now(),"-5m"),1,0), realLatest = strftime(latest,"%c")
| where recent=0

Question: Do you think this query will answer my original question of:

Writing a query that will be used to create a dashboard tracking 1 or 2 log feeds that would be colored based on some threshhold (last seen 24 hours red, last seen 10 mins green) ? Please add to the query if there is anything missing. I added the lines for colors in the XML edit for the dashboard but it is giving me validation error message. Your help would be appreciated.

Re: create a dashboard that tracks 1 or 2 log feeds

inventsekar — Sun, 20 Sep 2020 05:50:45 GMT

step 1 - when you run this query on search, do you get the logs/events you wanted?

step 2 - if yes, then, create a dashboard with this query and then you can plan about the coloring.

step 3 - if no, then, lets troubleshoot this query until you find out your expected logs.

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Sun, 20 Sep 2020 09:27:16 GMT

No I don't think it does what I think you want. What if you the latest entries in the index is over 24 hours? You are only going to get entries by host if the latest entry for that host is between 24 hours and 5 minutes ago.

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Sun, 20 Sep 2020 09:57:38 GMT

If you want to use tstats try:

| tstats latest(_time) as latest where index=* earliest=-48h by host | eval minutesago=round((now()-latest)/60,0)

Then set the colour ranges for the minutesago column as appropriate

Obviously, this still only goes back 48 hours so if your latest entry is older than that, you would not see it

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Mon, 21 Sep 2020 01:36:44 GMT

Thanks. I configured the single value 42 for colors but the green color is not showing. How do I get the green color to be showing for 24 hours threshold and how do I get the red color to show for the 10 minutes threshold? Please assist with queries or visualization configurations.

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Mon, 21 Sep 2020 01:46:32 GMT

Please why do you have earliest=-48h instead of 24h for the threshold? We only have 2 threshold to deal with. The threshold is 24 hours for red for the first log feed and last 10 minutes for green for the second log feed. So we are looking at 2 different splunk queries to accomplish this threshold. Please assist.

Re: create a dashboard that tracks 1 or 2 log feeds

ITWhisperer — Mon, 21 Sep 2020 07:06:08 GMT

If you limit your query to -24h, then you will get no results if the last time the log was written to is more than 24 hour ago. The -48h was at least giving you a chance at finding if the log was last written to between 48 and 24 hours ago and showing that in red. You could make it -7d or whatever timespan you want to go back looking for when the log was last written to.

Re: create a dashboard that tracks 1 or 2 log feeds

ngwodo — Mon, 21 Sep 2020 12:49:37 GMT

Thanks. Is the set color from your previous reply only for 24 hours threshold or for both 24 hours and 10 minutes? Please let me know. I set the color as you stated but did not see the color change. Please advise.