Why aren't all values shown in dashboard?

[neilmac64]

I have a problem where not all values are showing up in a chart - and the values that do show up are rather flatlined. For example, here is the data I gathered for this chart:

However, none of the earlier values show up in the chart.

I have remade the index and the data is good coming in from the CSV files.

Can anyone help me identify what's wrong?

Many thanks.

[neilmac64 · Accepted solution]

Fixed it!

I added the span option to the search and it broke it off into 15 minute slices.

 index=index
| where key="tx_nss_2" AND mac_address="aa:bb:cc:00:11:22"
| timechart span=15m values(value) by key

Kudos to both Rick and Rich who helped me understand the problem.

Thank you both!

NM

[PickleRick]

Just remember that if you happen to have multiple values within your - now shortened - span, you'll hit the issue again.

[neilmac64]

Thanks Rick. 

That shouldn't be the case, but I'll definitely look out for it in the future.

[neilmac64]

I've done some more testing today - this image shows it quite well. This is for a single device, though the problem exists for all. Both charts are of the same search - you can see the table returns values (for 11 Dec) that are not shown on the linear chart.

As pointed out by PickleRick, could it be the time scale? Data comes in every 15 minutes, but the chart seems to show the values only on the hour.

[PickleRick]

It's not the timescale. It's that you have multivalued fields in your results and they do not get graphed.

Run-anywhere example that will give you a nice ascending line graph:

| makeresults count=10 
| streamstats count
| eval _time=_time+count

But if you add

| eval count=mvappend(count,count)

at the end of that search, it will make the count field a multivalued field with two identical values in each row. And that will not show on graph since the visualization expects a single value, not a multivalued field.

[neilmac64]

Can you explain how it's a multivalued field? It shouldn't be - there is only one value per field in the csv file.

Splunk seems to be turning the data from 4 separate files (created every 15 minutes) into a multivalued field of 1 hour with 4 values. 

The events clearly show the timestamp is at 15 minute increments. 

[PickleRick]

I don't know. I don't know your events or your search. But I do know what the results in the opening post looked like.

You clearly have rows of results with multivalued fields.

[neilmac64]

Yeah - I think you have correctly identified why the values are not being graphed correctly.

So it seems the issue is how the data is being displayed.

The data is created every 15 minutes and each key has a single value. When I search the events I can see them timestamped correctly and they are indeed every 15 minutes.

When I run a search, Splunk appears to aggregate the data into on hour chunks and shows all the values of data gathered within the one hour period. Sometimes only a single value was collected, so these values are charted. However, if more than one value was collected within the hour, the search appears to show it as a multivalued field, which you correctly point out is then not charrted.

So the question now it to work out why Splunk is doing this, and to fix it.

Any ideas?

[PickleRick]

Most probably (I'm guessing here because you haven't given us the search that is run) you have some form of

| timechart span=1h values(tx_nss2)

Or you're binning first then doing

| stats values(tx_nss2) by _time

Maybe you don't need aggregation by whole hours and it's enough to simply do

| table _time tx_nss2

Or maybe you need shorter period of aggregation. Hard to say since I don't know your requirements.

If you need to rework your search, please create a new topic in the Splunk Search forum

[PickleRick]

I'd have to check it but I don't think the linear graph can visualize multivalued fields. It expects a single number per drawable row (or multiple separate numerical fields if you want to draw separate series).

[neilmac64]

Hi Rich.

Thanks for the reply. I have two things going on - 

If you look to the left of the chart, you will see no data for the start, even though I do have data indexed. It doesn't show in the chart even if I set the time period specifically to the period I want to see - the graph is empty. This data wasn't shown in my image of the table.

The second issue is that the graph is flat even though I do have differences in data. The data is incremental, so should show an increasing line, but seems to be flat. This is the data that is shown in the image of the table - you can see the values do go up, but the line is flat.

I've tried logarithmic scale, it doesn't help.

 I have to remote to the splunk server to get data so won't have a chance now till tomorrow. If you let me know what you need I will get it all and post back.

Many thanks again,

NM

[richgalloway]

The screenshot shows values that are high and fairly close together so a relatively flat graph should be expected.  Consider using the logarithmic scale on the Y-axis to spread things out more.

The screenshot also shows data for a single day so it's unclear what you mean by "none of the earlier values show up".

Please share the query.