Why does the timechart command display inconsistent results when the time range is changed?

[fvegdom]

When I use the following search (some criteria obfuscated for security):

index=main sourcetype=transaction application=foo component=bar  customerCode=x Type=y messageType=z | timechart span=1d count as count

and I set the time range to a single day (9th of jan 2017), the resulting table shows a single result:

_time                         count
2017-01-09T00:00:00.000+0100    1

as I would expect, this is consistent with the event I find when I omit the timechart command entirely, but when I change the time range to the whole month of jan 2017, I get this (just showing first 14 results)

_time                         count
2017-01-01T00:00:00.000+0100    0
2017-01-02T00:00:00.000+0100    0
2017-01-03T00:00:00.000+0100    0
2017-01-04T00:00:00.000+0100    0
2017-01-05T00:00:00.000+0100    3
2017-01-06T00:00:00.000+0100    0
2017-01-07T00:00:00.000+0100    0
2017-01-08T00:00:00.000+0100    0
2017-01-09T00:00:00.000+0100    0
2017-01-10T00:00:00.000+0100    0
2017-01-11T00:00:00.000+0100    0
2017-01-12T00:00:00.000+0100    15
2017-01-13T00:00:00.000+0100    25

suddenly there is nothing counted on the 9th, how can this happen?

in fact, if I click the cell that says zero in the table that gets returned, and select get view events, it takes me to the event that I would expect to be counted here:

2017-01-09T13:36:56.109+0100 TRAN b53a13ca-e1bc-4e64-964c-09c4714ba40e custom-operations process-engine 127.0.1.1 type:y|customerCode:x|duration:1327|bytesAllocated:15692632|executorUtilPct:0.0|messageType:z

some data here is modified for security reasons, but that should not affect the anwswers

[fvegdom · Accepted solution]

It turns out the problem is not because of any issues with timechart but due to the fact that the base search is not returning the missing event in the first place. I will create a different issue to answer why that is happening.

[niketn]

First of all, I see no point of renaming count as count, unless you want it to be uppercase. like count as Count.
Can you also verify the results with stats command instead of timechart

<Your Base Search>
| bin _time span=1d
| stats count as Count by date_mday

[fvegdom]

Hi niketnilay,

thanks for you reply.
There is indeed no purpose in it, this was just a relic of something else I did earlier(it was dc(uid) as count)

when I run the search you suggest I get
date_mday Count
12 15
13 25
16 25
19 7
20 13
24 4
5 3
so similar to what I got as a result of what somesoni2 suggested

index=main sourcetype=transaction
application=foo component=bar

customerCode=x Type=y messageType=z |
bucket span=1d _time | stats count by
_time

except this is not ordered by time.

in my last comment I found however that the timechart command itself is not the problem. The initial search seems to miss this transaction entirely when it is run with a larger time range:

when I run the search index=main
sourcetype=transaction application=foo
component=bar customerCode=x Type=y
messageType=z for only the 9th, I get
the event above

but when I run it from 8th to 10th, I
get no results !?

so I should focus on why the search is not returning this event in the first place and not on any problems with timechart. Should I make a different question for that?

[DalJeanis]

I think it would be a good idea. People reading the title of this one will focus on "why does timechart", and your real problem is "search time range returns inconsistent results".

[DalJeanis]

Please post the entire search that got you the last results.

[fvegdom]

It's the same search, but with a different timerange

index=main sourcetype=transaction application=foo component=bar customerCode=x Type=y messageType=z | timechart span=1d count as count

[DalJeanis]

Shot in the dark, but simplify "count as count" to "count" -- and verify that there is no existing field called "count" on the events -- and see what happens.

I've noticed that splunk has occasional trouble distinguishing between the count it's doing at any given time and the count that is a field already on an event record.

[fvegdom]

thanks for your suggestion but it makes no difference

index=main sourcetype=transaction application=foo component=bar
customerCode=x Type=y messageType=z | timechart span=1d count

gives the same result,

I'm positive that there is no count field, and
the same query with dc(uid) also gives no result (uid is the b53a13ca-e1bc-4e64-964c-09c4714ba40e code in the event above)

I also tried it with putting a | fields command before the timechart to rule out what you suggest:

index=main sourcetype=transaction application=foo component=bar
customerCode=x Type=y messageType=z | fields uid | timechart span=1d count

and that makes no difference either

[DalJeanis]

Hmmm. I'm seeing nothing at all. There's only one handle left to pull on. Try it without the timespan parameter, or with different timespan parameters - 8h or something.

[fvegdom]

my replies keep dissapearing...

I tried both your suggestions, but I discovered that the problem is not with the timechart but before it,

when I run the search index=main sourcetype=transaction application=foo component=bar
customerCode=x Type=y messageType=z for only the 9th, I get the event above

but when I run it from 8th to 10th, I get no results !?

[somesoni2]

How big is your main index? if you just run this for say jan8 to jan10, how many events that you get for each day?

index=main sourcetype=transaction | timechart span=1d count

[fvegdom]

@aaraneta, good to know, thanks
@somsoni2 just woke up and ran it for the month and this is what I get

_time count
2017-01-01T00:00:00.000+0100 436892
2017-01-02T00:00:00.000+0100 633636
2017-01-03T00:00:00.000+0100 700639
2017-01-04T00:00:00.000+0100 691269
2017-01-05T00:00:00.000+0100 754214
2017-01-06T00:00:00.000+0100 708054
2017-01-07T00:00:00.000+0100 537937
2017-01-08T00:00:00.000+0100 528364
2017-01-09T00:00:00.000+0100 726485
2017-01-10T00:00:00.000+0100 807973
2017-01-11T00:00:00.000+0100 790816
2017-01-12T00:00:00.000+0100 795250
2017-01-13T00:00:00.000+0100 775745

so 100ks of events per day

[somesoni2]

Depending upon the hardware configuration of your Splunk instance, this can be very large and/or indexers may have low memory dropping events. Just to confirm that large number of result is a problem and not with the data bucket that stores the data for Jan 09, can you run your original search with 24 hr time range from 2017-01-08T14:00:00+0100 to 2017-01-09T14:00:00+0100, so that we know there is a matching events and time range is one day for which we know it works fine. After that try with 2017-01-08T00:00:00+0100 to 2017-01-09T14:00:00+0100 (about 1.5 days)

[fvegdom]

hi @somesoni2

same search with time range 2017-01-08T14:00:00+0100 to 2017-01-09T14:00:00+0100
returns the event

time range
2017-01-08T00:00:00+0100 to 2017-01-09T14:00:00+0100 does too

If I leave the start of the range at 2017-01-08T00:00:00+0100 I can go up to 2017-01-10T11:00:00+010 and get the same results

if I set it to 2017-01-10T13:00:00+010 I get 40 results and then If I put it to
2017-01-10T14:00:00+010 I get no results again

[aaraneta_splunk]

@fvegdom - Regarding your comment about why your replies were disappearing: Since you a new user to Answers, your comments were sent to the moderation queue before being published; I just published one of them above.

Typically, users should see a banner up top that shows that their post is being moderated. I apologize if it did not display as it is unfortunately it's a bug on our current platform that will hopefully be fixed soon.

[somesoni2]

Do you get results for jan 9 when you run this??

index=main sourcetype=transaction application=foo component=bar 
customerCode=x Type=y messageType=z | bucket span=1d _time | stats count by _time

[fvegdom]

tried that and I get this

_time                         count
2017-01-05T00:00:00.000+0100    3
2017-01-12T00:00:00.000+0100    15
2017-01-13T00:00:00.000+0100    25
2017-01-16T00:00:00.000+0100    25
2017-01-19T00:00:00.000+0100    7
2017-01-20T00:00:00.000+0100    13
2017-01-24T00:00:00.000+0100    4

so basically no, no results for jan 9,
this is what I would get if I ran the original timechart command with cont=false (verified)

[DalJeanis]

Hmmm. How is the different time range being entered? standard search or dashboard?

[fvegdom]

Standard search, (I discovered it when I had a weird result in a dashboard, but I opened it in search)