topic Re: Events count changing on every search in Splunk Search

Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 02:17:21 GMT

HI,
I am facing a weird situation. I am executing a query that will give last one day data on hourly basis.

base search 
| search type="data" 
 | timechart span=1h count(eval(HTTP_Status="202")) as today_count

when i execute this 1st time i got some httpd requests count.
When i execute above command 2nd time, i am seeing around 100-200 difference in the count for every hour.
Why happening this?
any help?
Thank you!

Re: Events count changing on every search

cpetterborg — Tue, 12 Jun 2018 02:52:26 GMT

Are you seeing any yellow triangles on the results page? Have you used the job inspector to see if there are any problems? Are you using the same time period, like yesterday, or are you using something like last 24 hours?

Re: Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 03:13:41 GMT

No. I am not seeing any yellow triangles. I am using same command with
earliest=-24h@h latest=now.
at 1st run :
19th hour --> 52168
20th hour ---> 121115

at 2nd run :
19th hour : 52153
20th hour : 121082

Like above event count is decreasing for every search.

Re: Events count changing on every search

niketn — Tue, 29 Sep 2020 19:53:34 GMT

@prathapkcsc, are you using pot-processing? Also is there is a reason for type="data" to be outside the base search? If you are just showing HTTP_Status=202, can you also add HTTP_Status="202" to your base search?

Following will give you today's 202 events with span=1h (assuming by last one day you mean today as you have renamed the count also as today_count).

<yourBaseSearch> type="data" HTTP_Status=202 earliest=-0d@d latest=now
| timechart span=1h count as today_count

Re: Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 06:13:22 GMT

data is just like grep pattern from log data.
sourcetype="mylogdata" earliest=-5h@h latest=now HTTP_Status=202
| search type="data"
| timechart span=1h count
I am just giving the above query. some how values are changing on every search. At least am able to see 10-20 count difference.
I have no idea about pot-processing.

Re: Events count changing on every search

niketn — Tue, 12 Jun 2018 06:26:57 GMT

@prathapkcsc well if you are not using post processing there is nothing to investigate there. Can you try the following search and see the result counts?

sourcetype="mylogdata" earliest=-6h@h latest=-1h@h HTTP_Status=202 type="data"
| timechart span=1h count

If the counts are still not matching, can you verify the Time on Raw Data and Time picked by _time field whether they match or not?

Re: Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 06:28:25 GMT

I used the same command. Still am able to see the difference on every search.

Re: Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 06:34:50 GMT

Raw data has EDT time zone and _time field showing in UTC timezone.
Is this causing the issue?

Re: Events count changing on every search

niketn — Tue, 29 Sep 2020 19:53:36 GMT

Try the following

 sourcetype="mylogdata" earliest=-1d@h-5 latest=-1d@h-3 HTTP_Status=202 type="data"
 | timechart span=1h count

Also check (in verbose mode) whether your raw data has date_hour field extracted.

I am thinking for the following change might happen:
1) Latest hour bucket as data is still flowing in.
2) Your _time not picking up time correctly.

Because of which I which I wanted you to test the time which has already passed and for which already events have been indexed.

If date_hour is present you can also try stats count by date_hour to see whether counts add up or not.

Re: Events count changing on every search

prathapkcsc — Tue, 12 Jun 2018 08:59:27 GMT

Hi I used the below query
sourcetype="mylogdata" type="data" earliest=-1d@h-5 latest=-1d@h-3 HTTP_Status="202"
|timechart span=1h count

The below output i got
_time count

2018-06-11 07:00 34

Just fyi, i have extracted fileds. What did i take over from the above query?
I am observing event count change for almost all the hours.
Also Please let me know the meaning of the below line
earliest=-1d@h-5 latest=-1d@h-3

Re: Events count changing on every search

DalJeanis — Tue, 12 Jun 2018 23:14:17 GMT

Try something like this

 base search 
 | search type="data"
 | bin _time span=1h 
 | where HTTP_Status="202"
 | stats count as today_count by _time

I suspect that timechart is moving the boundaries of the span, although that should not happen with earliest=-24h@h

The above method should give you exactly the same number of records for an hour each time it is run across the same hour. If it does not, then you have (A) something going on with your data ingestion, or (B) something going on with records getting moved out of warm to cold or (C) indexers going offline for some reason.

Re: Events count changing on every search

prathapkcsc — Wed, 13 Jun 2018 01:25:18 GMT

Thanks for your reply..Let me try it out.will get you back with the result.

Re: Events count changing on every search

prathapkcsc — Wed, 13 Jun 2018 02:09:06 GMT

Still am getting event count change on every run. Don't know the exact problem. I am observing this kind of change first time. our data is a real time log which will generate from mobile application.

Re: Events count changing on every search

prathapkcsc — Thu, 14 Jun 2018 06:27:24 GMT

Still am getting event count change on every run. Don't know the exact problem. I am observing this kind of change first time. our data is a real time log which will generate from mobile application.
May be different time zones causing this issue. Thanks.

Re: Events count changing on every search

cpetterborg — Thu, 14 Jun 2018 11:39:47 GMT

Look at the difference between _time and the indextime. If you are east of UTC, that does seem like the most likely scenario. Especially if you are in UTC+2 or so, and the data count is only changing in the last two hours.

See the following answer for information on seeing the difference between the timestamp and the index time:

https://answers.splunk.com/answers/540344/how-to-compute-indextime-time-difference-average-w.html