Splunk report not pulling exact count for event for last 7 days.

[ashikuma]

I have report scheduled to run at 12 AM EST to search for last 7 days and just provide stats count for user ID's like below:

my search | stats count by date_wday -it will display week day field and then count of events corresponding to that. But for last one day it's giving exact count and and for other 6 days it seems to be average values not exact count.

one more thing each count can be between 10000 - 20000 , is this limitation in Splunk that we can have it for one day only and after that it will be taking average for other days.

[woodcock]

We need to know for sure what your Time picker values are. If you are running with something like earlieset=-8d@d and latest=@d-1 then you will be eliminating today and the data should always match. The problem is that you are looking at today and today is continuously adding data so every time you run the search, there is more data to count and the number goes up. Compacting this problem is the likelihood that you are mistimestamping some of your events into the past or the future so that events that were generated today and that were indexed today actually get thrown into the future and are not searchable until tomorrow turns into today. Download the meta woot! app and you will see that this is almost certainly part of your problem.

[woodcock]

How can we hope to help if you do not give us your exact search?

[ashikuma]

Hi Woodcock,

What details you required, I have mentioned everything above. please let me know what is required to solve this.

[Richfez]

@woodcock one of the searches that goes awry is

index=myindexname sourcetype=sourctypename "stringtobesearched" | dedup uid | stats count by date_wday

Which seems too simple to have a problem like this. In the other comment thread, we tried the only two things I could think of to make this better:

index=myindexname sourcetype=sourctypename "stringtobesearched" 
| eval my_weekday = strftime(_time, "%A")
| dedup uid | stats count by my_weekday

But that made no difference. Also I have specifically asked if the results when running this might include a note like "results may be incomplete because yo one peer seems busted" 🙂

@ashikuma Have you considered simply putting in a Splunk Support ticket? This might be beyond the reasonable help we can give here in a free forum, unless someone else thinks of a different direction to take.

[Richfez]

Can you provide sample data and results for this issue?

[ashikuma]

index=myindexname sourcetype=sourctypename "stringtobesearched" | dedup uid | stats count by date_wday

I searched this for last 7 days (see below) , it will pick userID from extracted field and dedup and then count on the basis of week days. This is my result when I run manually.
from Nov 15 through Nov 22, 2018

date_wday count
friday 213
monday 3698
saturday 538
sunday 599
thursday 11010
tuesday 5321
wednesday 7587

And this is the result when I schedule report to be run at 12 AM EST 23rd NOV.

date_wday,count
friday,195
monday,3639
saturday,509
sunday,1733
thursday,11010
tuesday,5173
wednesday,7587

See the difference between values, for last one day it's same and after that it's different.

[Richfez]

Interesting! I've never seen that except in two cases. Maybe one of these is it.

1) Time snapping ... no, NVM, this can't be it.

2) Do you have a warning telling you that "search results are incomplete" and warnings that "peer X isn't responding?" I hope not, but that could do it.

Although I DID think of something else while writing this. Try this instead:

index=myindexname sourcetype=sourctypename "stringtobesearched" 
| my_weekday = strftime(_time, "%A")
| dedup uid | stats count by my_weekday

The built in date fields have some issues which I won't enumerate here, but which makes them less reliable than one would hope. So, build your own, like I did above!

Give that a try and see if it's not better. Also, make sure you are snapping to days, like

index=myindexname sourcetype=sourctypename "stringtobesearched"  earliest=-7d@d latest=@d
| my_weekday = strftime(_time, "%A")
| dedup uid | stats count by my_weekday

Then you'll always get the same results each day.

[ashikuma]

I tried the same but still the same issue, it's giving exact count for last one day only and for other days it's giving average account again. Is there any other setting which we have to look into.

I am using Splunk 7.1.2.

[Richfez]

No, but I thought of something else - maybe this is a rendering issue. Like, the PDF generation routines are messing up?

1) Try running the search, noting the results, then exporting that as a PDF and seeing if it's different.

2) check Activity-> Jobs. Is that job in there? Can you rerun it? Can you see the job results? (There's like 13 different ways to click on parts of those jobs - the job itself by name, or "export" it, etc... see if ALL of those that have results show the same results, or at least all of them that have search results.

It's about the only thing left, because there's zero reason for it to do what it's doing otherwise.

[ashikuma]

I checked in activity>jobs and then export it to pdf , results are similar as I got earlier (as mentioned in last comment) , also when I am running the query manually as you suggested me earlier it's not giving the result that I am expecting .

[ashikuma]

Once I am using my_weekday = strftime(_time, "%A") in my query then it gives me below error:

Search Factory: Unknown search command 'my'.

so my_weekday is kind of function\variable that you are using just for reference or this is Splunk inbuilt command.

[Richfez]

Sorry! Forgot an eval in there.

index=myindexname sourcetype=sourctypename "stringtobesearched" 
| eval my_weekday = strftime(_time, "%A")
| dedup uid | stats count by my_weekday

That should do it. (Also fix the second one to be like that if you try it too)

[darrenfuller]

"It seems to be average values not exact count"...

In what way?

No, there's no limitation on stats count that would stop you at 20000 results.. When i run

earliest=-2d index=_internal 
| stats count by date_wday

I get

date_wday   count
----------------------
friday     225419
monday     124300
saturday     363460
sunday     334821
thursday     89546
tuesday   111742
wednesday   166883

[ashikuma]

If I am running manually then it is giving me exact count but when I am scheduling this as report to run at 12 AM EST everyday then it's giving exact count for last one day only and for other 6 days it's giving average count