topic Why is Splunk timechart not displaying data when last 30 days range is selected? in Splunk Search

Why is Splunk timechart not displaying data when last 30 days range is selected?

peterfox1992 — Wed, 23 Mar 2022 04:09:29 GMT

Hi Folks,

I'm new to Spunk and I was working on creating a dashboard for one of my Application.

Dashboard is built but when I want to populate the data for last 30 days, its giving result for only few day ( 7 to 8 days) and other days are populated as 0. When I look into that particular day, I can notice events are there.

Can someone please help here?

My Query format is as below,

Main Query [search <subquery> ] | timechart span=1d count as total | sort by "_time" desc

My Output is as below,

2022-03-22	647
2022-03-21	988
2022-03-20	279
2022-03-19	100
2022-03-18	879
2022-03-17	1169
2022-03-16	15
2022-03-15	0
2022-03-14	0
2022-03-13	0
2022-03-12	0
2022-03-11	0
2022-03-10	0
2022-03-09	0
2022-03-08	0
2022-03-07	0
2022-03-06	0
2022-03-05	0
2022-03-04	0
2022-03-03	0
2022-03-02	0
2022-03-01	0
2022-02-28	0

Before 15th March, I see data is populated as 0 but when the same query is ran for 15th March alone I noticed events are getting populated.

For eg, I selected time range as 14th March 00:00 to 15th March 24:00 for the same query, I got result as below. But this value not getting populated when last 30days time period is selected.

2022-03-15	587
2022-03-14	654

Kindly need help on this.

Thanks in Advance.

Re: Splunk timechart not displaying data when last 30 days range is selected

ITWhisperer — Tue, 22 Mar 2022 17:56:27 GMT

Perhaps your subquery is being truncated which is curtailing the results you are seeing. Does the job inspector show any warnings or other messages?

Re: Splunk timechart not displaying data when last 30 days range is selected

somesoni2 — Tue, 22 Mar 2022 19:52:57 GMT

Try to run it for a week and see the result. Your queries needs optimization. If you could share your query, experts here could give you suggestions.

Re: Splunk timechart not displaying data when last 30 days range is selected

peterfox1992 — Wed, 23 Mar 2022 07:27:02 GMT

Hi, I tried to ran for a week but result populated for only 5 days and last 2 days populated as 0.

This is the query which I'm using. Please let me know if I'm missing something.

index="foo" sourcetype="xyz" user!="abc" method=POST (url="*search*aspx*" AND code!=302 AND code!=304 AND code!=401 AND code!=403 AND code!=0) [search index="foo" method_name=pqr message="*Response Time for method pqr*" | fields uniqid]
| eval hour=strftime(_time,"%H") | where hour >=7 AND hour <=19
| timechart span=1d count(eval(time_took)) as Total , count(eval(time_took<2000)) as Success, count(eval(time_took>2000)) as misses | sort by "_time" desc

Re: Splunk timechart not displaying data when last 30 days range is selected

peterfox1992 — Wed, 23 Mar 2022 07:29:10 GMT

Hi, Yes. Job inspector shows the below message.

The following messages were returned by the search subsystem:

info : [subsearch]: Subsearch produced 10000 results, truncating to maxout [subsearch_maxout] 10000.

Re: Splunk timechart not displaying data when last 30 days range is selected

ITWhisperer — Wed, 23 Mar 2022 08:05:18 GMT

So this is why you are getting different results. Essentially, the subquery is being truncated before finding uniqids prior to sometime on 16th. You could try increasing the limit (limits.conf) or you may need to refactor your search to avoid this truncation.

Re: Splunk timechart not displaying data when last 30 days range is selected

somesoni2 — Wed, 23 Mar 2022 19:04:46 GMT

If you run below query for last 30 days, how many records do you get?

index="foo" method_name=pqr message="*Response Time for method pqr*" | stats count by uniqid

Also, give this a try

(index="foo" sourcetype="xyz" user!="abc" method=POST (url="*search*aspx*" AND code!=302 AND code!=304 AND code!=401 AND code!=403 AND code!=0)) OR ( index="foo" method_name=pqr message="*Response Time for method pqr*" ) | fields _time uniqid time_took | eval hour=strftime(_time,"%H") | where NOT (method=POST AND hour <7 AND hour >19) | bucket span=1d _time | stats dc(method) as methods count(time_took) as Total count(eval(time_took<2000)) as Success, count(eval(time_took>2000)) as misses by _time uniqid | where methods=2 | timechart span=1d sum(Total) as Total , sum(Success) as Success, sum(misses) as misses | sort by "_time" desc