I have a dashboard with multiple panels referring to the same base search.
I want all my data flowing in the base search and then the panels should refer to the base search for post processing. Splunk indicates we can only use transforming commands in base search so I came up with the following query.
index=aaa cz_cf_name=bbb | bucket _time span=5m | stats count by _time, cz_event_type,cz_message_type, cz_cf_app_id, cz_cf_app_name, cz_message_type,cz_source_type
Total result for above query for last 4 hours only 80K (actual count should be 195K). I see that it's removing the rows with some of the null fields when I run query this way.
So I changed my query as below just to validate whether I am receiving all counts.
index=aaa cz_cf_name=bbb | bucket _time span=5m | stats count, values(cz_event_type) as cz_event_type values(cz_cf_app_id) as cz_cf_app_id, values(cz_cf_app_name) as cz_cf_app_name, values(cz_source_type) as cz_source_type, values(cz_message_type) as cz_message_type by _time
Total result for above query for last 4 hours is 195K. So thats a good news but I can't use this result for any post processing.
Would anyone please let us know if there is a way to write optimal base search that can be used without loosing any data?
I also understand that there is a limit of 500K events for base search. Is it really advisable to use base query because we can easily cross this limit if a user expands the timeframe?
Since it depends on the configuration of the server and the resource, I think that it is better to try it the way it actually is tried.
However, I do not recommend it when dealing with large data because base search is slow.
Thanks for the response Hiroshi.
It really make sense to fetch the underlying data only once, instead of 25 times again and again in separate panels. I am looking for stats or any other command that consumes less space on disk when dashboard is loaded. The reason behind it is we have only 100 MB per user limit and with heavy dashboard load, the limit is easily reached and impacting user's ability to load any more dashboards or perform ad-hoc searches until the queries are expiring. Increasing disk space per user is not an option at this time (as we are talking about 1000+ users).
So overall I don't want to loose the data but still want to leverage base search or search template. Query 1 is rendering fewer than expected search results and query 2 is formatting search result in an non-useful way!!!!