topic Re: Performance issue in Getting Data In

Performance issue

rakesh44 — Tue, 18 Dec 2018 14:04:29 GMT

I have 24 months of data from Jan 2017 to Nov 2018

I have count of 23900000 for year 2017
27900000 for year 2018

I need all required columns given in below query.

Index=test sourcetype=historical Period="Jan-17 | table rft period expenses ID transaction balance region area country

If i run above command to fetch one month of data , it takes more than 30 minutes.
how do i improve performance, I cannot try creating lookup & summary Indexing as data is huge , do we have any other option to expedite search query to run fast.Thanks

Re: Performance issue

ddrillic — Tue, 18 Dec 2018 15:12:02 GMT

I think @adonio answer at Can we improve on a standard index=index_name sourcetype=*prod | stats query? can be applied in your case.

Re: Performance issue

rakesh44 — Tue, 18 Dec 2018 15:45:39 GMT

That Information is not useful in my case

I need below columns in results

index=test sourcetype=historical Period="Jan-17 | table rft period expenses ID transaction balance region area country

Re: Performance issue

niketn — Tue, 18 Dec 2018 16:38:58 GMT

@rakesh44 check out David Veuve 's .Conf Session for using Search Acceleration mechanisms:

https://conf.splunk.com/conf-online.html?search=FN120994#/

Re: Performance issue

DalJeanis — Wed, 02 Jan 2019 18:26:24 GMT

@rakesh44 - change table to fields and test again. table is not a streaming command, so it's causing all the data to be sent to the search head.

Re: Performance issue

rakesh44 — Thu, 03 Jan 2019 05:10:03 GMT

Hi DalJeanis,

I have view data in tabular format to check & compare each events & hence i could not fields command.Do we have any alternate option for table commands.

Thanks

Re: Performance issue

rakesh44 — Thu, 03 Jan 2019 05:10:46 GMT

@niketnilay , I dont used any reporting commands & hence it will not allow me to use report acceleration

Re: Performance issue

rakesh44 — Fri, 04 Jan 2019 05:48:45 GMT

Hi Daljeanis,

I have checked with fields command not that much improvement, which i expected .
Thanks

Re: Performance issue

woodcock — Sun, 03 Feb 2019 02:13:38 GMT

Splunk is about matching and aggregation: turning haystacks into needles. It is not a haystack delivery system. You don't really need all umpty-kazillion events, right? You need to report something that you calculate or isolate about them right? Tell us what that thing is and we can help you get there. If you need haystacks moved and presented, then Splunk is the wrong tool.