Performance issue

[rakesh44]

I have 24 months of data from Jan 2017 to Nov 2018

I have count of 23900000 for year 2017
27900000 for year 2018

I need all required columns given in below query.

Index=test sourcetype=historical Period="Jan-17 | table rft period expenses ID transaction balance region area country

If i run above command to fetch one month of data , it takes more than 30 minutes.
how do i improve performance, I cannot try creating lookup & summary Indexing as data is huge , do we have any other option to expedite search query to run fast.Thanks

[woodcock]

Splunk is about matching and aggregation: turning haystacks into needles. It is not a haystack delivery system. You don't really need all umpty-kazillion events, right? You need to report something that you calculate or isolate about them right? Tell us what that thing is and we can help you get there. If you need haystacks moved and presented, then Splunk is the wrong tool.

[DalJeanis]

@rakesh44 - change table to fields and test again. table is not a streaming command, so it's causing all the data to be sent to the search head.

[rakesh44]

Hi DalJeanis,

I have view data in tabular format to check & compare each events & hence i could not fields command.Do we have any alternate option for table commands.

Thanks

[rakesh44]

Hi Daljeanis,

I have checked with fields command not that much improvement, which i expected .
Thanks

[niketn]

@rakesh44 check out David Veuve 's .Conf Session for using Search Acceleration mechanisms:

https://conf.splunk.com/conf-online.html?search=FN120994#/

[rakesh44]

@niketnilay , I dont used any reporting commands & hence it will not allow me to use report acceleration

[ddrillic]

I think @adonio answer at Can we improve on a standard index=index_name sourcetype=*prod | stats query? can be applied in your case.

[rakesh44]

That Information is not useful in my case

I need below columns in results

index=test sourcetype=historical Period="Jan-17 | table rft period expenses ID transaction balance region area country