topic Re: long running job in Dashboards & Visualizations

long running job

surekhasplunk — Mon, 24 Aug 2020 02:42:01 GMT

Hi,

I have written a query to generate lookup file for last 30days, which is taking a lot of time like almost 4 hours which is high on cpu. So can is there a option to run query everyday but run only for last 24 hours and append to the same lookup file generated yesterday, so that the dashboard populates quickly with all the 30days data post comparison

Re: long running job

Nisha18789 — Mon, 24 Aug 2020 05:52:40 GMT

Hello @surekhasplunk , yes that possible, like below

<your query to generate the data for last 24 hour>| outputlookup <lookup name.csv> append=true

Also, you can use summary index fir storing this data in case the lookup has a chance to get very bulky with time.

Re: long running job

surekhasplunk — Mon, 24 Aug 2020 07:32:04 GMT

thanks @Nisha18789

So in case i use summary index, i have to schedule it to run everyday for last 24hrs ?

or once in a month with last 30 days ?

Also it will impact the license usage right where as when we write to lookup file it wont affect the license usage

Please explain

Re: long running job

Nisha18789 — Mon, 24 Aug 2020 10:42:08 GMT

Hi @surekhasplunk , running after midnight , for previous day will be good.

Also, logging to summary index does not add to license usage as this data is already ingested in your original index.