Need to output 4 different queries to the same lookup daily, but have the data refresh every 24 hours.

[antoniolamonica]

My org has millions of events coming in through firewalls.
I had a 24 hour timeframe search take 12.5 hours to run. 

I was curious if I broke it up into 6 hour timeframes (changing the earliest/latest statements accordingly), and having them outputlookup to the same lookup file.

I would then inputlookup the file and tailor enrich accordingly, however I want to reset after each day.
ie. I do not want the file to keep growing.

Would I set append=false on query1, and append=true for query2, query3, and query4? 

[antoniolamonica]

Yeah, I was starting to consider that afterwards. 
I appreciate the assistance. 

[PickleRick]

Either I misunderstand something or you'd have your lookup "full" only for 1/4th of a day (or you're re-creating it whole in which case it's confusing why you want to have several copies of the same data).

What problem are you trying to solve?

[antoniolamonica]

Each query would be offset in its scheduling

queryA would run at midnight, looking back from the previous midnight - to previous 0600
queryB would run a bit later, looking back from the previous 0600 - to previous 1200
queryC would run a bit later, looking back from the previous 1200 - to previous 1800
queryD would run a bit later, looking back from the previous 1800 - to previous 0000

Purpose is intended to not create so much resource utilization.

I essentially want to piecemeal the 4 outputs into 1 lookup, read that lookup, enrich it, and schedule that as the alert itself.

Then I want it do it all over again, but I do not want the lookup to keep appending after a 24hr cycle. 

TL;DR I want a solution to break up a 24hr alert into chunks and bring it back together. 

[PickleRick]

So you have it - if you run the first instance you'll overwrite earlier gathered data. True, subsequent three runs will append to your lookup but only after the fourth run you'll have the full 24h-long result set.

I'd rather consider summary indexing instead of building a lookup.