I am trying to run a search that returns one row of results over a long historical time window on a per hour basis.
Due to investigation method, I need to run this over around almost 60 days.
The search cannot be timecharted or timeboxed via stats because there are subsearches that return fields relevant for that particular hour...
I was thinking of the below search in a scheduled search...
Each time range will have a row with results as well because we have append=true
The problem is, if there are no results than the row is not generated, & as a result, the time lookup is stuck at that time, window & stale..
How do I account for this?
Is there a better approach to run searches multiple times on a time basis. e.g 1 hour.
index=_internal [| inputlookup results_and_time.csv | tail 1 | fields earliestTime latestTime | rename earliestTime as earliest , latestTime as latest | format] | addinfo | stats values(sourcetype) as sourcetype earliest(info_min_time) as earliestTime , latest(info_max_time) as latestTime | eval earliestTime = relative_time(earliestTime,"+1h") , latestTime = relative_time(latestTime,"+1h") | outputlookup append=true results_and_time.csv
p.s. Adding the time range manually for the initial run without the inputlookup to add the earliest & latest..
Just schedule the search with a time window defined for example: -400h@h till 399h@h and schedule it to run each hour, then the time window automatically shifts each hour. Just pick the offset such that you search over the desired window?
In this case we will have to wait for timespan='the number of reporting hours' to cover all the offset windows
or did i miss something in the logic please?
This is spanning 60+ days thus my attempt to auto increment the span for every subsequent search..
You can try to use
| gentimes start=-60 increment=1h | map maxsearches=10000 search="search index=_internal earliest=$starttime$ latest=$endtime$ | stats values(sourcetype) as sourcetypes | eval earliest=\"$starttime$\", latest=\"$endtime$\" "
thanks for the answer nryabykh!
This certainly gives me a way to spawn multiple searches off the same base search with incrementing timespans . Which I didnt realise before!
The problem is that The map command spawns them at a go?
is it sequential & will only fire off after once the 'map search' is completed for the 'active' row.
As far as I know, the map command handles the input rows sequentially, but there is no any kind of "preview results". You have to wait for the map command to handle all rows before you're able to do anything else within your query. But it doesn't matter if some of map's subsearches returns nothing. In this case there will be just several empty rows in output table.