TLDR: I'm trying to automate the large 25 day search to break up into 25 separate one day searches. I'm updating a lookup table that is tracking which indexes are affected by the new log4j exploit.  I do this so that I can only have to search through the affected indexes with subsequent searches.  This lookup table takes hours each time it is updated for a day.  Problem being, I need to know all of the affected indexes over all of the days log4j since December 10th or so.   Query that updates lookup table:   NOT [| inputlookup log4j_indexes.csv | fields index] | regex _raw="(\$|%24)(\{|%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|%3A|\$|%24|}|%7D)" | table index | inputlookup append=true log4j_indexes.csv | dedup index | outputlookup log4j_indexes.csv​   Each time this query finishes, it appends log4j-exploit-affected indexes to the lookup table.  I need to automate the scanning over a large timeframe (December 10th 2021 - January 5th 2022).  However, I want the lookup table to update as it runs over each day.  I'm trying to automate the large 25 day search to break up into 25 separate one day searches.  This also makes it so that if the search fails, then I don't lose all progress.  I can then apply this same methodology to other searches. Lookup Table (Log4J_affected_indexes) Index index_1 index_2   How I've tried to solve the problem Commands I've tried while attempting to solve: foreach map gentimes subsearch saved searches Gentimes (smaller timeframes) -> map Explanation of Query below: The gentimes part creates a table based on the selected timerange: Earliest  Latest 01/02/2022:00:00:00 01/03/2022:00:00:00 01/03/2022:00:00:00 01/04/2022:00:00:00 01/04/2022:00:00:00 01/05/2022:00:00:00   I try to pass those values to a subsearch as the earliest and latest parameters using map.  I understand now that map doesn't seem to work for this, and I get no results when the search runs. (gentimes and map) Query:   |gentimes start=-1 |addinfo |eval datetime=strftime(mvrange(info_min_time,info_max_time,"1d"),"%m/%d/%Y:%H:%M:%S") |mvexpand datetime |fields datetime |eval latest=datetime |eval input_earliest=strptime(datetime, "%m/%d/%Y:%H:%M:%S") - 86400 |eval earliest=strftime(input_earliest, "%m/%d/%Y:%H:%M:%S") |fields earliest, latest | map search="search NOT [| inputlookup log4j_indexes.csv | fields index] earliest=$earliest$ latest=$latest$ | regex _raw=\"(\$|\%24)(\{|\%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|\%3A|\$|\%24|}|\%7D)\" | table index | inputlookup append=true log4j_indexes.csv | dedup index | outputlookup log4j_indexes.csv"   Gentimes subsearch -> main search Explanation of Query below: I use gentimes in a subsearch to produce smaller timeframes from the larger selected timeframe: Earliest  Latest 01/02/2022:00:00:00 01/03/2022:00:00:00 01/03/2022:00:00:00 01/04/2022:00:00:00 01/04/2022:00:00:00 01/05/2022:00:00:00   This doesn't give me errors.  However, I get no matches.  I can almost guarantee this isn't running separate searches per value displayed in the above table.  I'm not sure how this can be done. (gentimes subsearch) Query:   NOT [| inputlookup log4j_indexes.csv | fields index] [|gentimes start=-1 |addinfo |eval datetime=strftime(mvrange(info_min_time,info_max_time,"1d"), "%m/%d/%Y:%H:%M:%S") |mvexpand datetime |fields datetime |eval latest=datetime |eval input_earliest=strptime(datetime,"%m/%d/%Y:%H:%M:%S") - 86400 |eval earliest=strftime(input_earliest,"%m/%d/%Y:%H:%M:%S") |fields earliest, latest] | regex _raw="(\$|\%24)(\{|\%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|\%3A|\$|\%24|}|\%7D)" | table index | inputlookup append=true log4j_indexes.csv | dedup index | outputlookup log4j_indexes.csv   Conclusion Other failed attempts: using foreach (can't do non-streaming) passing earliest and latest parameters to saved-search savedsearch doesn't work this way Other solutions I've thought of: Running subsearch that updates a smaller_timeframe.csv file that keeps track of the smaller timeframes.  Then, pass those timeframe parameters (earliest / latest) into a search somehow. Somehow do a recursive sort of search where each search triggers another search to go.  Consequently, I could have a search trigger another search with the earliest and latest values incremented forward one day (or any amount of time). Maybe, Splunk has a feature (not on the search head) that can automate the same search over small timeframes, and over a large period of time.  Maybe this unknown-to-me feature also has scheduling built into it. If there is any other information that I can give to help others solve this with me, then just ask.  I can edit this post... ... View more

Log4J Query:   index=* | regex _raw="(\$|%24)(\{|%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|%3A|\$|%24|}|%7D)" | eval action=coalesce(action_taken, elb_status_code, status) | where NOT (cidrmatch("192.168.0.0/16",src_ip) OR cidrmatch("10.0.0.0/8",src_ip) OR cidrmatch("172.16.0.0/12",src_ip)) OR Country="United States" | iplocation src_ip | eval notNULL="" | fillnull value="unknown" notNULL, src_ip, dest_ip, action, url, Country | stats count by src_ip, Country, dest_ip, url, action, sourcetype | sort - count​   This checks anywhere where there is a sign of the Log4J exploit being used.  I've done field extraction on any sourcetypes returned by my previous query:  index=* | regex _raw="(\$|%24)(\{|%7B)([^jJ]*[jJ])([^nN]*[nN])([^dD]*[dD])([^iI]*[iI])(:|%3A|\$|%24|}|%7D)" | stats count as "exploit attempts" by sourcetype | sort - "exploit attempts" I extracted fields so that I can get a table with src_ip, Country, dest_ip, url, action, sourcetype, and count. I want to then use this query in subsequent queries to get information on if the exploit was successful, and if there is any other communication that follows.   The query works and I get results like this (fake results): src_ip Country dest_ip url action sourcetype count 248.216.243.59 Unknown 192.168.1.148 192.168.1.148/ blocked firewall 3 207.191.80.208 US 192.168.1.216 192.168.1.216/ allowed firewall 2    Problem being... The query runs really slow after a few minutes of running. It starts out by doing millions of events every few seconds and slows down to doing thousands every few seconds.  Some info from logs: command.search, command.search.kv, and dispatch.stream.remote take up the most time of the run. I'm getting warnings in search.log like "Max bucket size is larger than the index size limit" , "Invalid field alias specification in stanza". However, these don't seem to be the reason for the error. Using high_perf and Fast Mode If there is any more information I can add, then feel free to ask and I will edit.   ... View more

I want to know how I can incrementally go through and add missing times (hours) per user across a number of users.   Also the fail_num for those times should be '0'.    I've thought of using foreach, but I'm not sure this is the route to go here.   Above is a result showing only 2 users from the query I'm making in one of my previous posts ("Detecting-Spikes-Anomalies-in-Failed-Logins-over-time"). For cases where the fail_num was 0 no entry was made and therefore I have no row for that timeslot.  When I use trendline to analyze this it won't work because it doesn't have enough points of data to compute the moving average.  I cannot use timechart to do this.   So, if there is a more programmatic way to add rows that are missing in by using regex or some more efficient method, then someone please enlighten me on this. I can't use timechart to fill these in because it breaks other things in my query.   I've thought of using foreach, but I'm not sure this is the route to go here. If you're wondering why I can't use timechart it makes it impossible for me to do the analysis unless 3d dimensional analysis or something exists in Splunk...sadly I'm not mathematically or programmatically gifted enough to think up such a solution. ... View more