My search does not complete even after giving it an over hour. The progress bar is all the way at the end, and it tells me that 441,760 of 28,000,000 odd events have been matched, which is the correct number of results expected. But in the Statistics tab, only 4,226 results are displayed. When I stop the search, Splunk lets me know that the search failed due to an error.
base_search1 | rename destination_ip as ip
| append [search base_search2 | rename source_ip as ip ]
| append [search base_search3 | rename source_ip as ip ]
| append [search base_search4 | rename destination_ip as ip ]
| dedup ip | table ip | sort 0 + ip
| outputlookup ip.csv
| lookup dnslookup clientip AS ip OUTPUTNEW clienthost AS ip_resolved
| fillnull value="not found" ip_resolved
| table ip, ip_resolved
| outputlookup hosts.csv
I have done the same search for a smaller dataset before which had 52,000 odd results of 28 million events and worked fine. Last time I made it work by ensuring I use sort 0, and edited limits.conf maxresultrows value to be 60,000. For the new search, even after increasing the maxresultrows value to 500,000 after my search first failed, it didn't work but returned 4,000 odd results instead of 2,000 odd results in my earlier attempt. However, the ip.csv is created correctly and contains all the values.
Do I need to make any other conf file changes or something else?
My limits.conf looks like this:
[searchresults]
maxresultrows = 500000
Maximum number of times to try in the atomic write operation
(1 = no retries)
tocsv_maxretry = 5
Retry period is 1/2 second (500 milliseconds)
tocsv_retryperiod_ms = 500
... View more