Search failing and not returning all results in Statistics

[splunklearner12]

My search does not complete even after giving it an over hour. The progress bar is all the way at the end, and it tells me that 441,760 of 28,000,000 odd events have been matched, which is the correct number of results expected. But in the Statistics tab, only 4,226 results are displayed. When I stop the search, Splunk lets me know that the search failed due to an error.

base_search1 | rename destination_ip as ip
| append [search base_search2 | rename source_ip as ip ]
| append [search base_search3 | rename source_ip as ip ]
| append [search base_search4 | rename destination_ip as ip ]
| dedup ip | table ip | sort 0 + ip
| outputlookup ip.csv
| lookup dnslookup clientip AS ip OUTPUTNEW clienthost AS ip_resolved
| fillnull value="not found" ip_resolved
| table ip, ip_resolved
| outputlookup hosts.csv

I have done the same search for a smaller dataset before which had 52,000 odd results of 28 million events and worked fine. Last time I made it work by ensuring I use sort 0, and edited limits.conf maxresultrows value to be 60,000. For the new search, even after increasing the maxresultrows value to 500,000 after my search first failed, it didn't work but returned 4,000 odd results instead of 2,000 odd results in my earlier attempt. However, the ip.csv is created correctly and contains all the values.

Do I need to make any other conf file changes or something else?

My limits.conf looks like this:
[searchresults]
maxresultrows = 500000

Maximum number of times to try in the atomic write operation

(1 = no retries)

tocsv_maxretry = 5

Retry period is 1/2 second (500 milliseconds)

tocsv_retryperiod_ms = 500

[splunklearner12]

The point of failure seems to be the dnslookup.

[teunlaan]

dnslookup can be a killer indeed. Please check if your DNS server if it can handle all the request. + check your timeout settings for DNS request.

[splunklearner12]

Thanks, I will probably leave as it is not vital for me to do this, though it would have been nice.

[adonio]

your appends are killer ...
maybe try something like that:
... (base_search1) OR (base_search2) OR (base_search2) OR (base_search4) ...
| eval ip = coalesce(source_ip,destination_ip)
| dedup ip .... | all the rest ...

[splunklearner12]

Thank you adonio. This won't work for me, as my base searches are like this:
source=source1 type=inbound | rename source_IP as ip
| append [search source=source1 type=outbound | rename destination_ip as ip ]
| append [search source=source2 internal_src="NONE" | rename source_ip as ip ]
| append [search source=source2 internal_dest"=NONE" | rename destination_ip as ip ]
If I did "source=source1 type=inbound source_ip=* OR type=outbound destination_ip=*", it wouldn't get me useful results because every event contains source_ip and destination_ip and I am looking to resolve a list of external IP's.