Splunk Search

Search failing and not returning all results in Statistics

splunklearner12
Path Finder

My search does not complete even after giving it an over hour. The progress bar is all the way at the end, and it tells me that 441,760 of 28,000,000 odd events have been matched, which is the correct number of results expected. But in the Statistics tab, only 4,226 results are displayed. When I stop the search, Splunk lets me know that the search failed due to an error.

base_search1 | rename destination_ip as ip
| append [search base_search2 | rename source_ip as ip ]
| append [search base_search3 | rename source_ip as ip ]
| append [search base_search4 | rename destination_ip as ip ]
| dedup ip | table ip | sort 0 + ip
| outputlookup ip.csv
| lookup dnslookup clientip AS ip OUTPUTNEW clienthost AS ip_resolved
| fillnull value="not found" ip_resolved
| table ip, ip_resolved
| outputlookup hosts.csv

I have done the same search for a smaller dataset before which had 52,000 odd results of 28 million events and worked fine. Last time I made it work by ensuring I use sort 0, and edited limits.conf maxresultrows value to be 60,000. For the new search, even after increasing the maxresultrows value to 500,000 after my search first failed, it didn't work but returned 4,000 odd results instead of 2,000 odd results in my earlier attempt. However, the ip.csv is created correctly and contains all the values.

Do I need to make any other conf file changes or something else?

My limits.conf looks like this:
[searchresults]
maxresultrows = 500000

Maximum number of times to try in the atomic write operation

(1 = no retries)

tocsv_maxretry = 5

Retry period is 1/2 second (500 milliseconds)

tocsv_retryperiod_ms = 500

Tags (2)
0 Karma

splunklearner12
Path Finder

The point of failure seems to be the dnslookup.

0 Karma

teunlaan
Contributor

dnslookup can be a killer indeed. Please check if your DNS server if it can handle all the request. + check your timeout settings for DNS request.

splunklearner12
Path Finder

Thanks, I will probably leave as it is not vital for me to do this, though it would have been nice.

0 Karma

adonio
Ultra Champion

your appends are killer ...
maybe try something like that:
... (base_search1) OR (base_search2) OR (base_search2) OR (base_search4) ...
| eval ip = coalesce(source_ip,destination_ip)
| dedup ip .... | all the rest ...

0 Karma

splunklearner12
Path Finder

Thank you adonio. This won't work for me, as my base searches are like this:
source=source1 type=inbound | rename source_IP as ip
| append [search source=source1 type=outbound | rename destination_ip as ip ]
| append [search source=source2 internal_src="NONE" | rename source_ip as ip ]
| append [search source=source2 internal_dest"=NONE" | rename destination_ip as ip ]
If I did "source=source1 type=inbound source_ip=* OR type=outbound destination_ip=*", it wouldn't get me useful results because every event contains source_ip and destination_ip and I am looking to resolve a list of external IP's.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...