Splunk searches returning partial or inconsistent results

[fedayn05]

Hi everyone,

Recently, I’ve been experiencing several issues with Splunk searches. I’m sometimes unable to query logs from certain indexes, and the behavior seems inconsistent.

For example, I can run the exact same SPL query multiple times. Sometimes it returns the expected results, while other times it returns no results or fails after running for some time with the following message:

"The search process with search_id="1785328480.2011339" may have returned partial results. Try running your search again. If you see this error repeatedly, review search.log for details or contact your Splunk administrator."

The issue seems to occur intermittently, and I’m not sure whether it is related to the search process, indexers, resource usage, or another Splunk component.

Has anyone encountered a similar issue or could point me in the right direction for troubleshooting? Any suggestions on what I should check (search.log, indexer health, resource utilization, etc.) would be greatly appreciated.

Thanks in advance for your help!

[cartergray70543]

Start with search.log and the Job Inspector, then check indexer health, CPU/memory, disk space, and search peer connectivity. If it only affects certain indexes, focus on the indexers hosting those indexes. Intermittent partial results often point to a resource or connectivity issue.

[gcusello]

Hi @fedayn05 ,

I frequently encountered this issue in an insufficient resources architecture:

the Indexers were installed on a VM-Ware cluster but resources weren't dedicated and I had around 400 virtual cpus working on 48 physical cpus.

So there were situations in which the physical infrastructure wasn't in condition to support the requests.

Check the vm-ware status report and the resources that you are dedicating to Splunk, remembering that Splunk requires, on a virtual infrastructure, to have dedicated resources and a plus of around 30% if resources are virtual.

Ciao.

Giuseppe

[fedayn05]

Hello @gcusello,

Thank you for your answer, I doubt this is some resources issue , As for my environment i am using ubuntu 22.04 with 40 GO of RAM and 38 in CPU , and 2 To NVME on local disk + enough resources on MInIO.

I am really confused and do not know what to do.

[gcusello]

Hi @fedayn05 ,

we also had apparently sufficient resources, but, viewing the VM-Ware console, how many applications are running on the same physical infrastructure?

We had 10 Indexers and 3 Search Heads with 40 CPUs each one, but all of them worked on the same 36 physical CPUs on the VM-Ware infrastructure!

Splunk required, for virtual installations, dedicated resources!

in addition, how many scheduled searches you are running?

how many of them are in real-time mode?

38 CPUs are many in general but maybe are not enough for your requirements.

The behaviour you described id usually in case of not enough resources.

Ciao.

Giuseppe

[livehybrid]

Hi @fedayn05  I would start by using Job inspector

 Updating Medialivehybrid_1-1786513748709.png

and then clicking on search.log

livehybrid_0-1786513744914.png

Review the output of this to see if there are any glaring errors or issues with the search, feel free to post the contents here but be sure it doesnt contain anything sensitive first.

You can also check the _internal index for errors around the time of your search which might get us some indication of what is going wrong, but the above is probably a good starting point.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

  Your feedback encourages the volunteers in this community to continue contributing.

[fedayn05]

Hello @livehybrid ,

Thank you for you answer , i followed your instructions , below is the error , it mentionts something about cache, For information , I am using smartstore as i am integrating my splunk with Minio.

Please find the error attached.

Thank you so much for you answer 

[isoutamo]

As @gcusello said it's high possibility that this is resource issue especially when you seem to use minio as your S3 backend.

Can you tell more about your env?

Anyhow you must Loog what your monitoring shows about resource usage on hw/vm/minio/splunk level.

[fedayn05]

Hello ,

Thank you for your answer, I doubt this is some resources issue , As for my environment i am using ubuntu 22.04 with 40 GO of RAM and 38 in CPU , and 2 To NVME on local disk + enough resources on MInIO.

I am really confused and do not know what to do.

[PickleRick]

08-12-2026 05:55:49.911 ERROR S2BucketCache [2772400 localCollectorThread] - waitFor probe, cache_id="bid|splunk_index~458~2366B991-21E7-46B8-A1A5-DA16C9370DBC|", did not localize all files before reaching download_status=idle files={"file_types":["tsidx","deletes"]} local_files={"file_types":["sourcetypes_data","sources_data","hosts_data","bucket_info","bfidx"]} failure_code=0 failure_reason=Cache was full and space could not be reserved

AFAIR this error means that your local Splunk indexer didn't have enough space (and couldn't make enough space free) to pull the bucket filex to cache. Cache manager knows that there is a remote file in Smartstore but couldn't download it. Check your Smartstore activity in Monitoring Console.