Splunk searches returning partial or inconsistent results

[fedayn05]

Hi everyone,

Recently, I’ve been experiencing several issues with Splunk searches. I’m sometimes unable to query logs from certain indexes, and the behavior seems inconsistent.

For example, I can run the exact same SPL query multiple times. Sometimes it returns the expected results, while other times it returns no results or fails after running for some time with the following message:

"The search process with search_id="1785328480.2011339" may have returned partial results. Try running your search again. If you see this error repeatedly, review search.log for details or contact your Splunk administrator."

The issue seems to occur intermittently, and I’m not sure whether it is related to the search process, indexers, resource usage, or another Splunk component.

Has anyone encountered a similar issue or could point me in the right direction for troubleshooting? Any suggestions on what I should check (search.log, indexer health, resource utilization, etc.) would be greatly appreciated.

Thanks in advance for your help!

[gcusello]

Hi @fedayn05 ,

I frequently encountered this issue in an insufficient resources architecture:

the Indexers were installed on a VM-Ware cluster but resources weren't dedicated and I had around 400 virtual cpus working on 48 physical cpus.

So there were situations in which the physical infrastructure wasn't in condition to support the requests.

Check the vm-ware status report and the resources that you are dedicating to Splunk, remembering that Splunk requires, on a virtual infrastructure, to have dedicated resources and a plus of around 30% if resources are virtual.

Ciao.

Giuseppe

[livehybrid]

Hi @fedayn05  I would start by using Job inspector

 Updating Media

and then clicking on search.log

Review the output of this to see if there are any glaring errors or issues with the search, feel free to post the contents here but be sure it doesnt contain anything sensitive first.

You can also check the _internal index for errors around the time of your search which might get us some indication of what is going wrong, but the above is probably a good starting point.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

  Your feedback encourages the volunteers in this community to continue contributing.

[fedayn05]

Hello @livehybrid ,

Thank you for you answer , i followed your instructions , below is the error , it mentionts something about cache, For information , I am using smartstore as i am integrating my splunk with Minio.

Please find the error attached.

Thank you so much for you answer