Incomplete search reults error

[sahityasweety]

I am getting this error,

may have returned partial results try running your search again.if you see this error repeatedly, review search.log for details or contact your Splunk administration

Can i please get solution for this

Thanks,

sahitya

[kiran_panchavat]

@sahityasweety 

Check the Job Inspector. In the Inspector popup window, there is another link to the search.log that gives you some very detailed information. Another way to see more info about your errors is to open a plain search window and do a search like:

index=_internal error

Review your search query to ensure it is correctly formulated. Check for any typos, missing keywords, or syntax errors.

Verify that the time range specified in your search covers the relevant data.

Ensure that you are searching within the appropriate indexes.

Check the Splunk logs (such as splunkd.log and search.log) for any errors or warnings related to your search.

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Searching_and_Reporting/Troubleshooting_and_...

https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Search/ViewsearchjobpropertieswiththeJobI...r

If this reply helps you, Karma would be appreciated.

[sahityasweety]

Hi @kiran_panchavat  @richgalloway 

thankyou for your response!

Full error message: Error message : [Indexer-x] The search process with search_id="remote_ip-<xxx>" may have returned partial results. Try running your search again. If you see this error repeatedly, review search.log for details or contact your Splunk administrator.

We implemented smartstore for a high volume index a few weeks ago and have since been experiencing issues with search and replication factors not meeting for smartstore enabled indexes. We raised a support case with Splunk about this and they informed us that it is a known bug with no fix version available. We hadn't seen this issue before last week, but it's been showing up in most searches for the past 3-4 days (with smartstore index).

Furthermore, we are experiencing search performance issues on the smartstore index; either it takes a long time to fetch results or the search gets stuck if we run the query for more than 24 hours.

index configuration:
[smartstore_index]
frozenTimePeriodInSecs = 15552000
repFactor = auto
maxDataSize = 1000
maxHotBuckets = 30
hotlist_recency_secs = 86400maxGlobalRawDataSizeMB = 62914560
homePath = /data/smartstore_index/db
coldPath = /data/smartstore_index/colddb
thawedPath = /data/smartstore_index/thaweddb
remotePath = volume:remote_store/smartstore_index

Approx daily ingestion on index: 2TB per day.
Local SSD volume size: 16TB
Remote location: S3 bucket

We're not sure, if we're receiving this error because of search & replication factor issue, Request help to fix.

[richgalloway]

That error usually is seen when an indexer stops while processing a search.  What relevant messages do you seen in search.log?  Are there any core files from the indexers?

The local SSD provides only 8 days of storage.  Any query that searches data more than 8 days old will have to fetch the data from SmartStore, which will slow processing of the search.  Splunk recommends the local cache be large enough to hold at least 30 days of data.

[richgalloway]

Did you run the search again and get the same message?  If so, what did you find in search.log?