Getting the error "Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info." frequently

[anhhoangduc]

Dear everyone,
Have a good day ahead.

I am having the following issue that need your advice. Recently, I have deployed Splunk in distributed environment as the following:
- 01 Master + License master
- 01 Search Head
- 02 Indexer
- 01 Heavy Forwarder

Without installing app on Search Head, the application is working fine without any error. However, whenever I install app on SH, the following error is appeared for one of our Indexing system:
"Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info."

By checking the search.log, we see a lot of the following error:
12-03-2018 14:53:28.293 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
12-03-2018 14:53:28.701 ERROR SRSSerializer - could not read number of columns
12-03-2018 14:53:28.701 WARN SRSSerializer - could not read schema
12-03-2018 14:53:28.723 INFO TimelineCreator - Commit timeline at cursor=1543804147.000000
12-03-2018 14:53:28.724 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
12-03-2018 14:53:29.073 ERROR SRSSerializer - could not read number of columns
12-03-2018 14:53:29.073 WARN SRSSerializer - could not read schema
12-03-2018 14:53:29.095 INFO TimelineCreator - Commit timeline at cursor=1543803804.000000
12-03-2018 14:53:29.096 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
12-03-2018 14:53:29.601 ERROR SRSSerializer - could not read number of columns
12-03-2018 14:53:29.601 WARN SRSSerializer - could not read schema

Due to this error, I cannot search any event which is indexed by the problematic node.
Can you please advice how I should proceed further to fix this issue?

Thank you for your time in advance.
Regards,
Anh

[anhhoangduc]

thanks Jacob for your help.
I have gone through this post but still cannot fix the error. Actually, my search is very simple: sourcetype=pan:traffic
I have 2 indexers and only 1 indexer is having this issue.
Still cannot figure out what's wrong...

[hijacob]

Hello Anh,

look at this Troubleshooting...
https://helgeklein.com/blog/2017/07/troubleshooting-splunk-error-search-process-not-exit-cleanly/

Does it work?

Best wishes,
Jacob