Search process did not exit cleanly, exit_code=255, description="exited with code 255"

[brod_geico]

we are running with splunk V4 with search peers 4 indexer are connected to 1 SH , when im trying to search particular keyword it is just
giving error for particular node only node sure what is the issue.

Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
[XXXX-node-name] Streamed search execute failed because: Error in 'surrounding': The following required arguments were not provided to the SurroundingDataOperator: bucket
Could not find target event on the remote server, unable to form the proper distributed search

[kamal_jagga]

After the following line of error, you would find 1 more line, which will give you name of lookup or knowledge object which is failing while replication.

"Search process did not exit cleanly, exit_code=255",
...[server] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'abc.csv' does not exist.

Try adding local=t in your search. This will direct Splunk to look for this csv only on search head and not indexer and remove the error.

Good Luck !!!

[brod_geico]

i just found my issues why this is hitting again
On indexer servers on problem server we have 5.x version splunk running on rest of the server as well SH v4 version running , im assuming on that V4 version on SH not able to copy the Knowledge objects on proper way that is the reason that is giving issues.It make sense to me.

http://docs.splunk.com/Documentation/Splunk/5.0.3/Deploy/Whatisdistributedsearch

What search heads send to search peers
When initiating a distributed search, the search head replicates and distributes its knowledge objects to its search peers. Knowledge objects include saved searches, event types, and other entities used in searching across indexes. The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf. The set of data that the search head distributes is called the knowledge bundle.

The indexers use the search head's knowledge bundle to execute queries on its behalf. When executing a distributed search, the indexers are ignorant of any local knowledge objects. They have access only to the objects in the search head's knowledge bundle.

The process of distributing knowledge bundles means that indexers by default receive nearly the entire contents of all the search head's apps. If an app contains large binaries that do not need to be shared with the indexers, you can reduce the size of the bundle by means of the [replicationWhitelist] or [replicationBlacklist] stanza in distsearch.conf. See "Limit knowledge bundle size" in this manual.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers/ directory on each search peer. Because the search head distributes its knowledge, search scripts should not hardcode paths to resources. The knowledge bundle will reside at a different location on the search peer's file system, so hardcoded paths will not work properly.

By default, the search head replicates and distributes the knowledge bundle to each search peer. For greater efficiency, you can instead tell the search peers to mount the knowledge bundle's directory location, eliminating the need for bundle replication. When you mount a knowledge bundle, it's referred to as a mounted bundle. To learn how to mount bundles, read "Mount the knowledge bundle".

[brod_geico]

No luck some times it is working some times it is not.