Why custom search command not working without being prefixed by dedup

[Cristian]

Hi,

I created a custom StreamingCommand which makes REST API calls to get user details, based on a userid.

If command is executed as below, it is working as expected

index="<hidden_index>" <hidden_filters> | dedup jobFields.user | getuserdetails fields="Division,FullName" userid=jobFields.user 

 If I remove the "dedup ..." then the command crashes:

index="<hidden_index>" <hidden_filters> | getuserdetails fields="Division,FullName" userid=jobFields.user 

Error: 
[hidden_index_server]Streamed search execute failed because: Error in 'getuserdetails' command: External search command exited unexpectedly with non-zero error code 1..

The stream method code is below:

def stream(self, records):
        for record in records:
            try:
                result = self.getUserDetails(record[self.userid])
                self.log(topic='STREAM', value=result)
                for field in self.fields:
                    if field in result:
                        record[field] = result[field]
                yield record
            except (Exception) as e:
                template = "An exception of type {0} occurred. Arguments:\n{1!r}"
                message = template.format(type(e).__name__, e.args)
                self.log(topic='ERROR', value=message)
        try:
            #update cache details for later use
            self.cachedDetails.updateCache()
        except (Exception) as e:
                template = "An exception of type {0} occurred. Arguments:\n{1!r}"
                message = template.format(type(e).__name__, e.args)
                self.log(topic='ERROR', value=message)

The only reason I needed dedup was because I wanted to save the API calls but now I have a cache and I do not need to do this anymore.

Somehow the error seem to come from the indexers. We have 12 indexers and I get 12 error, 1 for each indexer.

I tried with "localop" but while the errors disappeared,  this is too slow to even consider having it in production.

Any suggestion?

Thanks,

Cristian