Custom Search Command Fails to Return Results

[caseyra]

Hello,

I created a custom search command that queries an external service and returns a set of results using the v2 API (GeneratingCommand). This works perfectly on a standalone box, but when I put it onto a cluster, it appears to run but not display any results. Is there anything I could be missing that would cause this to occur?

Some details:

The cluster consists of a single standalone search node that is connected to 6 indexers. They are all part of the same cluster. This search node, however, is separate from the others for development purposes.

I can see that the search command runs on all of the index nodes and I can see that it is distributed to them. I have logging enabled, so I can see that it is getting results, but I just don't see the results showing up.

My command.conf looks similar to this:

[command]
filename = command.py
chunked = true
passauth = true
requires_srinfo = true
enableheader = true
stderr_dest = true

Thanks.

Edit 1: I decided to poke around a bit more and I can see that in the metrics.log file, I can see the command I'm trying to execute with the suffix "-too_small". Does this mean that the command simply isn't generating enough data? It's never going to get much, but it is necessary. Is there an option I can set to override this?

[David]

Have you tried adding | localop before your command? (You can add | localop before anything, even | localop | search index=* | head 1000.) You shouldn't need to, but it might be a viable workaround.

[caseyra]

That worked, but @kchamplin's answer works better in our case.

Thanks!

[David]

I'll come in second place to @kchamplin any day! Glad you got it working.

[svasani_splunk]

change stderr_dest = log from strderr_dest=true in commands.conf and post the error if there are any.

[caseyra]

Done, but I'm not seeing any errors.

[svasani_splunk]

Did you restart splunk?

[caseyra]

I was just doing the debug/refresh, but I did restart it was well. No change.

[kchamplin_splun · Accepted solution]

Are you using the Splunk SDK as part of your custom search command? If so, what version of the SDK?

[caseyra]

Yes, I am. Looks like I have 1.6.0-py2.7.

[kchamplin_splun]

You may want to try using the develop branch of the python SDK for now. There's a known bug for the generating command library and SCP2. You will still need to set distributed to false in your @configuration decorator, ex:
@Configuration(distributed=False

For the develop branch go here:
https://github.com/splunk/splunk-sdk-python/tree/develop

Reference for the issue:
https://github.com/splunk/splunk-sdk-python/pull/182/commits/edd5d1f2ddf1ab36a253588a7cdd472775c149c...

[caseyra]

Thanks. I didn't get a copy of the SDK yet, but I did add the distributed setting (True for now based on the bug) and it's working properly now. If I can, I'll get the version from the develop branch otherwise I'll wait until a new version is released.

[peterchenadded]

You probably want to set local=true in commands.conf so it only runs once on your search head.

If still not work what's the message you get in stderr?

[caseyra]

Unfortunately, this didn't work. As for errors, I'm not getting any. The only thing I see coming from stderr in the search log are messages from my logger.

[peterchenadded]

Have you tried setting local = true in your command.py?

Your first step should be making sure it runs just on search head once.

[caseyra]

Yes, I did try that, but it didn't work.

[kamlesh_vaghela]

HI

1) Can you please add below property and try again?

[command]
type = python

2) Have you checked the error log in splunkd.log and python.log??
3) Is that any error in search job ??

Thanks

[caseyra]

Hello,

1. No change using that property.
2. No errors in splunkd.log or python.log
3. If I look at the logs for the search job, the only "errors" I see are messages to STDERR that show the records I'm expecting to see in the search results, but that's it. I don't know why this occurs as I have it logging to a log file via the Python logging library.

Now, this brings me to something strange that I'm seeing. I log the fact that the the command is running 5 times, but I only see results coming back 4 times. Could the fact I'm not getting results back from every run be causing this? Why is this being executed multiple times?

[kamlesh_vaghela]

Hi
yeah, strange. Conf file looks ok. Can you please provide sample code of your python file?

[caseyra]

Sure. Here's basically what the code looks like minus imports and some logging config:

logger = logging.getLogger()

@Configuration
class GetRemoteData(GeneratingCommand):
    _defaultConfig = "/path/to/config.ini"
    _config = ConfigParser.RawConfigParser()
    _config.read(_defaultConfig)

    def generate(self):
        logger.info("Running GetRemoteData")
        try:
            md = self._metadata.searchinfo
            service = RemoteService(params)
            results = service.getData(str(md.username))
            if results == []:
                raise ValueError("No results could be found for the user: {}".format(md.username))

            for val in results:
                logger.info("Value: {}".format(val))
                parts = val.split(':')
                yield {'Name': parts[1], 'ID': parts[0]}
        except Exception as e:
            logger.exception(e)
            raise(e)

if __name__ == '__main__':
    dispatch(GetRemoteData, sys.argv, sys.stdin, sys.stdout, __name__)

Forgive me if my Python isn't quite right. I'm still fairly new to it.

[DalJeanis]

Have you verified that it is not running out of time or memory on the search head?