Splunk Search
Highlighted

Custom Search Command Fails to Return Results

Explorer

Hello,

I created a custom search command that queries an external service and returns a set of results using the v2 API (GeneratingCommand). This works perfectly on a standalone box, but when I put it onto a cluster, it appears to run but not display any results. Is there anything I could be missing that would cause this to occur?

Some details:

The cluster consists of a single standalone search node that is connected to 6 indexers. They are all part of the same cluster. This search node, however, is separate from the others for development purposes.

I can see that the search command runs on all of the index nodes and I can see that it is distributed to them. I have logging enabled, so I can see that it is getting results, but I just don't see the results showing up.

My command.conf looks similar to this:

[command]
filename = command.py
chunked = true
passauth = true
requires_srinfo = true
enableheader = true
stderr_dest = true

Thanks.

Edit 1: I decided to poke around a bit more and I can see that in the metrics.log file, I can see the command I'm trying to execute with the suffix "-too_small". Does this mean that the command simply isn't generating enough data? It's never going to get much, but it is necessary. Is there an option I can set to override this?

Highlighted

Re: Custom Search Command Fails to Return Results

SplunkTrust
SplunkTrust

Have you verified that it is not running out of time or memory on the search head?

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Explorer

I haven't seen any indication that it is running out of memory.

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

SplunkTrust
SplunkTrust

HI

1) Can you please add below property and try again?

[command]
type = python

2) Have you checked the error log in splunkd.log and python.log??
3) Is that any error in search job ??

Thanks

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Explorer

Hello,

  1. No change using that property.
  2. No errors in splunkd.log or python.log
  3. If I look at the logs for the search job, the only "errors" I see are messages to STDERR that show the records I'm expecting to see in the search results, but that's it. I don't know why this occurs as I have it logging to a log file via the Python logging library.

Now, this brings me to something strange that I'm seeing. I log the fact that the the command is running 5 times, but I only see results coming back 4 times. Could the fact I'm not getting results back from every run be causing this? Why is this being executed multiple times?

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

SplunkTrust
SplunkTrust

Hi
yeah, strange. Conf file looks ok. Can you please provide sample code of your python file?

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Explorer

Sure. Here's basically what the code looks like minus imports and some logging config:

logger = logging.getLogger()

@Configuration
class GetRemoteData(GeneratingCommand):
    _defaultConfig = "/path/to/config.ini"
    _config = ConfigParser.RawConfigParser()
    _config.read(_defaultConfig)

    def generate(self):
        logger.info("Running GetRemoteData")
        try:
            md = self._metadata.searchinfo
            service = RemoteService(params)
            results = service.getData(str(md.username))
            if results == []:
                raise ValueError("No results could be found for the user: {}".format(md.username))

            for val in results:
                logger.info("Value: {}".format(val))
                parts = val.split(':')
                yield {'Name': parts[1], 'ID': parts[0]}
        except Exception as e:
            logger.exception(e)
            raise(e)

if __name__ == '__main__':
    dispatch(GetRemoteData, sys.argv, sys.stdin, sys.stdout, __name__)

Forgive me if my Python isn't quite right. I'm still fairly new to it.

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Path Finder

You probably want to set local=true in commands.conf so it only runs once on your search head.

If still not work what's the message you get in stderr?

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Explorer

Unfortunately, this didn't work. As for errors, I'm not getting any. The only thing I see coming from stderr in the search log are messages from my logger.

0 Karma
Highlighted

Re: Custom Search Command Fails to Return Results

Path Finder

Have you tried setting local = true in your command.py?

Your first step should be making sure it runs just on search head once.

0 Karma