topic Re: Custom Search Command Fails to Return Results in Splunk Search

Custom Search Command Fails to Return Results

caseyra — Fri, 27 Oct 2017 16:56:38 GMT

Hello,

I created a custom search command that queries an external service and returns a set of results using the v2 API (GeneratingCommand). This works perfectly on a standalone box, but when I put it onto a cluster, it appears to run but not display any results. Is there anything I could be missing that would cause this to occur?

Some details:

The cluster consists of a single standalone search node that is connected to 6 indexers. They are all part of the same cluster. This search node, however, is separate from the others for development purposes.

I can see that the search command runs on all of the index nodes and I can see that it is distributed to them. I have logging enabled, so I can see that it is getting results, but I just don't see the results showing up.

My command.conf looks similar to this:

[command]
filename = command.py
chunked = true
passauth = true
requires_srinfo = true
enableheader = true
stderr_dest = true

Thanks.

Edit 1: I decided to poke around a bit more and I can see that in the metrics.log file, I can see the command I'm trying to execute with the suffix "-too_small". Does this mean that the command simply isn't generating enough data? It's never going to get much, but it is necessary. Is there an option I can set to override this?

Re: Custom Search Command Fails to Return Results

DalJeanis — Fri, 27 Oct 2017 22:46:25 GMT

Have you verified that it is not running out of time or memory on the search head?

Re: Custom Search Command Fails to Return Results

kamlesh_vaghela — Sun, 29 Oct 2017 10:16:14 GMT

1) Can you please add below property and try again?

[command]
type = python

2) Have you checked the error log in splunkd.log and python.log??
3) Is that any error in search job ??

Thanks

Re: Custom Search Command Fails to Return Results

caseyra — Mon, 30 Oct 2017 12:14:19 GMT

I haven't seen any indication that it is running out of memory.

Re: Custom Search Command Fails to Return Results

caseyra — Mon, 30 Oct 2017 12:37:06 GMT

Hello,

No change using that property.
No errors in splunkd.log or python.log
If I look at the logs for the search job, the only "errors" I see are messages to STDERR that show the records I'm expecting to see in the search results, but that's it. I don't know why this occurs as I have it logging to a log file via the Python logging library.

Now, this brings me to something strange that I'm seeing. I log the fact that the the command is running 5 times, but I only see results coming back 4 times. Could the fact I'm not getting results back from every run be causing this? Why is this being executed multiple times?

Re: Custom Search Command Fails to Return Results

kamlesh_vaghela — Mon, 30 Oct 2017 16:21:02 GMT

Hi
yeah, strange. Conf file looks ok. Can you please provide sample code of your python file?

Re: Custom Search Command Fails to Return Results

caseyra — Mon, 30 Oct 2017 16:53:32 GMT

Sure. Here's basically what the code looks like minus imports and some logging config:

logger = logging.getLogger()

@Configuration
class GetRemoteData(GeneratingCommand):
    _defaultConfig = "/path/to/config.ini"
    _config = ConfigParser.RawConfigParser()
    _config.read(_defaultConfig)

    def generate(self):
        logger.info("Running GetRemoteData")
        try:
            md = self._metadata.searchinfo
            service = RemoteService(params)
            results = service.getData(str(md.username))
            if results == []:
                raise ValueError("No results could be found for the user: {}".format(md.username))

            for val in results:
                logger.info("Value: {}".format(val))
                parts = val.split(':')
                yield {'Name': parts[1], 'ID': parts[0]}
        except Exception as e:
            logger.exception(e)
            raise(e)

if __name__ == '__main__':
    dispatch(GetRemoteData, sys.argv, sys.stdin, sys.stdout, __name__)

Forgive me if my Python isn't quite right. I'm still fairly new to it.

Re: Custom Search Command Fails to Return Results

peterchenadded — Tue, 31 Oct 2017 12:52:37 GMT

You probably want to set local=true in commands.conf so it only runs once on your search head.

If still not work what's the message you get in stderr?

Re: Custom Search Command Fails to Return Results

caseyra — Tue, 31 Oct 2017 16:12:50 GMT

Unfortunately, this didn't work. As for errors, I'm not getting any. The only thing I see coming from stderr in the search log are messages from my logger.

Re: Custom Search Command Fails to Return Results

kchamplin_splun — Tue, 31 Oct 2017 17:30:34 GMT

Are you using the Splunk SDK as part of your custom search command? If so, what version of the SDK?

Re: Custom Search Command Fails to Return Results

caseyra — Tue, 31 Oct 2017 17:40:36 GMT

Yes, I am. Looks like I have 1.6.0-py2.7.

Re: Custom Search Command Fails to Return Results

svasani_splunk — Tue, 29 Sep 2020 16:35:54 GMT

change stderr_dest = log from strderr_dest=true in commands.conf and post the error if there are any.

Re: Custom Search Command Fails to Return Results

caseyra — Tue, 31 Oct 2017 17:51:02 GMT

Done, but I'm not seeing any errors.

Re: Custom Search Command Fails to Return Results

svasani_splunk — Tue, 31 Oct 2017 17:52:12 GMT

Did you restart splunk?

Re: Custom Search Command Fails to Return Results

caseyra — Tue, 31 Oct 2017 18:43:38 GMT

I was just doing the debug/refresh, but I did restart it was well. No change.

Re: Custom Search Command Fails to Return Results

peterchenadded — Tue, 31 Oct 2017 20:42:33 GMT

Have you tried setting local = true in your command.py?

Your first step should be making sure it runs just on search head once.

Re: Custom Search Command Fails to Return Results

David — Wed, 01 Nov 2017 01:32:30 GMT

Re: Custom Search Command Fails to Return Results

kchamplin_splun — Wed, 01 Nov 2017 02:38:36 GMT

You may want to try using the develop branch of the python SDK for now. There's a known bug for the generating command library and SCP2. You will still need to set distributed to false in your @configuration decorator, ex:
@Configuration(distributed=False

For the develop branch go here:
https://github.com/splunk/splunk-sdk-python/tree/develop

Reference for the issue:
https://github.com/splunk/splunk-sdk-python/pull/182/commits/edd5d1f2ddf1ab36a253588a7cdd472775c149c6

Re: Custom Search Command Fails to Return Results

caseyra — Wed, 01 Nov 2017 16:30:37 GMT

Yes, I did try that, but it didn't work.

Re: Custom Search Command Fails to Return Results

caseyra — Wed, 01 Nov 2017 16:32:57 GMT

Thanks. I didn't get a copy of the SDK yet, but I did add the distributed setting (True for now based on the bug) and it's working properly now. If I can, I'll get the version from the develop branch otherwise I'll wait until a new version is released.