Splunk Search

Custom Search Command Fails to Return Results

caseyra
Explorer

Hello,

I created a custom search command that queries an external service and returns a set of results using the v2 API (GeneratingCommand). This works perfectly on a standalone box, but when I put it onto a cluster, it appears to run but not display any results. Is there anything I could be missing that would cause this to occur?

Some details:

The cluster consists of a single standalone search node that is connected to 6 indexers. They are all part of the same cluster. This search node, however, is separate from the others for development purposes.

I can see that the search command runs on all of the index nodes and I can see that it is distributed to them. I have logging enabled, so I can see that it is getting results, but I just don't see the results showing up.

My command.conf looks similar to this:

[command]
filename = command.py
chunked = true
passauth = true
requires_srinfo = true
enableheader = true
stderr_dest = true

Thanks.

Edit 1: I decided to poke around a bit more and I can see that in the metrics.log file, I can see the command I'm trying to execute with the suffix "-too_small". Does this mean that the command simply isn't generating enough data? It's never going to get much, but it is necessary. Is there an option I can set to override this?

1 Solution

kchamplin_splun
Splunk Employee
Splunk Employee

Are you using the Splunk SDK as part of your custom search command? If so, what version of the SDK?

View solution in original post

David
Splunk Employee
Splunk Employee

Have you tried adding | localop before your command? (You can add | localop before anything, even | localop | search index=* | head 1000.) You shouldn't need to, but it might be a viable workaround.

caseyra
Explorer

That worked, but @kchamplin's answer works better in our case.

Thanks!

David
Splunk Employee
Splunk Employee

I'll come in second place to @kchamplin any day! Glad you got it working.

0 Karma

svasani_splunk
Splunk Employee
Splunk Employee

change stderr_dest = log from strderr_dest=true in commands.conf and post the error if there are any.

0 Karma

caseyra
Explorer

Done, but I'm not seeing any errors.

0 Karma

svasani_splunk
Splunk Employee
Splunk Employee

Did you restart splunk?

0 Karma

caseyra
Explorer

I was just doing the debug/refresh, but I did restart it was well. No change.

0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

Are you using the Splunk SDK as part of your custom search command? If so, what version of the SDK?

caseyra
Explorer

Yes, I am. Looks like I have 1.6.0-py2.7.

0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

You may want to try using the develop branch of the python SDK for now. There's a known bug for the generating command library and SCP2. You will still need to set distributed to false in your @configuration decorator, ex:
@Configuration(distributed=False

For the develop branch go here:
https://github.com/splunk/splunk-sdk-python/tree/develop

Reference for the issue:
https://github.com/splunk/splunk-sdk-python/pull/182/commits/edd5d1f2ddf1ab36a253588a7cdd472775c149c...

0 Karma

caseyra
Explorer

Thanks. I didn't get a copy of the SDK yet, but I did add the distributed setting (True for now based on the bug) and it's working properly now. If I can, I'll get the version from the develop branch otherwise I'll wait until a new version is released.

0 Karma

peterchenadded
Path Finder

You probably want to set local=true in commands.conf so it only runs once on your search head.

If still not work what's the message you get in stderr?

0 Karma

caseyra
Explorer

Unfortunately, this didn't work. As for errors, I'm not getting any. The only thing I see coming from stderr in the search log are messages from my logger.

0 Karma

peterchenadded
Path Finder

Have you tried setting local = true in your command.py?

Your first step should be making sure it runs just on search head once.

0 Karma

caseyra
Explorer

Yes, I did try that, but it didn't work.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

HI

1) Can you please add below property and try again?

[command]
type = python

2) Have you checked the error log in splunkd.log and python.log??
3) Is that any error in search job ??

Thanks

0 Karma

caseyra
Explorer

Hello,

  1. No change using that property.
  2. No errors in splunkd.log or python.log
  3. If I look at the logs for the search job, the only "errors" I see are messages to STDERR that show the records I'm expecting to see in the search results, but that's it. I don't know why this occurs as I have it logging to a log file via the Python logging library.

Now, this brings me to something strange that I'm seeing. I log the fact that the the command is running 5 times, but I only see results coming back 4 times. Could the fact I'm not getting results back from every run be causing this? Why is this being executed multiple times?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi
yeah, strange. Conf file looks ok. Can you please provide sample code of your python file?

0 Karma

caseyra
Explorer

Sure. Here's basically what the code looks like minus imports and some logging config:

logger = logging.getLogger()

@Configuration
class GetRemoteData(GeneratingCommand):
    _defaultConfig = "/path/to/config.ini"
    _config = ConfigParser.RawConfigParser()
    _config.read(_defaultConfig)

    def generate(self):
        logger.info("Running GetRemoteData")
        try:
            md = self._metadata.searchinfo
            service = RemoteService(params)
            results = service.getData(str(md.username))
            if results == []:
                raise ValueError("No results could be found for the user: {}".format(md.username))

            for val in results:
                logger.info("Value: {}".format(val))
                parts = val.split(':')
                yield {'Name': parts[1], 'ID': parts[0]}
        except Exception as e:
            logger.exception(e)
            raise(e)

if __name__ == '__main__':
    dispatch(GetRemoteData, sys.argv, sys.stdin, sys.stdout, __name__)

Forgive me if my Python isn't quite right. I'm still fairly new to it.

0 Karma

DalJeanis
Legend

Have you verified that it is not running out of time or memory on the search head?

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...