How to configure proper logging for custom search command

[tommasoscarpa1]

Hi all,

I am trying to develop a custom command.
The custom command works as expected and now I am working to setup proper logging, but I can't seem to be able to make the python script log anything or I'm looking in the wrong place.
I built it following what's written here: Create a custom search command | Documentation | Splunk Developer Program

Here's a quick python code example:

#!/usr/bin/env python
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import os, sys, requests, json

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators, splunklib_logger as logger


@Configuration()
class TestCustomCMD(StreamingCommand):

    def getFieldValue(self, field_name, record):
        return record[field_name] if field_name in record else ""

    def writeFieldValue(self, field_name, field_value, record):
        record[field_name] = field_value

    def stream(self, records):
        for record in records:
            self.writeFieldValue("TEST FIELD", "TEST CUSTOM COMMAND", record)
            logger.fatal("FATAL logging example")
            logger.error("ERROR logging example")
            logger.warning("WARNING logging example")
            logger.info("INFO logging example")
            yield record

dispatch(TestCustomCMD, sys.argv, sys.stdin, sys.stdout, __name__)

command.conf:

[testcustcmd]
filename = test_custom_command.py
python.version = python3
chunked = true

and search to test:

| makeresults count=2 
| testcustcmd

The search completes correctly and returns this:

However, I don't find the logged lines anywhere.
On my Splunk server I ran this:

grep -rni "logging example" "/opt/splunk/var/log/splunk/"

But the result is empty.

Can you help me understand what I am doing wrong here?

Thank you in advance,

Tommaso

[VatsalJagani · Accepted solution]

@tommasoscarpa1 - I prefer to put the logs in a custom log files always. Which will come into custom sourcetype as well on Splunk.

Inside _internal index of Splunk so no Splunk license will be used for that as well.

Here is the reference code for some other App's custom command.

* https://github.com/CrossRealms/Splunk-App-Auto-Update-MaxMind-Database/blob/master/bin/maxmind_db_up... (Custom search command)

* https://github.com/CrossRealms/Splunk-App-Auto-Update-MaxMind-Database/blob/master/bin/logger_manage... (Custom logger manager file)

* https://github.com/CrossRealms/Splunk-App-Auto-Update-MaxMind-Database/blob/master/default/props.con... (For assigning the custom sourcetype)

You can search the data in the index=_internal sourcetype=<sourcetype-you-asssign-in-props.conf>

Also, just FYI, you can use the same the same code for custom rest-endpoints, python modular inputs, or anything else.

I hope this helps!!! Kindly upload the answer if you like the solution.

[tommasoscarpa1]

Thank you @VatsalJagani !!

This is so much helpful. A lot of examples to do exactly what I wanted. Works like a charm now.

[LAME-Creations]

This is not my area of expertise, I have done a little bit with the custom commands to know you are on the right path, but looking at your code is not making anything obvious pop up, but I do know the most common issues to ingesting logs that are created by firing off splunk actions (which is what you are doing) is rights.

I have had two issues pop up when doing what you are doing, one was easy to fix, the other was one that resulted in a lot of head banging and frustration but I found a workaround that worked in our environment. 

The first thing you want to validate is that logs are actually being created.  I am sure you are doing this, but as a person who has done everything wrong in Splunk, I have actually tried to troubleshoot why my logs are not coming in only to find that no physical logs actually exist. 

After you verify that I recommend putting a "test" log in the exact same site as your python logs.  Can you ingest the "test" log.  If you can't, you know that it is probably related to rights.  

If you can have splunk ingest the "test" logs you may be in the glitchy world that no one has ever truly explained to me, but it relates to where you write the logs.  Logs that are dynamically created in Splunk (which is what you are doing) for some reason could not be read in certain locations on the disk, even though it could read the "test" logs.  

So ultimately I found that I had to change the location of the logs being written to another location on disk and then pull the logs from there.  You are making me have to think back on painful traumatic times, but I think anytime I tried to write the logs inside the directory of the app that was built to make the custom command, it would not read.  But when I moved it to /var/logs it worked.  I hope this is not the problem you run into, and hopefully the pain and trauma I suffered from has been fixed or there was some other underlying issue that I was dealing with and you will never have to experience this, but it was enough to make me still wake up in the middle of the night with nightmares 🙂 

But hopefully everything can be attributed to Splunk not being able to read the location of your python logs, you change the permissions and everything works.  If not, hopefully someone has a silver bullet in these answer forums, and if that doesn't work just try different locations on your OS and see if they work (I know this cannot be the true answer, but it was what ultimately worked for me)

[tommasoscarpa1]

Hi @LAME-Creations 

Thank you for your answer! That's actually one of the open points. I have no idea if my logs are actually being written or not. I assumed that, if they are logged, they would have been logged in the standard Splunk log folder /opt/splunk/var/log/splunk, but I might be wrong here.

Do you have any suggestion that could help me understand if I am simply looking in the wrong place?

[LAME-Creations]

I have only dabbled in this area, but I am pretty sure (not going to place any bets in Las Vegas on it though) that your python code is going to determine where you write the logs.  It would be awesome if it does write to the splunk logs area, but I am pretty sure it does not.  I don't think you specified a location for where you are writing the logs, so my guess is the python script is going to try to write the logs to the same location as your .py script.  Now someone could come along and tell me I am completely wrong, and that is ok, because as I said, this is something I have only done a couple times and it was years ago.

If I were going to troubleshoot to find out where the logs are being written, the first thing I would do is spin up my local / dev instance of Splunk.  I really encourage anyone doing development, especially on something as complicated as what you are doing, that you have a dev instance - whether that be on a spare computer or laptop or spin up a local vm, or whatever, but it is really difficult to troubleshoot on a production environment and from an app dev perspective this is also a good practice to have a test environment. 

On this test box, you should have access to the command line.  Put your code on that test box and then look and see where the logs are being written.  

Sorry I don't have any silver bullet, but my guess is that the log file is "trying" to be written to the same location as your python script, and that means your inputs.conf needs to be pointed there as well or it won't be able to grab it.  (I don't recommend writing your logs to your scripts folder so you probably will want to change its location in the python script)