This is not my area of expertise, I have done a little bit with the custom commands to know you are on the right path, but looking at your code is not making anything obvious pop up, but I do know th...
See more...
This is not my area of expertise, I have done a little bit with the custom commands to know you are on the right path, but looking at your code is not making anything obvious pop up, but I do know the most common issues to ingesting logs that are created by firing off splunk actions (which is what you are doing) is rights. I have had two issues pop up when doing what you are doing, one was easy to fix, the other was one that resulted in a lot of head banging and frustration but I found a workaround that worked in our environment. The first thing you want to validate is that logs are actually being created. I am sure you are doing this, but as a person who has done everything wrong in Splunk, I have actually tried to troubleshoot why my logs are not coming in only to find that no physical logs actually exist. After you verify that I recommend putting a "test" log in the exact same site as your python logs. Can you ingest the "test" log. If you can't, you know that it is probably related to rights. If you can have splunk ingest the "test" logs you may be in the glitchy world that no one has ever truly explained to me, but it relates to where you write the logs. Logs that are dynamically created in Splunk (which is what you are doing) for some reason could not be read in certain locations on the disk, even though it could read the "test" logs. So ultimately I found that I had to change the location of the logs being written to another location on disk and then pull the logs from there. You are making me have to think back on painful traumatic times, but I think anytime I tried to write the logs inside the directory of the app that was built to make the custom command, it would not read. But when I moved it to /var/logs it worked. I hope this is not the problem you run into, and hopefully the pain and trauma I suffered from has been fixed or there was some other underlying issue that I was dealing with and you will never have to experience this, but it was enough to make me still wake up in the middle of the night with nightmares But hopefully everything can be attributed to Splunk not being able to read the location of your python logs, you change the permissions and everything works. If not, hopefully someone has a silver bullet in these answer forums, and if that doesn't work just try different locations on your OS and see if they work (I know this cannot be the true answer, but it was what ultimately worked for me)