Getting Data In

Logging to Job Inspector from Custom Search Command


Hi folks.
I have a custom search command and I am using self.logger to log messages from the command. Please see my logging.conf attached.

# root is mandatory.
keys = root, CustomSearchCommand

keys = ConsoleHandler

keys = SimpleFormatter

level = INFO
handlers = ConsoleHandler

level = INFO
handlers = ConsoleHandler
# qualname is mandatory.
qualname = CustomSearchCommand
# propagate is disabled in order not to log same events twice.
propagate = 0

class = StreamHandler
# sys.stdout causes weird errors.
args = (sys.stderr,)
level = INFO
formatter = SimpleFormatter

format = %(asctime)s - %(process)s - %(name)s - %(levelname)s - %(message)s

First of all, I don't understand why Splunk doesn't allow us to use sys.stdout in ConsoleHandler, it just keeps failing with a very strange error message:

11-18-2019 16:49:38.169 ERROR ChunkedExternProcessor - Failed attempting to parse transport header: 2019-11-18 16:49:38,168 - 23427 - CustomSeachCommand - INFO - <ORIGINAL LOG MESSAGE>

At the same time, using sys.stderr in ConsoleHandler works fine, though the output doesn't look good:

11-18-2019 16:40:46.346 ERROR ChunkedExternProcessor - stderr: 2019-11-18 16:40:46,345 - 22025 - CustomSearchCommand - INFO - <ORIGINAL LOG MESSAGE>

So why does Splunk always log messages from child processes as errors? It's clear that each final log record consists of two parts: 11-18-2019 16:40:46.346 ERROR ChunkedExternProcessor - stderr: (from the parent process, I guess) and 2019-11-18 16:40:46,345 - 22025 - CustomSearchCommand - INFO - <ORIGINAL LOG MESSAGE> (from the child process, executing my custom search command). By the way, I'm inheriting my own command from StreamingCommand and using chunked = true.

Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...