Splunk CSC - Output Debug Logs to Search Log (Not as Log Level: ERROR)

[jocobwknight]

I'm working on a Splunk CSC and I've found it really helpful to output logs to the search log with:

print('Whatever I want', file=sys.stderr)

Which appears in the search log as:

10-07-2022 23:47:19.915 ERROR ChunkedExternProcessor [10374 ChunkedExternProcessorStderrLogger] - stderr: Whatever I want

 But that's a very beefy (even misleading) preamble. So my question is: Can I control the output that gets displayed in the search log?

I'm assuming there's some file handle somewhere I can write to and I would love to get a hold of it! It's obviously not sys.stdout because that's what the actual event data gets transmitted to.

Thanks!

[jocobwknight]

If anyone is like me and would be perfectly happy with a very ugly, shameless solution, then they can manipulate the page rather than Splunk.

For example, I use a Chrome add-on called "Custom JavaScript for Websites 2." It lets me save JavaScript that gets run when a page-pattern loads. The script I'm running is this:

function cleanCustom() {
    window.cleanCustomGiveUpOn--;
    if (window.cleanCustomGiveUpOn <= 0) {
        clearInterval(window.cleanCustomInterval);
        return;
    }
    var log_text = document.getElementsByTagName("pre");
    if (log_text.length == 0) { return; }
    console.log("cleaning custom...");
	var log_text = log_text[0];
	var log_lines = log_text.innerHTML.split('\n');
	for (var i = 0; i < log_lines.length; i++) {
		var line = log_lines[i];
		if (line.includes("ChunkedExternProcessor")) {
			line = line.replace("ERROR ChunkedExternProcessor ", "");
			line = line.replace(" ChunkedExternProcessorStderrLogger] - stderr:", "]");
			log_lines[i] = line;
		}
	}
	log_text.innerHTML = log_lines.join("\n");
    clearInterval(window.cleanCustomInterval);
}

window.cleanCustomGiveUpOn = 30;
window.cleanCustomInterval = setInterval(cleanCustom, 500);

It looks for the pre that contains the search log for 15 seconds, and upon finding it will scrub out the stuff I don't want.

Not a pretty solution, but an ugly solution for ugly logs seems oddly symmetrical.

[jocobwknight]

That code stinks, it disables the search bar in a normal Splunk search. I had to modify it:

function cleanCustom() {
    window.cleanCustomGiveUpOn--;
    if (window.cleanCustomGiveUpOn <= 0) {
        clearInterval(window.cleanCustomInterval);
        return;
    }
    var log_text = document.getElementsByTagName("pre");
    if (log_text.length == 0) { return; }
    console.log("cleaning custom...");
	var log_text = log_text[0];
	var log_lines = log_text.innerHTML.split('\n');
	var pointless = true;
	for (var i = 0; i < log_lines.length; i++) {
		var line = log_lines[i];
		if (line.includes("ChunkedExternProcessor")) {
			line = line.replace("ERROR ChunkedExternProcessor ", "");
			line = line.replace(" ChunkedExternProcessorStderrLogger] - stderr:", "]");
			log_lines[i] = line;
			pointless = false;
		}
	}
	if (!pointless) {
	  log_text.innerHTML = log_lines.join("\n");
	}
  clearInterval(window.cleanCustomInterval);
}

window.cleanCustomGiveUpOn = 30;
window.cleanCustomInterval = setInterval(cleanCustom, 500);

And again, this entire thing is only for aesthetics and only for looking at live debug logs from csc development. I don't know why I care so much about something that probably less than millionth of the human population will ever even come close to sympathizing with.

[jocobwknight]

Unfortunately it would seem like this isn't possible after all. I tested @richgalloway's suggestion by following the Splunk development guide:

import logging

logging.root
logging.root.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - MyCommand [%(levelname)s] %(message)s')
handler = logging.StreamHandler(stream=sys.stderr)
handler.setFormatter(formatter)
logging.root.addHandler(handler)

# CSC stuff...
logging.info('Whatever I want')

And my output was a humorous surprise:

10-08-2022 00:38:32.400 ERROR ChunkedExternProcessor [16402 ChunkedExternProcessorStderrLogger] - stderr: 2022-10-07 16:38:32,400 - MyCommand [INFO] Whatever I want

But I can at least put my mind at ease now that I've at least tried to make it better. 

[richgalloway · Accepted solution]

You can control the log message by using a log formatter.  See https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/#Set...

[jocobwknight]

Although this is for scripted inputs rather than custom search commands, I'm really hoping Splunk's engine was built to handle both processes in the same way. It seems likely because they've also set it up to log stderr, so I'm optimistic.

(And if it doesn't work, then I can at least assume it's hopeless and stop worrying about the format)

[jocobwknight]

This looks like what I need, thanks!