Splunk Dev

Splunk CSC - Output Debug Logs to Search Log (Not as Log Level: ERROR)

jocobwknight
Explorer

I'm working on a Splunk CSC and I've found it really helpful to output logs to the search log with:

 

print('Whatever I want', file=sys.stderr)

 

Which appears in the search log as:

10-07-2022 23:47:19.915 ERROR ChunkedExternProcessor [10374 ChunkedExternProcessorStderrLogger] - stderr: Whatever I want

 But that's a very beefy (even misleading) preamble. So my question is: Can I control the output that gets displayed in the search log?

I'm assuming there's some file handle somewhere I can write to and I would love to get a hold of it! It's obviously not sys.stdout because that's what the actual event data gets transmitted to.

Thanks!

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You can control the log message by using a log formatter.  See https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/#Set...

---
If this reply helps you, Karma would be appreciated.

View solution in original post

jocobwknight
Explorer

If anyone is like me and would be perfectly happy with a very ugly, shameless solution, then they can manipulate the page rather than Splunk.

For example, I use a Chrome add-on called "Custom JavaScript for Websites 2." It lets me save JavaScript that gets run when a page-pattern loads. The script I'm running is this:

function cleanCustom() {
    window.cleanCustomGiveUpOn--;
    if (window.cleanCustomGiveUpOn <= 0) {
        clearInterval(window.cleanCustomInterval);
        return;
    }
    var log_text = document.getElementsByTagName("pre");
    if (log_text.length == 0) { return; }
    console.log("cleaning custom...");
	var log_text = log_text[0];
	var log_lines = log_text.innerHTML.split('\n');
	for (var i = 0; i < log_lines.length; i++) {
		var line = log_lines[i];
		if (line.includes("ChunkedExternProcessor")) {
			line = line.replace("ERROR ChunkedExternProcessor ", "");
			line = line.replace(" ChunkedExternProcessorStderrLogger] - stderr:", "]");
			log_lines[i] = line;
		}
	}
	log_text.innerHTML = log_lines.join("\n");
    clearInterval(window.cleanCustomInterval);
}

window.cleanCustomGiveUpOn = 30;
window.cleanCustomInterval = setInterval(cleanCustom, 500);

It looks for the pre that contains the search log for 15 seconds, and upon finding it will scrub out the stuff I don't want.

Not a pretty solution, but an ugly solution for ugly logs seems oddly symmetrical.

0 Karma

jocobwknight
Explorer

That code stinks, it disables the search bar in a normal Splunk search. I had to modify it:

function cleanCustom() {
    window.cleanCustomGiveUpOn--;
    if (window.cleanCustomGiveUpOn <= 0) {
        clearInterval(window.cleanCustomInterval);
        return;
    }
    var log_text = document.getElementsByTagName("pre");
    if (log_text.length == 0) { return; }
    console.log("cleaning custom...");
	var log_text = log_text[0];
	var log_lines = log_text.innerHTML.split('\n');
	var pointless = true;
	for (var i = 0; i < log_lines.length; i++) {
		var line = log_lines[i];
		if (line.includes("ChunkedExternProcessor")) {
			line = line.replace("ERROR ChunkedExternProcessor ", "");
			line = line.replace(" ChunkedExternProcessorStderrLogger] - stderr:", "]");
			log_lines[i] = line;
			pointless = false;
		}
	}
	if (!pointless) {
	  log_text.innerHTML = log_lines.join("\n");
	}
  clearInterval(window.cleanCustomInterval);
}

window.cleanCustomGiveUpOn = 30;
window.cleanCustomInterval = setInterval(cleanCustom, 500);

 

 

And again, this entire thing is only for aesthetics and only for looking at live debug logs from csc development. I don't know why I care so much about something that probably less than millionth of the human population will ever even come close to sympathizing with.

0 Karma

jocobwknight
Explorer

Unfortunately it would seem like this isn't possible after all. I tested @richgalloway's suggestion by following the Splunk development guide:

import logging

logging.root
logging.root.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - MyCommand [%(levelname)s] %(message)s')
handler = logging.StreamHandler(stream=sys.stderr)
handler.setFormatter(formatter)
logging.root.addHandler(handler)

# CSC stuff...
logging.info('Whatever I want')

 

And my output was a humorous surprise:

10-08-2022 00:38:32.400 ERROR ChunkedExternProcessor [16402 ChunkedExternProcessorStderrLogger] - stderr: 2022-10-07 16:38:32,400 - MyCommand [INFO] Whatever I want

But I can at least put my mind at ease now that I've at least tried to make it better. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You can control the log message by using a log formatter.  See https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/#Set...

---
If this reply helps you, Karma would be appreciated.

jocobwknight
Explorer

Although this is for scripted inputs rather than custom search commands, I'm really hoping Splunk's engine was built to handle both processes in the same way. It seems likely because they've also set it up to log stderr, so I'm optimistic.

(And if it doesn't work, then I can at least assume it's hopeless and stop worrying about the format)

0 Karma

jocobwknight
Explorer

This looks like what I need, thanks!

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...