Our developers send TRACE and DEBUG logs in massive quantities. They don't need them on 24/7. The test systems are not in developer control, so they can't easily control logging levels. Submit a service desk ticket, wait wait wait. No good.
So my solution is to send TRACE and DEBUG to a different port on the indexers. I plan to briefly enable the ports on demand. Something like a 15 minute window before they get turned off again. Setting up a simple web-based scripty for this would be easy... if inputs were controllable from the CLI. Based on CLI help, this isn't possible, leaving me with web scraping scripts. Yeck. Or iptables I suppose.
Anyone else in this predicament? Other options?
Inputs can be controlled from the CLI:
# splunk add tcp 8514 -sourcetype syslog -index os
Yes, I was aware of that. I was really looking for the CLI equivalent of the enable/disable switch available in the GUI. I guess the more brutish add/delete would work. I'll need to research the add command more to see if all the input settings I use are available vi CLI. Thanks.