Hidden syslog configuration on heavy forwarders

[Dominic32]

My organization has a handful of heavy forwarders that were configured to listen to syslog sources through udp://514. This was set up by a 3rd party, and now we are trying to understand the configuration. Searching the heavy forwarders' /etc/* recursively for "514", "tcp", "udp", "syslog", or "SC4S" returns no relevant results.

We know syslog is working, because we have multiple sources that are pointed at the heavy forwarders using udp over port 514 and their data is being indexed.

Curiously, when a new syslog source is pointed at the HFs, a new index with a random name pops up in our LastChanceIndex. We have no idea how any of this is configured - the index selection, or the syslog listener. We usually create an index that matches the name given, since we've never been able to find the config to set it manually.

Any suggestions on how syslog might be set up, or what else I could try searching for?

[Dominic32 · Accepted solution]

SC4S was running in a docker container through Podman. Mystery solved!

Splunk Setup - Splunk Connect for Syslog

[mattymo]

Hi! you'll want to poke thru the docs on splunk config files: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/AboutConfigurationFiles

But tldr is I would use "btool" - https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Usebtooltotroubleshootconfigurati...

and you'll want to go hunting for "inputs.conf" - this is where your spunk instances would be taking the data in, then comb thru props.conf - where the sourcetypes and event parsing/transformation/routing happens..

It is also common to have splunk co-located with a syslog listener who puts logs down that we pick up. So a quick `ss -tulpn` or `netstat -tulpn` will show what ports, if any, are open on your Heavy Forwarders. 

so getting good with btool or reviewing Inputs and sourcetypes in your splunk ui will be key

[Dominic32]

As stated I've tried searching through all files within /etc/* (including all .conf files) for the following: "514", "tcp", "udp", "syslog", or "SC4S". I get no results. You mentioned I should check inputs.conf, but I've already done this and found nothing - could you elaborate on what exactly I should be searching for? Are there additional keywords I should try?
I confirmed that the Heavy Forwarders are listening on port 514. Syslog is working... I just don't see how it's configured. 

Edit: I also want to ask - what could btool find that a sudo grep search wouldn't have located?