Ok. If I understand you correctly, you are using UFs which send data directly to indexers. And those indexers index locally as well as send a copy to a syslog destination, right? And you're doing tha...
See more...
Ok. If I understand you correctly, you are using UFs which send data directly to indexers. And those indexers index locally as well as send a copy to a syslog destination, right? And you're doing that by defining transforms manipulating _SYSLOG_ROUTING on the indexers. Do I get it right? In this case, data already processed by other "full" Splunk Enterprise components (SHs, CM and so on) is _not_ processed by the indexers. tl;dr - You must create syslog outputs and transforms for Splunk-originating events on the source servers (SHs, CM...) as well. You might be able to try to address your problem with ingest actions but I'm no expert here. Longer explanation: Data in Splunk can be in one of three "states". Normally an input reads raw data. This raw data - if received on UF - is split into chunks and sent to an output as so-called "cooked data". This data is not yet split into separate events, it's not timestamped... It's just chunks of raw data along with a very basic set of metadata. If raw data from input or cooked data from UF is received by a "heavy" component (a full Splunk Enterprise instance, regardless of its role) it's getting parsed - the data is split into single events, timestamp is assigned to those events, indexed fields are extracted and so on. At this point we have data which is "cooked and parsed", often called just "parsed" for short. Depending on server's role that data might be indexed locally or sent to output(s). But if parsed data is received on an input it's not touched again except for earlier mentioned ingest actions. It's not reparsed again, no transforms are run on the data you receive in parsed form. So if you're receiving internal data from your Splunk servers, that data ihas already been parsed on the source Splunk server - any transforms you have defined on your indexers do not apply to this data.