I have syslogs coming into Splunk that need some cleaning up - it's essentially JSON with a few extra characters here and there (but enough to be improperly formatted). I'd really like to be able to use KV_MODE = json to auto extract fields, but those additional characters prevent this from happening. So I wrote a few SEDCMDs to remove those additional characters and applied the following stanzas to a new sourcetype: However, in our distributed Splunk Cloud environment, these SEDCMDs are not working. There are no errors in the _internal index pertaining to this sourcetype, and I can tell the sourcetype is applying because any key/value pairs in the data that pop up before the extra characters are automatically extracted at search-time as expected (so at least I know the KV_MODE stanza is trying to work). Because the SEDCMDs are not removing the extra characters, the other fields are not being auto-extracted. In my all-in-one test environment, the SEDCMDs work perfectly alongside KV_MODE to clean up the data and pull out the fields. I can't quite determine why it isn't working in Cloud - the syslog servers forwarding this data have Universal Forwarders so I understand why the sourcetype isn't applying at that level... but this sourcetype should be hitting the indexers and applied there, no? What am I missing?
... View more