Why is Splunk Cloud not stripping syslog headers?

[Falko]

I have a Splunk Enterprise installation and a Splunk Cloud stack. I want to migrate logging from Enterprise to Splunk Cloud.

My EC2-Machines have an old Splunk Forwarder installed and are forwarding to the Splunk Enterprise instance. The log file I'm ingesting is JSON format, but each line contains a SYSLOG prefix. This prefix seems to be stripped out by Splunk Enterprise from what I can tell. The sourcetype of the log is a custom type which is NOT explicitly defined on the Splunk Enteprise server. Since the log is JSON, no explicit field extraction is needed. the log events are just JSON messages and are properly extracted.

Now I've changed the outputs.conf on the EC2 machine to send the logs to Splunk Cloud. Nothing else changed. Splunk Cloud indexes the events and the SYSLOG header shows up in Splunk Cloud. Thats why the event doesn't seem to be recognized as JSON and field extraction is not working.

Any idea how to tell Splunk Cloud to Strip the SYSLOG header from these events? And especially... why this was working apparently automatically on the Splunk Enterprise side?

Both Splunkn installations have the Splunk Add-On for Unix installed, which seems to contain configuration for stripping SYSLOG headers from events.

But I don't understand yet, how these come into action.

My inputs.conf:

[monitor:///var/log/slbs/tors_access.log]
disabled = false
blacklist = \.(gz|bz2|z|zip)$
sourcetype = tors_access
index = torsindex

There is no props.conf or transform.conf on the EC2 machine with the Splunk forwarder for this app (and if so, that should have kicked in when I change the output to Splunk Cloud).

[richgalloway]

Please share the [tos_access] props.conf stanza from both the Enterprise and Cloud installations.

Is it possible the data goes through a heavy forwarder before reach Splunk Enterprise and by redirecting the EC2 instance to Splunk Cloud the HF is skipped and any transforms done by it are not applied?

[Falko]

on the EC2 instances there are a few props.conf in the apps/<appname>/default dir. But etc/system/local doesn't have a props.conf

From what I can tell, most of the apps are disabled and their props.conf shouldn't apply to my source type.

I can't access the props.conf of the Splunk Cloud stack. But I can tell from the admin-UI that there is nothing configured for my sourcetype either. It's a custom sourcetype

I'll check regarding the heavy forwarder. I'm not aware of those. But I can't rule that out either.

Thanks for the help!

[richgalloway]

If you don't know which props.conf file has the sourcetype, use btool to find it.

splunk btool --debug props list tos_access

That will show all of the settings that apply to the sourcetype.  The --debug option adds the name of the file where the setting is made.

If you don't see the tos_access sourcetype in the Splunk Cloud UI (Settings->Source types) then there are no settings for it, which might very well explain the difference in behavior.

[Falko]

Thanks again!

the output didn't return any config for this sourcetype on the EC2 node with the Universal Forwarder. As I expected, the forwarding machine is not modifying the log lines then.

Unfortunately I don't have access to the Splunk Servers, so can't run the command on the machines. But I checked the Admin UI for both environments. The sourcetype is not defined on either stack. Which makes me believe they should not have any transformation configured on the server side, correct?

[richgalloway]

The lack of props for the sourcetype is not a good thing.  It means Splunk is guessing about how to interpret the data and may be guessing wrong.  Perhaps Splunk Cloud makes different assumptions about the data than Splunk Enterprise does.

Create an app with good props.conf settings for the sourcetype and install the app in both environments.  That should fix it.