topic Re: transforms and props.conf splitting sourcetype not matching in Getting Data In

transforms and props.conf splitting sourcetype not matching

dstark — Tue, 29 Sep 2020 10:48:26 GMT

I'm trying to follow the pattern of matching a string and transforming the event into a new sourcetype. I'm using a sourcetype for syslog defined in inputs.conf; it is being read from logs.

/var/log/syslog contains events matching string "sqsd" that I would like to rewrite to a new sqsd sourcetype

I've found multiple answers posts about this topic but can't seem to determine why I'm not getting any data as the transformed sourcetype. Originally, I thought the problem was in my REGEX in transforms.conf, but if I set it to .* or remove it completely I still don't get results.

inputs.conf

[monitor:///var/log/syslog]
sourcetype=syslog
index = test
ignoreOlderThan = 24h

props.conf

[syslog]
TRANSFORMS-syslog = set_sqsd_sourcetype

transforms.conf

[set_sqsd_sourcetype]
REGEX = sqsd
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::sqsd

splunkd.log does not show any errors, so I don't think my conf files are invalid. Right now I am seeing all events as sourcetype=syslog.

Re: transforms and props.conf splitting sourcetype not matching

Nicolo_Figiani — Tue, 29 Sep 2020 10:48:37 GMT

Hi,
try using the btool command to troubleshoot your issue. On the Splunk instance that should do the sourcetype rewrite (keep in mind that this happens at index time), do the following from the CLI:

splunk cmd btool props list syslog --debug

Thie above will lists all the specs of the "syslog" sourcetype. Make sure you're seeing the transformation you've defined "set_sqsd_sourcetype".

Do the same for the actual transformation:

splunk cmd btool transforms list set_sqsd_sourcetype --debug

Once again, make sure, the specs you have defined are listed.

Very likely, the configuration is not applied correctly or it is overridden by something else..the btool command should give you the exact snapshot of which configuration is enabled and it helped me hundreds of time (many times for the issue of mis-overwritten sourcetypes at index time).

Hope it help,
N

Re: transforms and props.conf splitting sourcetype not matching

dstark — Thu, 01 Sep 2016 15:39:43 GMT

As far as I can tell btool is showing exactly what I expect. There's a unique TRANSFORMS for syslog in /etc/system/local and the default TRANFSORMS from /etc/system/default:

/opt/splunkforwarder/etc/system/default/props.conf TRANSFORMS = syslog-host
/opt/splunkforwarder/etc/system/local/props.conf TRANSFORMS-syslog = set_sqsd_sourcetype

Re: transforms and props.conf splitting sourcetype not matching

somesoni2 — Thu, 01 Sep 2016 15:52:31 GMT

Are you using Universal forwarder OR heavy forwarder to collect the data? If it's UF then your props.conf and transforms.conf should be on Indexer/next heavy forwarder in the flow. If it's HF then your props and transforms should be on Heavy Forwarder.

Re: transforms and props.conf splitting sourcetype not matching

dstark — Thu, 01 Sep 2016 17:18:08 GMT

Ah....I'm using a universal forwarder and Splunk Cloud. I found this link (search "im-struggling-with-how-i-should-be-doing-inputs-and-also-props-transforms-etc-stuff-within-splunk-cloud" as I can't post links) which explains how to modify props and transforms.conf using the GUI in Splunk Cloud. I'll give it a shot and post screenshots if I can get it working.

Thanks somesoni2!

Re: transforms and props.conf splitting sourcetype not matching

Nicolo_Figiani — Thu, 01 Sep 2016 17:18:49 GMT

I guess we know why now. You're configuring the source type override on your universal forwarder. However, it is not a full Splunk instance and It has not parsing capabilities. You should configure everything on the first FULL Splunk instance ( HFW or IDX ) of your chain.
Your configuration looks ok so, just place it on the right Splunk instance and everything should work.

In addition, the entry regarding the props.conf has a different name "syslog-host" whereas you're looking for "syslog" and it refers to /system/default/props.conf. This is not you're configuration so, be sure to deploy your props config as well..

Hope it helps and let me know..

Re: transforms and props.conf splitting sourcetype not matching

dstark — Fri, 02 Sep 2016 12:35:22 GMT

I've verified that I should be changing the Splunk Cloud instance with the link above. However, I'm still not seeing results with sourcetype=sqsd.

I'm not able to set the DEST_KEY = MetaData:Sourcetype using the GUI, maybe this is the problem?
http://imgur.com/a/jFy8B

Re: transforms and props.conf splitting sourcetype not matching

somesoni2 — Fri, 02 Sep 2016 13:29:35 GMT

That's probably it. You need Splunk's held to get this props and transforms deployed.

Re: transforms and props.conf splitting sourcetype not matching

dstark — Fri, 02 Sep 2016 14:52:52 GMT

I had to contact Splunk support to apply these configs to my Splunk Cloud indexer. Thank you Nicolo_Figiani and somesoni2 for your help!