SEDCMD not working in Splunk Cloud

[best-west]

I have syslogs coming into Splunk that need some cleaning up - it's essentially JSON with a few extra characters here and there (but enough to be improperly formatted). I'd really like to be able to use KV_MODE = json to auto extract fields, but those additional characters prevent this from happening. So I wrote a few SEDCMDs to remove those additional characters and applied the following stanzas to a new sourcetype:

However, in our distributed Splunk Cloud environment, these SEDCMDs are not working.

There are no errors in the _internal index pertaining to this sourcetype, and I can tell the sourcetype is applying because any key/value pairs in the data that pop up before the extra characters are automatically extracted at search-time as expected (so at least I know the KV_MODE stanza is trying to work). Because the SEDCMDs are not removing the extra characters, the other fields are not being auto-extracted.

In my all-in-one test environment, the SEDCMDs work perfectly alongside KV_MODE to clean up the data and pull out the fields. I can't quite determine why it isn't working in Cloud - the syslog servers forwarding this data have Universal Forwarders so I understand why the sourcetype isn't applying at that level... but this sourcetype should be hitting the indexers and applied there, no?

What am I missing? 

[PickleRick]

Key question here is, since you're saying it's syslog and you definitely not sending syslog straight to Cloud, what your ingestion process look like? Do you have any HFs on-prem?

[best-west]

Thank you for responding! Yes, it's coming from syslog server with UF installed going to Cloud. I unfortunately don't have any HFs available for use and setting up another one at this time is not an option for me.

[PickleRick]

The question was because if you had HF in front of your indexers, there's were your index-time props would be applied. Since you're using UF to push data to Cloud, you indeed need to push an app to the Cloud as @sainag_splunk wrote.

[best-west]

Replied to my own post. Derp. 🙂
Hi! Thank you so much for your response and explanation. It seems like maybe I have not properly deployed these to the indexing tier.

Forgive me for the beginner question, but I think the sourcetype I created already belongs to the 000-self-service app - is this what you meant by deploying the config using self service? Screenshot below (I didn't capture the full sourcetype name):

[sainag_splunk]

The key insight is that KV_MODE=json is applied at search-time on the Search Head, while SEDCMDs are part of the parsing pipeline (Typing / Regexreplacement) that must occur during indexing.

In Splunk Cloud, that should've done it we need make sure your sourcetype configuration with these SEDCMDs is properly deployed to the indexing tier, not just the search head (could use SEDCMDs on sh), since that's where the actual parsing/transformation of the data needs to happen.

Try to deploy your SEDCMD config using self service app and see if that makes difference. 

Also if you don't want to write props and transforms.
checkout: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/DataIngest#Create_a_ruleset_with_the_I...

https://docs.splunk.com/Documentation/SplunkCloud/9.2.2406/IngestProcessor/AboutIngestProcessorSolut...

If my reply helps, please upvote.

[best-west]

Hi! Thank you so much for your response and explanation. It seems like maybe I have not properly deployed these to the indexing tier.

Forgive me for the beginner question, but I think the sourcetype I created already belongs to the 000-self-service app - is this what you meant by deploying the config using self service? Screenshot below (I didn't capture the full sourcetype name): 

[sainag_splunk]

@best-west basically we need to package an new app  that has props.conf for the SEDCMD, referencing your sourcetype for the data needing to transform and deploy from UI from uploaded apps.  I think the issue might be because of 000-self-service-app . You can also ask splunk support to make this update for you. 
Is this Classic or Victoria stack?

If you want to create props/transforms as mentioned try using ingest actions and see as an example.
https://lantern.splunk.com/Splunk_Platform/Product_Tips/Data_Management/Using_ingest_actions_to_filt...





If my reply helps, please upvote.