Getting Data In

Replacing backslash not working in SEDCMD after re-directing through transforms.conf and applying it in props.conf.

chaitalynavare
Engager

Hi,

I am trying to escape backslash character from json data. It works when I apply SEDCMD definations in props.conf soucetype - mysrc. But when I re-direct the definations to transforms.conf (custom_data_one and custom_data_two) to transform data for particular pattern & extract required data from the json event, and then apply the SEDCMD in mentioned sourcetype (mysrc_two) it doesnt works.
Please share your thoughts on this.

Data:

{"docker":{"container_id":"852241528698541tzfjztdgtzjsxf"},"kubernetes":{"container_name":"a-kterminal","namespace_name":"kterminal","pod_name":"a-kterminal-555-85chghv","pod_id":"858gh-zgzh-gjh-ghg-896545213","labels":{"application":"a-kterminal","createdBy":"k-rass-template","deployment":"a-kterminal-555","deploymentConfig":"a-kterminal","deploymentconfig":"a-kterminal"},"host":"sdeb-gv-g58","master_url":"https://kubernetes.default.hgfbsjbgsk","namespace_id":"uzsefgvshj-dsgfvjhdv-ztfvsjhybv","namespace_labels":{"app_code":"mycode","network-policy":"true","splunk":"true","splunkindex":"myindex"}},"message":"2019-11-04 14:07:12.321 TRACE 1 --- [nio-8080-exec-4] c.k.k.d.trackinglogger.TrackingLogger    : {\"timeStamp\":\"2019-11-04T14:07:12.321Z\",\"country\":\"DE\",\"environment\":\"at\",\"payload\":\"/bye/0\",\"loggingVersion\":\"1.0.0\",\"sessionId\":\"uzsefgvshj-dsgfvjhdv-ztfvsjhybv\",\"terminalId\":\"ABC-12345TST0103\",\"storeId\":\"8950\",\"floor\":\"0\",\"type\":\"System\"}\n","level":"info","hostname":"abc-555-g85","pipeline_metadata":{"collector":{"ipaddr4":"123.12.00.123","ipaddr6":"abc::abc5:abc54:a12:12a","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2019-11-04T14:07:13.101993+00:00","version":"0.12.43 1.6.0"}},"@timestamp":"2019-11-04T14:07:12.321816+00:00","viaq_index_name":"project.kterminal.uzsefgvshj-dsgfvjhdv-ztfvsjhybv","viaq_msg_id":"uzsefgvshj-dsgfvjhdv-ztfvsjhybv","forwarded_by":"splunk-connect-1-854ik","source_component":"t01"}

Data from which all backslash () need to be removed to view the data in proper json format:

{\"timeStamp\":\"2019-11-04T14:07:12.321Z\",\"country\":\"DE\",\"environment\":\"at\",\"payload\":\"/bye/0\",\"loggingVersion\":\"1.0.0\",\"sessionId\":\"uzsefgvshj-dsgfvjhdv-ztfvsjhybv\",\"terminalId\":\"ABC-12345TST0103\",\"storeId\":\"8950\",\"floor\":\"0\",\"type\":\"System\"}

Configurations :-

OLD CONFIG which works as expected -
props.conf

[mysrc]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
SEDCMD-remove_header = s/{\"docker.*\,\"message":.*\s+\:\s+//g
SEDCMD-remove_footer = s/\\n"\,\"level"\:.*//g
SEDCMD-replace_backslash = s/\\//g

NEW CONFIG which doesnt remove backslash -

[mysrc_one]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
TRANSFORMS-kdt-one = custom_data_one
TRANSFORMS-kdt-two = custom_data_two

[mysrc_two]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
SEDCMD-replace_backslash = s/\\//g

transforms.conf

[custom_data_one]
REGEX = "splunkindex":"myindex"
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::mysrc_two

[custom_data_two]
REGEX = ({\"docker.*"splunkindex":"myindex"}},\"message":.*\s+\:\s+)(.*)(\\n"\,\"level"\:.*)
DEST_KEY = _raw
FORMAT = $2

Thanks!
Chiataly Navare

0 Karma
1 Solution

FrankVl
Ultra Champion

Any index-time props.conf settings are only applied based on the original sourcetype. Splunk does not process index-time props.conf settings (like SEDCMD) again, after rewriting the sourcetype using TRANSFORMS.

View solution in original post

0 Karma

FrankVl
Ultra Champion

Any index-time props.conf settings are only applied based on the original sourcetype. Splunk does not process index-time props.conf settings (like SEDCMD) again, after rewriting the sourcetype using TRANSFORMS.

View solution in original post

0 Karma

chaitalynavare
Engager

Yeah looks like you are right. Whatever changes I do to transforms.conf and primary sourcetype they are adapted quikly but the secondry sourcetype doesnt work at all.

But when I went through the below link, I learnt that the props SEDCMD and transforms thing happen at same time. (I am reffering to regex replacement block here) -

https://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

FrankVl
Ultra Champion

Correct, they both take place in the same phase. Not sure how it exactly works under the hood, but basically when the event enters the phase, splunk looks at host, source and sourcetype and based on that determines what props and transforms to apply during typing phase and then applies all those in a certain order. Those props and transforms may change host, source and sourcetype, but this does not trigger Splunk to re-evaluate what props and transforms to apply.

Changing those 3 fields is basically only useful for search time.

The only exception is when you do a CLONE_SOURCETYPE transform, there the cloned events are injected back in to the start of the pipeline and SED etc. is applied as if it had come in originally with the new sourcetype. But I guess you're not interested in cloning anything.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @chaitalynavare,
in your regexes there are some special chars no escaped (as { or "), so try something like this:
In props.conf:

SEDCMD-remove_header = s/\{\"docker.*,\"message\":.*\s+:\s+//g
SEDCMD-remove_footer = s/\\n\"\,\"level\":.*//g
SEDCMD-replace_backslash = s/\\//g

In transforms.conf:

REGEX = \"splunkindex\":\"myindex\"    
REGEX = (\{\"docker.*\"splunkindex\":\"myindex\"}},\"message\":.*\s+:\s+)(.*)(\\n\",\"level\":.*)

Ciao.
Giuseppe

0 Karma

chaitalynavare
Engager

Hi @gcusello,

I am able to extract the required data as expected but revoming backslash not working in props (Refer NEW CONFIG from question above). Whereas the same config works if I apply without involving transforms.conf (Refer OLD CONFIG from question above).

NOTE: I am using NEW configs as I want to limit the slash to be removed and extract required data only from specific pattern ("splunkindex":"myindex").

Thanks,
Chaitaly Navare

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!