topic Re: Using a transform i cant use SEDCMD in Getting Data In

Using a transform i cant use SEDCMD

robertlynch2020 — Tue, 29 Sep 2020 18:53:48 GMT

I am taking in data and making a new source type, so i need to use a transform for this.
The issue is when i use this i cant seem to use SEDCMD to trim some of the lines i am taking in.

Props.conf
[AMBER_RAW:METRIC]
SEDCMD-remove_header = s/^.*?{/{/1
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = \"ts\":\"
INDEXED_EXTRACTIONS = JSON

Transform.conf
[AMBER_RAW_json_METRIC]
DEST_KEY = MetaData:Sourcetype
REGEX = {"v":"1.0\"
FORMAT = sourcetype::AMBER_RAW:METRIC

If i don't use the Transform the SEDCMD-remove_header works, but in the case i need to, for another issue i am having.

Any ideas, how to get around this?
Example of the data, but i have it working if i take it in difectly. However i have to use a transform in this case as i have multiple sourcetype in one files.

2018-01-10 15:50:03 [metrics-application-1-thread-1] INFO  METRIC:41 - {"v":"1.0","t":"MTR","ts":"2018-01-10T15:50:03.704Z","h":"mx7654vm","pid":12483,"src":{"c":"authn-app","d":"auth"},"mtr":{"counters":{"process":{"cpu":{"time_cumulated_s":35},"memory":{"gc":{"ps_marksweep":{"total_duration_ms":814},"ps_scavenge":{"total_duration_ms":539}}}}},"gauges":{"com.murex.serviceframework.rest.datalayer.DataSourceMetrics.datasources.authn-authn-app-1":{"availableConnectionCount":1,"borrowedConnectionCount":0,"currPoolSize":1,"maxPoolSize":50,"poolName":"authn-authn-app-1"},"process":{"cpu":{"percentage":0.0014801778579070887},"files":{"open_files":37},"memory":{"jvm":{"heap":{"committed_kb":195072,"used_kb":111654},"nonheap":{"committed_kb":91456,"used_kb":89829}},"rss_kb":32880864,"vsz_kb":2295108}}},"histograms":{},"meters":{},"timers":{"process":{"memory":{"gc":{"ps_marksweep":{"events":{"count":1,"rate_1m":0.010541994097562058,"rate_5m":0.0030413993186727347,"rate_15m":0.001077675326868502,"rate_mean":0.023586525047214212},"duration_ms":{"max":620.0,"mean":620.0,"median":620.0,"min":620.0,"percentile_75":620.0,"percentile_95":620.0,"percentile_98":620.0,"percentile_99":620.0,"percentile_999":620.0,"standard_deviation":0.0}},"ps_scavenge":{"events":{"count":32,"rate_1m":1.3370365746688775,"rate_5m":1.8460208181687348,"rate_15m":1.9473463934310977,"rate_mean":0.7547234936660224},"duration_ms":{"max":18.0,"mean":9.125,"median":6.5,"min":3.0,"percentile_75":13.0,"percentile_95":18.0,"percentile_98":18.0,"percentile_99":18.0,"percentile_999":18.0,"standard_deviation":5.014495118187132}}}}}}}}

Thanks, in advance
Rob

Re: Using a transform i cant use SEDCMD

mayurr98 — Fri, 06 Apr 2018 12:28:43 GMT

Hello

The problem is that the 2 rules are for index time.
The first transform will apply renaming the sourcetype original to new(AMBER_RAW:METRIC).
But the event will not be parsed a second time (at index time) for the new sourcetype rules. Therefore the SEDCMD will never happen.

NOTE : at search time. the new sourcetype rules may stilll apply ( by example a field extraction)
so you will need to use sedcmd at search time like this

index=<your_index> sourcetype=AMBER_RAW:METRIC |  rex mode=sed "s/^.*?{/{/1"

As per below doc
http://wiki.splunk.com/Community:HowIndexingWorks
Both SEDCMD and transforms.conf occurs during 'Typing' queue process and since data is coming for original sourcetype so configuration for original sourcetype will take effect and configuration (index-time) for new sourcetype(AMBER_RAW:METRIC ) will never take place.

let me know if this helps!

Re: Using a transform i cant use SEDCMD

robertlynch2020 — Fri, 06 Apr 2018 13:16:24 GMT

Thanks. Do you think i can move it from one transform to another so i can take out this data?

Is there anything that can be done to help in this case.

I need to push this data into a datamodel this is why i need this.

Thanks
Robert

Re: Using a transform i cant use SEDCMD

mayurr98 — Fri, 06 Apr 2018 13:23:31 GMT

again it will conflict. well what you can do is try applying sedcmd and transforms on the original sourcetype and see what happens.this should work as you are applying configuration on the original sourcetype.

Re: Using a transform i cant use SEDCMD

jconger — Fri, 06 Apr 2018 14:47:58 GMT

@mayurr98 is correct about the order of operations in the indexing pipeline. To get around this, you could use CLONE_SOURCETYPE in transforms.conf. According to the documentation, "The cloned event will be further processed by index-time transforms and SEDCMD expressions according to its new sourcetype."

Re: Using a transform i cant use SEDCMD

robertlynch2020 — Mon, 09 Apr 2018 21:22:05 GMT

hi
Thanks for this this worked 🙂

Re: Using a transform i cant use SEDCMD

robertlynch2020 — Mon, 09 Apr 2018 21:22:56 GMT

Thanks for this, the first answer did work for me. But this is good to know for the future.

Cheers
Robert