topic Re: Transforms.conf not working as expected in Getting Data In

Why isn't my Transforms.conf working as expected?

DarshanBK — Tue, 04 Apr 2023 19:35:44 GMT

I have below configurations in transforms and props config files to fetch only events containing keyword 'splunking' in the log files. But it seems to be not working .

transforms.conf

[keepOnly10Lines]
REGEX=splunking
FORMAT=indexQueue
DEST_KEY=queue

props.conf
[test-GP]
TRANSFORMS-set = keepOnly10Lines

inputs.conf

[monitor:///opt/splunk/data/osheanTest/darsha_test*.log]
index = main
sourcetype = test-GP
disabled = 0
whitelist = .log$
move_policy = sinkhole
crcSalt = <source>

Below are the logs:

05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - iueyrh8923f 2f82hob3f 208fhob 23f802ofb 2f8uo2bj f28ufb 2f892uobf2803fbuo j2f028bof j20fi oj
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkk - be27tf829fb 2u79fg2uibf 20fb 2f972gbu f20fb f0h2if 20f8bo f2hinfp 2fip 2f802fio2nf l
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - uewhwf8iew cewuwbkj cobvjl ced08 jlwcuwojl vcew0vbjl wevcowejbl vwpeubvjl wvujwlevhwpivnwepviblj m
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - 73ye9ubf 2fy92ou3bfj 2fhuo2bj f2yfdou2bj f208fhoub2jf02obfjl20fhinkf2pihbfl f9ip2knf c-92pjfpi2k 2-hpifn;k
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkk - ye08ru280fihn2 f20hfoib 2f0h2bi f2-9fpi2n f2fhpi2nk f2-9phifnk; 2fh2pibk f2fhpin;k
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - ifhone 2n0ifnlk2 mfn082oihldj ovuce2h083do2bj fc028ifh3f8oih2lfdn2fob2jf80hi2pblj m9-2ufjpn;k f082hif 2
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkkkk - 8yd802hoifn 2fu2bj f28foub 2f9i2uk f2fobj 2fb 292fpin2 f29jpfin;k 2fpi2nf 0iphnfl 2fiplk 2fhipbl
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkk - d80dfh2inf280fyhoin2lf082hfoibnl 3df032u2inf2083yfh2n3f082y3fhn2 n2803f2ifn 2f820bf 280f2ob f280foi 2jl82u0ib
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - e3ue832oin 23ifh23oilkf 2380ifb 23f802obuf 29-fhpi2 f290fpi 2f-2ipk
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkk - 3hd982yo802in f230f92hin3 f23fhpib2 3f230hpifn23fpi2b l
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkkkk - wyud8230foidn 2f02hiofn2fhpi2bf2hipfb2fpi2b3 f23f2-93fpi2n;k3 f2-fhpi2n3k; f2-39hpifnk; m
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - feature="IOWait" color=green due_to_stanza="feature:iowait" node_type=feature node_path=splunkkkkkkkkkkd.resource_usage.iowait
05-12-2019 22:07:53.705 +0100 INFO splunkingkkkkkkkkk - vgavsgcavs chcyvashc msacyhasvc asasycvas casycvajs casyicxh darshan
05-12-2019 22:07:53.705 +0100 INFO splunkkkkkkkkkk - 10520523 3412 0520523 120523 120534gtey54y darshan
05-12-2019 21:37:53.702 +0100 INFO splunkkkkkkkkkk - 2052052ftrfquxutfxyiyqigx yhghck scxixb qcyicgkhqwmn cqwicykh darshan.

Please help me in figuring out what is hindering splunk from applying transforms and props configuartions.

Re: Transforms.conf not working as expected

gcusello — Tue, 04 Apr 2023 06:45:26 GMT

Hi @DarshanBK,

let me understand: you want to take all the events containing "splunking" and discard all the other events?

if this is your requirement, your configuration isn't correct, as described at https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues you have to configure two queues: one to take events and one to discard them.

something like this:

in props.conf

[test-GP] TRANSFORMS-set = keepOnly10Lines, setnull

in transforms.conf

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [keepOnly10Lines] REGEX=splunking DEST_KEY = queue FORMAT = indexQueue

the order of commands (keepOnly10Lines before setnull) is relevant in props.conf, instead isn't relevant in transforms.conf the order of stanzas.

Ciao.

Giuseppe

Re: Transforms.conf not working as expected

DarshanBK — Tue, 04 Apr 2023 08:35:53 GMT

@gcusello

Thanks for your suggestion!

I have one more requirement where we need to fetch only first 10lines from the above logs.

Its a huge file and consumes lot of license. so we need to index only first 10 lines.

Is it possible? if yes, how can we do it?

I have tried many option with no success.

Can you please guide me ?

Re: Transforms.conf not working as expected

gcusello — Tue, 04 Apr 2023 10:30:55 GMT

Hi @DarshanBK,

when you say ten rows are you meaning ten rows of each event or only ten events?

if the event's I'm not sure that's possible.

If you mean ten rows of each event, you can configurate the TRUNCATE parameter in props.conf to take only the first X chard of each event.

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Ciao.

Giuseppe