fun with sedcmd

fisuser1 — Wed, 30 Sep 2020 00:42:40 GMT

Having issues with a sedcmd in my props. When I test this in my dev environment, I see expected results. However, when I apply this to my distributed environment and push it to my indexers, I am not seeing the same results. Does not seem to be stripping the excess data when applied to the indexer cluster. Also, I did try this in a props.conf on a HF fronting our cluster, and still did not work.

props.conf
[my_lil_old_sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=15
disabled=false
TIME_FORMAT=%Y%m%d:%H%M%S
TIME_PREFIX=##TUXIF:
TZ=GMT
SEDCMD-purge=s/^(?!##TUXIF).+//g

DEV results: (good)

TUXIF:20190523:164904:000001.273:529021******2906/0:110:100:1:MCRD->APS:150.000:978:2052658003:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.450:550129******1803/0:110:100:0:MCRD->FRT:22.800:978:118875070:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.346:446223******7606/0:110:100:0:VISA->BOV:760.000:826:118875080:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.257:559147******8697/0:110:100:0:MCRD->BNK:1.800:978:118875090:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.385:446223******3604/0:110:100:0:VISA->BOV:7.000:826:118875060:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.412:550129******3036/0:110:100:0:MCRD->FRT:2.300:978:118875050:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.231:446223******3902/0:110:100:0:VISA->BOV:14.710:826:118875065:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.285:479613******0487/0:110:100:0:VISA->PMM:3.100:826:118875055:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.248:557418******4844/0:110:100:0:MCRD->PCT:20.050:826:118875040:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.216:465850******1704/0:110:100:0:VISA->BOV:4.730:826:118875045:0/00:NP:NP:NP

TUXIF:20190523:164904:000002.785:472628******5886/0:110:100:0:VISA->PKT:2.750:826:118874935:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.122:401773******6492/0:110:100:0:VISA->GVS:38.000:978:118875035:1/16:NP:NP:Insufficient Funds ! (Txn amount 38.000 (including commission 0.000), Available amount 30.000, Override 0.000 )

TUXIF:20190523:164903:000000.293:446223******6804/0:110:100:0:VISA->BOV:2.990:826:118875030:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.598:550129******9908/0:110:100:0:MCRD->FRT:37.000:978:118874990:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.218:446223******8302/0:110:100:0:VISA->BOV:9.690:826:118875025:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.337:557362******0812/0:110:100:0:MCRD->ANP:16.060:978:118875020:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.593:533935******3582/0:110:100:0:MCRD->PSH:92.250:978:118874985:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.283:511656******0698/0:110:100:1:MCRD->ACL:203.500:840:118875000:0/00:NP:NP:NP

TUXIF:20190523:164903:000002.552:472628******8479/0:110:100:0:VISA->PKT:3.200:826:118874900:0/00:NP:NP:NP

TUXIF:20190523:164903:000002.596:472628******5328/0:110:100:0:VISA->PKT:1.000:826:118874895:0/00:NP:NP:NP

raw log data (without stripping data)

3528484:20190523:16490195:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/06/559147_2406_0001]

TUXIF:20190523:164902:000000.394:559147******2406/0:110:100:0:MCRD->BNK:26.950:949:118874950:0/00:NP:NP:NP

6161662:20190523:16490245:TUXCLT:slfdbg.c:1033:Could not SET NAME BY PAN - NO PAN AVAILABLE !
******************************** TUXCLT service ********************************
6161662:20190523:16490245:TUXCLT:slfdbg.c: 667:Service called with:
C_FNCODE 831
I_REQ_CHARSET 0
C_STAN 704534

46923944:20190523:16490128:TUXIF :slfdbg.c:1252:Setting file to I_DBGFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/47/540450_6247_0007]

TUXIF:20190523:164901:000000.261:540450******6247/0:110:100:1:MCRD->APS:40.000:826:2052657978:1/16:NP:NP:Insufficient Funds ! (Txn amount 40.500 (including commission 0.500), Available amount 4.930, Override 0.000 )

40239190:20190523:16490090:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/71/529021_6671_0006]

TUXIF:20190523:164901:000000.628:529021******6671/0:110:100:0:MCRD->APS:147.440:826:2052657973:0/00:NP:NP:NP

37748910:20190523:16490158:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/49/540450_2249_0002]

TUXIF:20190523:164901:000000.244:540450******2249/0:110:100:0:MCRD->APS:9.480:826:2052657988:0/00:NP:NP:NP

49217788:20190523:16490051:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/85/516240_0385_0003]

TUXIF:20190523:164901:000001.363:516240******0385/0:110:100:0:MCRD->YTG:100.000:702:2052657968:0/00:NP:NP:NP

35782700:20190523:16490133:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/00/516240_1200_0002]

TUXIF:20190523:164901:000000.573:516240******1200/0:110:100:0:MCRD->YTG:27.080:840:2052657983:0/00:NP:NP:NP

50921508:20190523:16490184:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/28/540450_8228_0006]

TUXIF:20190523:164902:000000.231:540450******8228/0:110:100:0:MCRD->APS:12.000:826:2052657993:0/00:NP:NP:NP

39190644:20190523:16490207:TUXCLT:slfdbg.c:1033:Could not SET NAME BY PAN - NO PAN AVAILABLE !
******************************** TUXCLT service ********************************
39190644:20190523:16490207:TUXCLT:slfdbg.c: 667:Service called with:
C_FNCODE 831
I_REQ_CHARSET 0
C_STAN 705292

C_RRN 914315704534
C_RSPCODE 00
M_SEVCODE TUXIF
M_SEVCODE TUXCLT
6161662:20190523:164902:TUXCLT:GSM: Event ev_ok
6161662:20190523:16490245:TUXCLT:GSM: State st_return (0.00)[0.00]
6161662:20190523:16490245:TUXCLT:xcltsm.c: 545:M_err: 0
6161662:20190523:164902:TUXCLT:GSM: event DEFAULT [0]
6161662:20190523:16490245:TUXCLT: tuxif.c: 340:TUXCLT Return: TPSUCCESS
************************* Start of Fielded Buffer Diff *************************
Diff FB's : (+)Added by svc, (-) Deleted by svc, (C) Changed by svc
(C) C_MSGFN [0] : from 0 to 1
(+) C_ACTIONCODE [0] : 8
(+) C_RSPCODE [0] : 00
*************************** ntp_return from : TUXCLT ***************************
30344220:20190523:16490147:TUXIF :slfdbg.c:1252:Setting file to I_DBGFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/14/528838_8914_0003]

TUXIF:20190523:164902:000001.218:528838******8914/0:110:100:1:MCRD->NET:50.000:978:118874945:0/00:NP:NP:NP

21824832:20190523:16490003:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/472628_0904_0001]

TUXIF:20190523:164902:000002.656:472628******0904/0:110:100:0:VISA->PKT:46.900:826:118874865:0/00:NP:NP:NP

45745202:20190523:16490258:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/446223_5904_0048]

TUXIF:20190523:164902:000000.235:446223******5904/0:110:100:0:VISA->BOV:6.990:826:118874955:0/00:NP:NP:NP

36046094:20190523:16490022:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/84/472628_4984_0003]

TUXIF:20190523:164902:000002.656:472628******4984/0:110:100:0:VISA->PKT:2.000:826:118874875:0/00:NP:NP:NP

36767094:20190523:16490266:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/446223_9804_0083]

TUXIF:20190523:164902:000000.319:446223******9804/0:110:100:0:VISA->BOV:1.000:978:118874960:0/00:NP:NP:NP

61015798:20190523:16490270:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/95/459340_6895_0009]

TUXIF:20190523:164903:000000.295:459340******6895/0:110:100:0:VISA->SDS:6.440:978:118874965:0/00:NP:NP:NP

1377470:20190523:16490301:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/12/550129_3212_0007]
5702746:20190523:16490284:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/76/479613_7376_0004]

TUXIF:20190523:164903:000000.295:479613******7376/0:110:100:0:VISA->PMM:7.320:826:118874975:0/00:NP:NP:NP

1377470:20190523:16490302:TUXIF :slfdbg.c:1252:Setting file to I_DBGFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/12/550129_3212_0007]

TUXIF:20190523:164903:000000.245:550129******3212/0:110:100:1:MCRD->FRT:403.000:978:118874980:1/16:NP:NP:Insufficient Funds ! (Txn amount 405.000 (including commission 2.000), Available amount 218.360, Override 0.000 )

62064218:20190523:16490329:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/98/511656_0698_0000]
43189706:20190523:16490335:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/93/472628_4893_0001]
27919656:20190523:16490336:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/94/472628_3194_0002]
6161730:20190523:16490322:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/66/401773_4766_0001]

TUXIF:20190523:164903:000000.195:401773******4766/0:110:100:0:VISA->GVS:5.000:978:118874995:0/00:NP:NP:NP

52757842:20190523:16490330:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/38/401773_3438_0005]

TUXIF:20190523:164903:000000.167:401773******3438/0:110:100:0:VISA->GVS:11.970:978:118875005:1/16:NP:NP:Insufficient Funds ! (Txn amount 11.970 (including commission 0.000), Available amount 10.030, Override 0.000 )

29362098:20190523:16490089:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/28/472628_5328_0006]

TUXIF:20190523:164903:000002.596:472628******5328/0:110:100:0:VISA->PKT:1.000:826:118874895:0/00:NP:NP:NP

41223354:20190523:16490094:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/79/472628_8479_0003]

TUXIF:20190523:164903:000002.552:472628******8479/0:110:100:0:VISA->PKT:3.200:826:118874900:0/00:NP:NP:NP

62064218:20190523:16490331:TUXIF :slfdbg.c:1252:Setting file to I_DBGFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/98/511656_0698_0000]

TUXIF:20190523:164903:000000.283:511656******0698/0:110:100:1:MCRD->ACL:203.500:840:118875000:0/00:NP:NP:NP

65406164:20190523:16490304:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/82/533935_3582_0001]

TUXIF:20190523:164903:000000.593:533935******3582/0:110:100:0:MCRD->PSH:92.250:978:118874985:0/00:NP:NP:NP

15664278:20190523:16490339:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/12/557362_0812_0000]

TUXIF:20190523:164903:000000.337:557362******0812/0:110:100:0:MCRD->ANP:16.060:978:118875020:0/00:NP:NP:NP

43647120:20190523:16490353:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/02/446223_8302_0072]

TUXIF:20190523:164903:000000.218:446223******8302/0:110:100:0:VISA->BOV:9.690:826:118875025:0/00:NP:NP:NP

34997348:20190523:16490316:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/08/550129_9908_0003]

TUXIF:20190523:164903:000000.598:550129******9908/0:110:100:0:MCRD->FRT:37.000:978:118874990:0/00:NP:NP:NP

32572628:20190523:16490366:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/446223_6804_0063]

TUXIF:20190523:164903:000000.293:446223******6804/0:110:100:0:VISA->BOV:2.990:826:118875030:0/00:NP:NP:NP

39061038:20190523:16490386:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/92/401773_6492_0001]

TUXIF:20190523:164903:000000.122:401773******6492/0:110:100:0:VISA->GVS:38.000:978:118875035:1/16:NP:NP:Insufficient Funds ! (Txn amount 38.000 (including commission 0.000), Available amount 30.000, Override 0.000 )

21497278:20190523:16490131:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/86/472628_5886_0001]

TUXIF:20190523:164904:000002.785:472628******5886/0:110:100:0:VISA->PKT:2.750:826:118874935:0/00:NP:NP:NP

10880444:20190523:16490389:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/465850_1704_0007]

TUXIF:20190523:164904:000000.216:465850******1704/0:110:100:0:VISA->BOV:4.730:826:118875045:0/00:NP:NP:NP

48039384:20190523:16490389:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/44/557418_4844_0004]

TUXIF:20190523:164904:000000.248:557418******4844/0:110:100:0:MCRD->PCT:20.050:826:118875040:0/00:NP:NP:NP

15402590:20190523:16490393:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/87/479613_0487_0002]

TUXIF:20190523:164904:000000.285:479613******0487/0:110:100:0:VISA->PMM:3.100:826:118875055:0/00:NP:NP:NP

62981760:20190523:16490405:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/02/446223_3902_0078]

TUXIF:20190523:164904:000000.231:446223******3902/0:110:100:0:VISA->BOV:14.710:826:118875065:0/00:NP:NP:NP

14091452:20190523:16490390:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/36/550129_3036_0003]

TUXIF:20190523:164904:000000.412:550129******3036/0:110:100:0:MCRD->FRT:2.300:978:118875050:0/00:NP:NP:NP

6555816:20190523:16490404:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/04/446223_3604_0055]

TUXIF:20190523:164904:000000.385:446223******3604/0:110:100:0:VISA->BOV:7.000:826:118875060:0/00:NP:NP:NP

29165424:20190523:16490442:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/97/559147_8697_0003]

TUXIF:20190523:164904:000000.257:559147******8697/0:110:100:0:MCRD->BNK:1.800:978:118875090:0/00:NP:NP:NP

51381636:20190523:16490435:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/06/446223_7606_0079]

TUXIF:20190523:164904:000000.346:446223******7606/0:110:100:0:VISA->BOV:760.000:826:118875080:0/00:NP:NP:NP

27002354:20190523:16490429:TUXIF :slfdbg.c: 993:Setting file to I_DBGPANFILE fb value [/cortex/dbd/tmp/TXNS_BY_PAN/03/550129_1803_0000]

TUXIF:20190523:164904:000000.450:550129******1803/0:110:100:0:MCRD->FRT:22.800:978:118875070:0/00:NP:NP:NP

M_SEVSEC 1558626542
M_SEVUSEC 453936
M_SEVUSEC 459348
C_MSGCLS 8
C_MSGFN 1
C_TXNSRC 0
C_ACTIONCODE 8
I_SRCHOST APS
I_DSTHOST DBD
C_DATEXMIT 20190523
C_TIMEXMIT 154902

M_SEVSEC 1558626542
C_RRN 914315704534
M_SEVCODE TUXIF
M_SEVCODE TUXCLT
6161662:20190523:16490245:TUXCLT:GSM: State st_chk_msg (0.00)[0.00]
6161662:20190523:16490245:TUXCLT:xcltsm.c: 259:Looking for I_SRCHOST [APS] in hostmap
6161662:20190523:16490245:TUXCLT:xcltsm.c: 266:Host IBB not match
6161662:20190523:16490245:TUXCLT:xcltsm.c: 266:Host IDT not match
6161662:20190523:16490245:TUXCLT:xcltsm.c: 266:Host NWC not match
6161662:20190523:16490245:TUXCLT:xcltsm.c: 266:Host BOI not match
6161662:20190523:16490245:TUXCLT:xcltsm.c: 266:Host NAG not match
6161662:20190523:16490245:TUXCLT:xcltsm.c: 273:Host APS OK, afe = APS
6161662:20190523:16490245:TUXCLT:xcltsm.c: 300:I_AFE changed
6161662:20190523:164902:TUXCLT:GSM: Event ev_nmgrq
6161662:20190523:16490245:TUXCLT:GSM: State st_snd_nmg (0.00)[0.00]
6161662:20190523:16490245:TUXCLT: cocbf.c: 588:MFN_REQ -> MFN_REQRSP
6161662:20190523:16490245:TUXCLT: cocbf.c: 628:No actioncode originally
6161662:20190523:16490245:TUXCLT: cocbf.c: 652:Set action/rsp codes to 800
6161662:20190523:164902:TUXCLT:GSM: Event ev_ok
6161662:20190523:16490245:TUXCLT:GSM: State st_snd_rply (0.00)[0.00]
6161662:20190523:16490245:TUXCLT:uxifcv.c: 188:Entered conv_fb
6161662:20190523:16490245:TUXCLT:uxifcv.c: 196:Conversion table for Incoming transaction response found
6161662:20190523:16490245:TUXCLT:convcs.c: 253:I_REQ_CHARSET found
6161662:20190523:16490245:TUXCLT:convcs.c: 125:Characterset is same
6161662:20190523:16490245:TUXCLT:convcs.c: 138:No conversion is done
6161662:20190523:16490245:TUXCLT:uxifcv.c: 221:Convertion started
6161662:20190523:16490245:TUXCLT:uxifcv.c: 328:Convertion end
6161662:20190523:16490245:TUXCLT:uxifcv.c: 346:Leaving conv_fb
6161662:20190523:16490245:TUXCLT:slfdbg.c: 667:FB after second conv_fb:
C_FNCODE 831
I_REQ_CHARSET 0
C_STAN 704534

M_SEVSEC 1558626542
M_SEVUSEC 453936
M_SEVUSEC 459348
C_MSGCLS 8
C_MSGFN 0
C_TXNSRC 0
I_SRCHOST APS
I_DSTHOST DBD
C_DATEXMIT 20190523
C_TIMEXMIT 154902

M_SEVSEC 1558626542
C_RRN 914315705292
C_RSPCODE 00
M_SEVCODE TUXIF
M_SEVCODE TUXCLT
39190644:20190523:164902:TUXCLT:GSM: Event ev_ok
39190644:20190523:16490207:TUXCLT:GSM: State st_return (0.00)[0.00]
39190644:20190523:16490207:TUXCLT:xcltsm.c: 545:M_err: 0
39190644:20190523:164902:TUXCLT:GSM: event DEFAULT [0]
39190644:20190523:16490207:TUXCLT: tuxif.c: 340:TUXCLT Return: TPSUCCESS
************************* Start of Fielded Buffer Diff *************************
Diff FB's : (+)Added by svc, (-) Deleted by svc, (C) Changed by svc
(C) C_MSGFN [0] : from 0 to 1
(+) C_ACTIONCODE [0] : 8
(+) C_RSPCODE [0] : 00
*************************** ntp_return from : TUXCLT ***************************

Re: fun with sedcmd

DavidHourani — Fri, 24 May 2019 11:37:55 GMT

Hi @fisuser1,

So it's working on your dev, but not working on your production cluster this means it's not really a regex issue.

A couple of questions that might help you spot the issue :
-Where's the data coming from ? Is it being generated on an HF instance; if so does that HF have the sedcmd ?
-Are you applying your sedcmd on the right sourcetype, is it the same sourcetype between prod and dev ?
-Did you restart your indexer cluster after applying the config from the CM ?

Let me know what your answers are and we can work to resolve this.

Cheers,
David

Re: fun with sedcmd

fisuser1 — Fri, 24 May 2019 11:42:34 GMT

Hi @DavidHourani , thanks for the response. Yes, I have validated the sourcetype is correct and restarted the indexer cluster nodes after I applied the props.conf change. This data is moving through a heavy forwarder and while I did try this same sedcmd on both the HF (set this in system/local on the HF just to verify no inheritance issues) and the indexer cluster, neither works as expected.

Re: fun with sedcmd

DavidHourani — Fri, 24 May 2019 11:48:26 GMT

Hi @fisuser1, if the data is going through the HF then sedcmd must be applied there on the right sourcetype. So the config should be on the HF closest to your data, the first one receiving it before pushing to the indexers, go ahead and apply the config again, restart the HF and have a look at new data coming in see if sed is being applied. Also run the following btool command to double check that your config is there :

$SPLUNK_HOME/bin/splunk cmd btool props list <sourcetype> | grep SEDCMD

Re: fun with sedcmd

fisuser1 — Wed, 30 Sep 2020 00:42:42 GMT

Thanks again @DavidHourani, I did apply it on the HF, nulled out of the indexer cluster and restated both. Still no luck. Verified via btool all looks correct. just at a loss here why this is not working. doesn't make sense.

[/opt/splunk/etc/system/local]
$ /opt/splunk/bin/splunk cmd btool props list emea_prd_aps_auths_cortex_logs | grep SEDCMD
SEDCMD-purge = s/^(?!##TUXIF).+//g

Re: fun with sedcmd

DavidHourani — Sat, 25 May 2019 06:57:12 GMT

this could have something to do with the line breaking, are you using the same linebreaking configuration on your HF ? Maybe your lines are merged and therefore your regex matches the whole line instead of just a part of it, add this and see :

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)

Re: fun with sedcmd

fisuser1 — Mon, 27 May 2019 23:00:08 GMT

Hi @DavidHourani , funny I had the same idea in mind. I already applied this to the HF and it seems to be working as expected. Thanks again for all the suggestions.

topic Re: fun with sedcmd in Getting Data In

fun with sedcmd

TUXIF:20190523:164904:000001.273:529021******2906/0:110:100:1:MCRD->APS:150.000:978:2052658003:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.450:550129******1803/0:110:100:0:MCRD->FRT:22.800:978:118875070:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.346:446223******7606/0:110:100:0:VISA->BOV:760.000:826:118875080:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.257:559147******8697/0:110:100:0:MCRD->BNK:1.800:978:118875090:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.385:446223******3604/0:110:100:0:VISA->BOV:7.000:826:118875060:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.412:550129******3036/0:110:100:0:MCRD->FRT:2.300:978:118875050:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.231:446223******3902/0:110:100:0:VISA->BOV:14.710:826:118875065:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.285:479613******0487/0:110:100:0:VISA->PMM:3.100:826:118875055:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.248:557418******4844/0:110:100:0:MCRD->PCT:20.050:826:118875040:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.216:465850******1704/0:110:100:0:VISA->BOV:4.730:826:118875045:0/00:NP:NP:NP

TUXIF:20190523:164904:000002.785:472628******5886/0:110:100:0:VISA->PKT:2.750:826:118874935:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.122:401773******6492/0:110:100:0:VISA->GVS:38.000:978:118875035:1/16:NP:NP:Insufficient Funds ! (Txn amount 38.000 (including commission 0.000), Available amount 30.000, Override 0.000 )

TUXIF:20190523:164903:000000.293:446223******6804/0:110:100:0:VISA->BOV:2.990:826:118875030:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.598:550129******9908/0:110:100:0:MCRD->FRT:37.000:978:118874990:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.218:446223******8302/0:110:100:0:VISA->BOV:9.690:826:118875025:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.337:557362******0812/0:110:100:0:MCRD->ANP:16.060:978:118875020:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.593:533935******3582/0:110:100:0:MCRD->PSH:92.250:978:118874985:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.283:511656******0698/0:110:100:1:MCRD->ACL:203.500:840:118875000:0/00:NP:NP:NP

TUXIF:20190523:164903:000002.552:472628******8479/0:110:100:0:VISA->PKT:3.200:826:118874900:0/00:NP:NP:NP

TUXIF:20190523:164903:000002.596:472628******5328/0:110:100:0:VISA->PKT:1.000:826:118874895:0/00:NP:NP:NP

TUXIF:20190523:164902:000000.394:559147******2406/0:110:100:0:MCRD->BNK:26.950:949:118874950:0/00:NP:NP:NP

TUXIF:20190523:164901:000000.261:540450******6247/0:110:100:1:MCRD->APS:40.000:826:2052657978:1/16:NP:NP:Insufficient Funds ! (Txn amount 40.500 (including commission 0.500), Available amount 4.930, Override 0.000 )

TUXIF:20190523:164901:000000.628:529021******6671/0:110:100:0:MCRD->APS:147.440:826:2052657973:0/00:NP:NP:NP

TUXIF:20190523:164901:000000.244:540450******2249/0:110:100:0:MCRD->APS:9.480:826:2052657988:0/00:NP:NP:NP

TUXIF:20190523:164901:000001.363:516240******0385/0:110:100:0:MCRD->YTG:100.000:702:2052657968:0/00:NP:NP:NP

TUXIF:20190523:164901:000000.573:516240******1200/0:110:100:0:MCRD->YTG:27.080:840:2052657983:0/00:NP:NP:NP

TUXIF:20190523:164902:000000.231:540450******8228/0:110:100:0:MCRD->APS:12.000:826:2052657993:0/00:NP:NP:NP

TUXIF:20190523:164902:000001.218:528838******8914/0:110:100:1:MCRD->NET:50.000:978:118874945:0/00:NP:NP:NP

TUXIF:20190523:164902:000002.656:472628******0904/0:110:100:0:VISA->PKT:46.900:826:118874865:0/00:NP:NP:NP

TUXIF:20190523:164902:000000.235:446223******5904/0:110:100:0:VISA->BOV:6.990:826:118874955:0/00:NP:NP:NP

TUXIF:20190523:164902:000002.656:472628******4984/0:110:100:0:VISA->PKT:2.000:826:118874875:0/00:NP:NP:NP

TUXIF:20190523:164902:000000.319:446223******9804/0:110:100:0:VISA->BOV:1.000:978:118874960:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.295:459340******6895/0:110:100:0:VISA->SDS:6.440:978:118874965:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.295:479613******7376/0:110:100:0:VISA->PMM:7.320:826:118874975:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.245:550129******3212/0:110:100:1:MCRD->FRT:403.000:978:118874980:1/16:NP:NP:Insufficient Funds ! (Txn amount 405.000 (including commission 2.000), Available amount 218.360, Override 0.000 )

TUXIF:20190523:164903:000000.195:401773******4766/0:110:100:0:VISA->GVS:5.000:978:118874995:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.167:401773******3438/0:110:100:0:VISA->GVS:11.970:978:118875005:1/16:NP:NP:Insufficient Funds ! (Txn amount 11.970 (including commission 0.000), Available amount 10.030, Override 0.000 )

TUXIF:20190523:164903:000002.596:472628******5328/0:110:100:0:VISA->PKT:1.000:826:118874895:0/00:NP:NP:NP

TUXIF:20190523:164903:000002.552:472628******8479/0:110:100:0:VISA->PKT:3.200:826:118874900:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.283:511656******0698/0:110:100:1:MCRD->ACL:203.500:840:118875000:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.593:533935******3582/0:110:100:0:MCRD->PSH:92.250:978:118874985:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.337:557362******0812/0:110:100:0:MCRD->ANP:16.060:978:118875020:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.218:446223******8302/0:110:100:0:VISA->BOV:9.690:826:118875025:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.598:550129******9908/0:110:100:0:MCRD->FRT:37.000:978:118874990:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.293:446223******6804/0:110:100:0:VISA->BOV:2.990:826:118875030:0/00:NP:NP:NP

TUXIF:20190523:164903:000000.122:401773******6492/0:110:100:0:VISA->GVS:38.000:978:118875035:1/16:NP:NP:Insufficient Funds ! (Txn amount 38.000 (including commission 0.000), Available amount 30.000, Override 0.000 )

TUXIF:20190523:164904:000002.785:472628******5886/0:110:100:0:VISA->PKT:2.750:826:118874935:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.216:465850******1704/0:110:100:0:VISA->BOV:4.730:826:118875045:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.248:557418******4844/0:110:100:0:MCRD->PCT:20.050:826:118875040:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.285:479613******0487/0:110:100:0:VISA->PMM:3.100:826:118875055:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.231:446223******3902/0:110:100:0:VISA->BOV:14.710:826:118875065:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.412:550129******3036/0:110:100:0:MCRD->FRT:2.300:978:118875050:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.385:446223******3604/0:110:100:0:VISA->BOV:7.000:826:118875060:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.257:559147******8697/0:110:100:0:MCRD->BNK:1.800:978:118875090:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.346:446223******7606/0:110:100:0:VISA->BOV:760.000:826:118875080:0/00:NP:NP:NP

TUXIF:20190523:164904:000000.450:550129******1803/0:110:100:0:MCRD->FRT:22.800:978:118875070:0/00:NP:NP:NP

Re: fun with sedcmd

Re: fun with sedcmd

Re: fun with sedcmd

Re: fun with sedcmd

Re: fun with sedcmd

Re: fun with sedcmd