[INGEST_EVAL] Regex replace stops after 1000 chars

[tobias_stegmann]

Hi,

I'm trying to convert a hex string to readable ascii text at index time, inspired by this solution: https://answers.splunk.com/answering/204869/view.html

This works fine at search time but I need it at index time, because I have to extract the timestamp from the hex string. But at index time replace(X,Y,Z) seems to stop/break after exactly 1000 charachters using INGEST_EVAL.

To accomplish this I have the following stanzas:
transforms.conf

[test_hex]
INGEST_EVAL = raw_ascii=replace(_raw,"([0-9A-Fa-f]{2})","%\1")

props.conf

[test_eval]
TRANSFORMS-hex = test_hex

So e.g., if I index following event:

0x01234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009

The field raw_ascii looks like this:

0x%01%23%45%67%89%10%11%12%13%14%15%16%17%18%19%20%21%22%23%24%25%26%27%28%29%30%31%32%33%34%35%36%37%38%39%40%41%42%43%44%45%46%47%48%49%50%51%52%53%54%55%56%57%58%59%60%61%62%63%64%65%66%67%68%69%70%71%72%73%74%75%76%77%78%79%80%81%82%83%84%85%86%87%88%89%90%91%92%93%94%95%96%97%98%99%10%01%01%10%21%03%10%41%05%10%61%07%10%81%09%11%01%11%11%21%13%11%41%15%11%61%17%11%81%19%12%01%21%12%21%23%12%41%25%12%61%27%12%81%29%13%01%31%13%21%33%13%41%35%13%61%37%13%81%39%14%01%41%14%21%43%14%41%45%14%61%47%14%81%49%15%01%51%15%21%53%15%41%55%15%61%57%15%81%59%16%01%61%16%21%63%16%41%65%16%61%67%16%81%69%17%01%71%17%21%73%17%41%75%17%61%77%17%81%79%18%01%81%18%21%83%18%41%85%18%61%87%18%81%89%19%01%91%19%21%93%19%41%95%19%61%97%19%81%99%20%02%01%20%22%03%20%42%05%20%62%07%20%82%09%21%02%11%21%22%13%21%42%15%21%62%17%21%82%19%22%02%21%22%22%23%22%42%25%22%62%27%22%82%29%23%02%31%23%22%33%23%42%35%23%62%37%23%82%39%24%02%41%24%22%43%24%42%45%24%62%47%24%82%49%25%02%51%25%22%53%25%42%55%25%62%57%2

...Which is exactly 1000 characters.
Using the same replace statement at search time it works fine:

0x%01%23%45%67%89%10%11%12%13%14%15%16%17%18%19%20%21%22%23%24%25%26%27%28%29%30%31%32%33%34%35%36%37%38%39%40%41%42%43%44%45%46%47%48%49%50%51%52%53%54%55%56%57%58%59%60%61%62%63%64%65%66%67%68%69%70%71%72%73%74%75%76%77%78%79%80%81%82%83%84%85%86%87%88%89%90%91%92%93%94%95%96%97%98%99%10%01%01%10%21%03%10%41%05%10%61%07%10%81%09%11%01%11%11%21%13%11%41%15%11%61%17%11%81%19%12%01%21%12%21%23%12%41%25%12%61%27%12%81%29%13%01%31%13%21%33%13%41%35%13%61%37%13%81%39%14%01%41%14%21%43%14%41%45%14%61%47%14%81%49%15%01%51%15%21%53%15%41%55%15%61%57%15%81%59%16%01%61%16%21%63%16%41%65%16%61%67%16%81%69%17%01%71%17%21%73%17%41%75%17%61%77%17%81%79%18%01%81%18%21%83%18%41%85%18%61%87%18%81%89%19%01%91%19%21%93%19%41%95%19%61%97%19%81%99%20%02%01%20%22%03%20%42%05%20%62%07%20%82%09%21%02%11%21%22%13%21%42%15%21%62%17%21%82%19%22%02%21%22%22%23%22%42%25%22%62%27%22%82%29%23%02%31%23%22%33%23%42%35%23%62%37%23%82%39%24%02%41%24%22%43%24%42%45%24%62%47%24%82%49%25%02%51%25%22%53%25%42%55%25%62%57%25%82%59%26%02%61%26%22%63%26%42%65%26%62%67%26%82%69%27%02%71%27%22%73%27%42%75%27%62%77%27%82%79%28%02%81%28%22%83%28%42%85%28%62%87%28%82%89%29%02%91%29%22%93%29%42%95%29%62%97%29%82%99%30%03%01%30%23%03%30%43%05%30%63%07%30%83%09%31%03%11%31%23%13%31%43%15%31%63%17%31%83%19%32%03%21%32%23%23%32%43%25%32%63%27%32%83%29%33%03%31%33%23%33%33%43%35%33%63%37%33%83%39%34%03%41%34%23%43%34%43%45%34%63%47%34%83%49%35%03%51%35%23%53%35%43%55%35%63%57%35%83%59%36%03%61%36%23%63%36%43%65%36%63%67%36%83%69%37%03%71%37%23%73%37%43%75%37%63%77%37%83%79%38%03%81%38%23%83%38%43%85%38%63%87%38%83%89%39%03%91%39%23%93%39%43%95%39%63%97%39%83%99%40%04%01%40%24%03%40%44%05%40%64%07%40%84%09%41%04%11%41%24%13%41%44%15%41%64%17%41%84%19%42%04%21%42%24%23%42%44%25%42%64%27%42%84%29%43%04%31%43%24%33%43%44%35%43%64%37%43%84%39%44%04%41%44%24%43%44%44%45%44%64%47%44%84%49%45%04%51%45%24%53%45%44%55%45%64%57%45%84%59%46%04%61%46%24%63%46%44%65%46%64%67%46%84%69%47%04%71%47%24%73%47%44%75%47%64%77%47%84%79%48%04%81%48%24%83%48%44%85%48%64%87%48%84%89%49%04%91%49%24%93%49%44%95%49%64%97%49%84%99%50%05%01%50%25%03%50%45%05%50%65%07%50%85%09%51%05%11%51%25%13%51%45%15%51%65%17%51%85%19%52%05%21%52%25%23%52%45%25%52%65%27%52%85%29%53%05%31%53%25%33%53%45%35%53%65%37%53%85%39%54%05%41%54%25%43%54%45%45%54%65%47%54%85%49%55%05%51%55%25%53%55%45%55%55%65%57%55%85%59%56%05%61%56%25%63%56%45%65%56%65%67%56%85%69%57%05%71%57%25%73%57%45%75%57%65%77%57%85%79%58%05%81%58%25%83%58%45%85%58%65%87%58%85%89%59%05%91%59%25%93%59%45%95%59%65%97%59%85%99%60%06%01%60%26%03%60%46%05%60%66%07%60%86%09%61%06%11%61%26%13%61%46%15%61%66%17%61%86%19%62%06%21%62%26%23%62%46%25%62%66%27%62%86%29%63%06%31%63%26%33%63%46%35%63%66%37%63%86%39%64%06%41%64%26%43%64%46%45%64%66%47%64%86%49%65%06%51%65%26%53%65%46%55%65%66%57%65%86%59%66%06%61%66%26%63%66%46%65%66%66%67%66%86%69%67%06%71%67%26%73%67%46%75%67%66%77%67%86%79%68%06%81%68%26%83%68%46%85%68%66%87%68%86%89%69%06%91%69%26%93%69%46%95%69%66%97%69%86%99%70%07%01%70%27%03%70%47%05%70%67%07%70%87%09%71%07%11%71%27%13%71%47%15%71%67%17%71%87%19%72%07%21%72%27%23%72%47%25%72%67%27%72%87%29%73%07%31%73%27%33%73%47%35%73%67%37%73%87%39%74%07%41%74%27%43%74%47%45%74%67%47%74%87%49%75%07%51%75%27%53%75%47%55%75%67%57%75%87%59%76%07%61%76%27%63%76%47%65%76%67%67%76%87%69%77%07%71%77%27%73%77%47%75%77%67%77%77%87%79%78%07%81%78%27%83%78%47%85%78%67%87%78%87%89%79%07%91%79%27%93%79%47%95%79%67%97%79%87%99%80%08%01%80%28%03%80%48%05%80%68%07%80%88%09%81%08%11%81%28%13%81%48%15%81%68%17%81%88%19%82%08%21%82%28%23%82%48%25%82%68%27%82%88%29%83%08%31%83%28%33%83%48%35%83%68%37%83%88%39%84%08%41%84%28%43%84%48%45%84%68%47%84%88%49%85%08%51%85%28%53%85%48%55%85%68%57%85%88%59%86%08%61%86%28%63%86%48%65%86%68%67%86%88%69%87%08%71%87%28%73%87%48%75%87%68%77%87%88%79%88%08%81%88%28%83%88%48%85%88%68%87%88%88%89%89%08%91%89%28%93%89%48%95%89%68%97%89%88%99%90%09%01%90%29%03%90%49%05%90%69%07%90%89%09%91%09%11%91%29%13%91%49%15%91%69%17%91%89%19%92%09%21%92%29%23%92%49%25%92%69%27%92%89%29%93%09%31%93%29%33%93%49%35%93%69%37%93%89%39%94%09%41%94%29%43%94%49%45%94%69%47%94%89%49%95%09%51%95%29%53%95%49%55%95%69%57%95%89%59%96%09%61%96%29%63%96%49%65%96%69%67%96%89%69%97%09%71%97%29%73%97%49%75%97%69%77%97%89%79%98%09%81%98%29%83%98%49%85%98%69%87%98%89%89%99%09%91%99%29%93%99%49%95%99%69%97%99%89%99%10%00%10%01%10%02%10%03%10%04%10%05%10%06%10%07%10%08%10%09

Surprisingly enough, if I remove the percent sign and just use replace(_raw,"([0-9A-Fa-f]{2})","\1") (weird way to clone _raw) the field is complete.

I've tried increasing depth_limit in limits.conf but without luck.

Any ideas?

Thanks in advance!

[skalliger]

In limits.conf? DEPTH_LIMIT has to be set in transforms.conf.

Skalli

[tobias_stegmann]

Thanks for the hint. There's also a depth_limit in limits.conf. I guess limits.conf is the global one, while transforms.conf is only valid for the stanza.

I've also tried depth_limit in tranforms.conf, but that didn't work either.

[skalliger]

Hm, that is unfortunate. If that doesn't work, I don't have any other ideas. Maybe ask Splunk support about it because I haven't found anything in the specs either. If you get a solution keep us updated. 🙂

Skalli