Index Time Extractions Regex meeting character limit? (props.conf)

[oliverja]

I have been fighting with a regex in my props.conf (Regex-working-on-search-but-not-props-transforms ) and after a lot of testing, I came to the conclusion that my regex is fine. And my props.conf is fine. I had originally failed to see SOME were processing fine, and assumed all were bad. So, I checked to see if there was a cutoff where some were good vs bad, and found one, inside my len(_raw) field. Turns out, my props.conf seems to stop working somewhere after 4096 character count. 

So, I made some test data and pulled it into splunk

Bash:

#!/bin/bash
x="a"
count=1
while [ $count -le 5000 ]
do
  echo "$x regex"
  ((count+=1))
  x+="a"
done

This just makes a bunch of lines, incrementing with another char each time.

a regex
aa regex
aaa regex
aaaa regex
aaaaa regex
aaaaaa regex
aaaaaaa regex

Then I pull this into Splunk, 1 line per log. Static host field = "x"

Props:

[host::x]
SHOULD_LINEMERGE = false
TRANSFORMS-again = regexExtract

Transforms:

[regexExtract]
REGEX = .*(regex)
DEST_KEY= _raw
FORMAT = $1
WRITE_META = true

My search: 

index="delete" sourcetype="output.log"   | eval length=len(_raw) | search  length<4100  |  table length | sort - length

Assuming my KV limits are ok, because this btool snippit:

/opt/splunk/etc/system/default/limits.conf [kv]
/opt/splunk/etc/system/default/limits.conf avg_extractor_time = 500
/opt/splunk/etc/system/default/limits.conf indexed_kv_limit = 200
/opt/splunk/etc/system/default/limits.conf limit = 100
/opt/splunk/etc/system/default/limits.conf max_extractor_time = 1000
/opt/splunk/etc/system/default/limits.conf max_mem_usage_mb = 200
/opt/splunk/etc/system/default/limits.conf maxchars = 10240
/opt/splunk/etc/system/default/limits.conf maxcols = 512

------------------------------------------

Now that we have all the setup, I search my index and see that every field up thought 4097 chars long will process the index time regex.

After that, the search time regex works fine, but index time is no longer functional.

How do i get it to continue processing beyond that ~4100 chars? 

[oliverja · Accepted solution]

Once I had "4096" as my limit, I was able to change focus a bit and figure out what was happening. So, way way too much time fighting with my logs, I fixed it.

[regexExtract]
REGEX = .*(regex)
LOOKAHEAD=20000   <----THIS DEFAULTS TO 4096
DEST_KEY= _raw
FORMAT = $1
WRITE_META = true