Hi @jhooper33,
Internally, your regular expression compiles to a length that exceeds the offset limits set in PCRE2 at build time.
For example, the regular expression (?<bar>.){3119} will compile:
| makeresults
| eval foo="anything"
| rex field=foo "(?<bar>.){3119}"
but this regular expression (?<bar>.){3120} will not:
| rex field=foo "(?<bar>.){3120}"
Error in 'rex' command: Encountered the following error while compiling the regex '(?<bar>.){3120}': Regex: regular expression is too large.
Repeating the match 3,120 times exceeds PCRE2's compile-time limit. If we add a second character to the pattern, we'll exceed the limit in fewer repetitions:
Good:
| rex field=foo "(?<bar>..){2339}"
Bad:
| rex field=foo "(?<bar>..){2340}"
Error in 'rex' command: Encountered the following error while compiling the regex '(?<bar>..){2340}': Regex: regular expression is too large.
The error message should contain a pattern offset to help with identification of the error; however, Splunk does not expose that, and enabling DEBUG output on SearchOperator:rex adds no other information.
In short, the code generated by the regular expression compiler is too long, and you'll need to modify your regular expression.
With respect to CSV lookups versus KV store lookups, test, test, and test again. A CSV file deserialized to an optimized data structure in memory should have a search complexity similar to a MongoDB data store, but because the entire structure is in memory, the CSV file may outperform a similarly configured KV store lookup. If you want to replicate your lookup to your indexing tier as part of your search bundle, you also need to consider that a KV store lookup will be serialized to a CSV file and used as a CSV file on the indexer.
Finally, if you're using Splunk Enterprise Security, consider integrating your threat intelligence with the Splunk Enterprise Security threat intelligence framework. The latter may not meet 100% of your requirements, so as before, test, test, and test again.
