Host Extraction REGEX in Transforms failing for lengthy events.

lzellmer_splunk — Mon, 28 Sep 2020 20:08:32 GMT

After realizing the hostname of a Blue Coat appliance was at the end of the incoming events, we created a host name extraction within props and transforms of our modified Blue Coat TA to extract the correct x_bluecoat_appliance_name. We verified the RegEx work by testing with | rex field=_raw "OUR_REGEX" and by testing externally on regex101.com. Both were 100% successful in test.

When applied to the new incoming data, we experienced failure on all lengthy messages.

Sample data and Props/Transforms are below.

It appears that there is a maximum length on RegEx for index-time extractions?

props.conf

[bcoat_proxysg]
TRANSFORMS-hostchange = bluecoat_host

transforms.conf

[bluecoat_host]
#we tried both of these regexes - starting with the lookbehind ...
REGEX = \"(\S+)\"\s{1,3}\S+\s\S+(?&lt;=$)
#then the one starting from the beginning of the message - as inefficient as it may be. 
#REGEX = \S+\s+\S+\s+\S+\s+\S+\s+\S+\s\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\"(.+)\"\s+\S+\s+\S+
DEST_KEY = MetaData:Host
FORMAT = host::$1

REGEX = (?<date>\S+)\s+(?<time>\S+)\s+(?<time_taken>\S+)\s+(?<c_ip>\S+)\s+(?<sc_status>\S+)\s(?<s_action>\S+)\s+(?<sc_bytes>\S+)\s+(?<cs_bytes>\S+)\s+(?<cs_method>\S+)\s+(?<cs_uri_scheme>\S+)\s+(?<cs_host>\S+)\s+(?<cs_uri_port>\S+)\s+(?<cs_uri_path>\S+)\s+(?<cs_uri_query>\S+)\s+(?<cs_username>\S+)\s+(?<cs_auth_group>\S+)\s+(?<s_hierarchy>\S+)\s+(?<s_supplier_name>\S+)\s+(?<rs_content_type>\S+)\s+(?<cs_referer>\S+)\s+\"?(?<cs_user_agent>.+)\"?\s+(?<sc_filter_result>\S+)\s+\"(?<cs_categories>.+)\"\s+(?<x_virus_id>\S+)\s+(?<s_ip>\S+)\s+(?<c_port>\S+)\s+(?<x_exception_id>\S+)\s+\"(?<cs_category>.+)\"\s+(?<cs_uri_extension>\S+)\s+(?<cs_uri>\S+)\s+(?<s_sitename>\S+)\s+(?<r_ip>\S+)\s+(?<r_dns>\S+)\s+(?<s_session_id>\S+)\s+\"(?<x_bluecoat_appliance_name>.+)\"\s+(?<x_cache_info>\S+)\s+(?<x_rs_streaming_content>\S+)

FIELDS Listing from Headers
FIELDS="date","time","time_taken","c_ip","sc_status","s_action","sc_bytes","cs_bytes","cs_method","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_username","cs_auth_group","s_hierarchy","s_supplier_name","rs_content_type","cs_referer","cs_user_agent","sc_filter_result","cs_categories","x_virus_id","s_ip","c_port","x_exception_id","cs_category","cs_uri_extension","cs_uri","s_sitename","r_ip","r_dns","s_session_id","x_bluecoat_appliance_name","x_cache_info","x_rs_streaming_content"

sample data

SHORT MSG:
2015-06-05 12:14:47 44 8.8.8.8 200 TCP_HIT 1206 2324 GET http data.t.bleacherreport.com 80 /jsonp/MLB_Reg/Baseball/2015/6/5/e19d5d29-13ce-4f34-b3c3-c13814afe6da/line_scores.json ?callback=BRLineScore_isOver_76524 username - - 8.8.8.8 application/javascript http://bleacherreport.com/articles/2484112-the-miami-heat-in-surprising-showdown-simply-cant-afford-to-lose-dwyane-wade "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/7.0; NISSC)" OBSERVED "Sports/Recreation" - 4.4.4.4 62169 - "Sports/Recreation" json http://data.t.bleacherreport.com/jsonp/MLB_Reg/Baseball/2015/6/5/e19d5d29-13ce-4f34-b3c3-c13814afe6da/line_scores.json?callback=BRLineScore_isOver_76524 SG-HTTP-Service 157.166.249.67 data.t.bleacherreport.com - "bc-appliance-hostname" - -

Re: Host Extraction REGEX in Transforms failing for lengthy events.

lzellmer_splunk — Fri, 05 Jun 2015 17:34:26 GMT

For messages longer than 4096 characters in length, the LOOKAHEAD parameter must be applied to your RegEx:

LOOKAHEAD =
* NOTE: This option is valid for all index time transforms, such as index-time
field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Defaults to 4096. You may want to increase this value if you have event line lengths that
exceed 4096 characters (before linebreaking).

transforms.conf

[bluecoat_host]
LOOKAHEAD = 10000
REGEX =\"(\S+)\"\s{1,3}\S+\s\S+(?<=$)
DEST_KEY = MetaData:Host
FORMAT = host::$1

Re: Host Extraction REGEX in Transforms failing for lengthy events.

richgalloway — Fri, 05 Jun 2015 17:38:25 GMT

If you solved your problem, please accept your answer.

topic Re: Host Extraction REGEX in Transforms failing for lengthy events. in Splunk Search

Host Extraction REGEX in Transforms failing for lengthy events.

Re: Host Extraction REGEX in Transforms failing for lengthy events.

Re: Host Extraction REGEX in Transforms failing for lengthy events.