Custom Splunk_TA_apache for new access log format

kenntun — Tue, 29 Sep 2020 22:36:12 GMT

We have new apache access log and ssl access log format as follow:

ssl_access_log

test_server:18301 172.31.107.148 172.31.4.40 - - [08/Jan/2019:16:32:15 +0800] 2985 "GET /monitor/check.txt HTTP/1.1" 200 12 617 3011 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" "-" "TLSv1.2" "ECDHE-RSA-AES256-GCM-SHA384"

access_log

test_server:18300 172.31.107.148 172.31.4.178 - - [08/Jan/2019:16:22:40 +0800] 2865 "GET /server-status HTTP/1.1" 200 16164 136 16451 "-" "libwww-perl/6.13" "-" "-" "-"

I would like know how to modifythe props.conf so that it can extract the correct fields. Our current configuration are as follow:

[apache:access:test_server]
category = Web
description = Access logs produced by Apache Web Server (test_server)
pulldown_type = true
SHOULD_LINEMERGE = false
KV_MODE = none

EXTRACT-apache_access= ^(?[^:]+):(?\d+)\s+(?[^ ]+)\s+(?\S+)\s+(?\S+)\s+\[(?\d+\/\w+\/\d+:\d+:\d+:\d+\s+[-+]\d+)\]\s+(?\d+)[^"\n]*"(?[^"]+)[^ \n]*\s+(?\d+)\s+(?\d+|[-])\s+\"(?[^"]+)\"\s+(?[^ ]+)\s+(?[^ ]+)\s*\"*(?[^"]+)\"*
EXTRACT-apache_request = (?\w*)\s+(?[^ ]*)\s+(?[^"]+)[^ \n]* in request
EXTRACT-source_filename = (?[^/]*)$ in source
EXTRACT-site = ^(?[^_]+)_access_(?[^_]+).log in source_filename

EVAL-bytes_in                                           = 0
EVAL-bytes_out                                          = 0

FIELDALIAS-bytes_in                             = request_bytes as bytes_in
FIELDALIAS-bytes_out                            = response_bytes as bytes_out
FIELDALIAS-src_ip                   = src as src_ip
FIELDALIAS-dest                     = host as dest
FIELDALIAS-http_referrer                        = http_referer as http_referrer

EVAL-site                                                       = ""
EVAL-web_server                                     = host . ":" . site
EVAL-bytes                                                      = bytes_in+bytes_out
#EVAL-response_time                                     = response_time_microseconds/1000
EVAL-response_time                                      = response_time_milliseconds

EVAL-product                                            = "Web Server"
EVAL-vendor                                             = "Apache"
EVAL-vendor_product                             = "Apache Web Server"
EVAL-dest_ip                                = if(match(host,"\d+.\d+.\d+.\d+"), host, null())

LOOKUP-apache_httpstatus_lookup         = apache_httpstatus_lookup status OUTPUT status_description status_type

topic Custom Splunk_TA_apache for new access log format in Getting Data In

Custom Splunk_TA_apache for new access log format