Hi all,
I'm trying to modify the SplunkforSquid app to read my squid custom log file format correctly. As per squid.conf it is-
logformat test %ts.%03tu %6tr %>a %Ss/%03Hs 0 %03Hs %st %rm %ru %un %<A
Log format codes (trimmed):
# >a Client source IP address
# <A Server IP address or peer name
# ts Seconds since epoch
# tu subsecond time (milliseconds)
# tr Response time (milliseconds)
# un User name
# Hs HTTP status code
# Ss Squid request status (TCP_MISS etc)
# rm Request method (GET/POST etc)
# ru Request URL
# st Request+Reply size including HTTP headers
I've tried a few things here, creating field extractions in Splunk was working OK until I got to the username field, as often the username is just "-" the regex creator in Splunk would not detect this. My regex knowledge is nowhere near enough to debug this. Some help would be greatly appreciated.
UPDATE
Attempting to use delimExtractions:
props.conf-
[squid]
REPORT-main=delimExtractions
SHOULD_LINEMERGE=false
TIME_FORMAT=%+ #log format time is in epoch. not sure if this is right
MAX_TIMESTAMP_LOOKAHEAD=19
KV_MODE = none
transforms.conf-
[delimExtractions]
DELIMS=" "
FIELDS="timestamp","responsetime","clientip","not_needed","zero","http_status","total_size","method","uri","username","server_ip
Fields such as 'responsetime', 'clientip' are not showing in the search tab, however 'not_needed','http_status' and a few others are.
I removed the other field extractions entry thinking I only needed the delimExtraction.
Sample squid logs:
1302571599.112 32 10.10.10.10 TCP_DENIED/407 0 407 2581 CONNECT armmf.adobe.com:443 - -
1302571599.112 465 10.10.10.10 TCP_MISS/200 0 200 13314 GET http://www.ebay.com.au/ username 203.5.76.11
1302571599.115 0 10.10.10.10 TCP_DENIED/407 0 407 2415 CONNECT armmf.adobe.com:443 - -
1302571599.115 17 10.10.10.10 TCP_IMS_HIT/304 0 304 1302 GET http://vtr.elections.nsw.gov.au/images/eGlooApp.gif username -
1302571599.118 195 10.10.10.10 TCP_MISS/200 0 200 1729 GET http://toolbarqueries.google.com.au/tbr? username 10.10.10.10
1302571599.119 19 10.10.10.10 TCP_NEGATIVE_HIT/404 0 404 2459 GET http://vtr.elections.nsw.gov.au/css/mysource_files/arrow.png username -
1302571599.119 796 10.10.10.10 TCP_MISS/200 0 200 1734 GET http://t.adcloud.net/t.gif? username 10.10.10.10
1302571599.122 148 10.10.10.10 TCP_MISS/200 0 200 5050 GET http://someurl.net username 10.10.10.10
1302571599.122 22 10.10.10.10 TCP_IMS_HIT/304 0 304 1321 GET http://vtr.elections.nsw.gov.au/images/panel-sprite.png username -
I'd really like to just change the squid log format back to default, but we have a few apps using this weird format for some reason... I mean really why need the '0' and have the status code twice 😕
... View more