Splunk Search

Parse decoded HTTP data

me74fhfd
Path Finder

Hi all, 

I am new to Splunk and trying here to parse decoded HTTP data 
to table with unique fields like "Method", "URI", "Host", "X-Forwarded-IP" etc
I order to achieve this I was thinking to set unique separators between fields
and values but this is as far as I've got. Any suggestions how to do this better and
elegant are welcome. One of the other issues is that not every request is gonna have
same set of fields so have in mind that it can be variable, however majority is gonna be the same.

Thanks

SPL:
index="index2" EventType=type2
| base64 field=RequestContent action=decode mode=replace suppress_error=True
| rex field=RequestContent mode=sed "s/\\\x0d\\\x0a/\n/g"
| rex field=RequestContent mode=sed "s/ \//\nURI::/g"
| rex field=RequestContent mode=sed "s/ HTTP Version\//\nHTTP::/g"
| rex field=RequestContent mode=sed "s/\n\n/\n/g"
| rex field=RequestContent mode=sed "s/\n{/\nOther Info::{/g"
| rex field=RequestContent mode=sed "s/\n</\nOther Info::</g"
| rex field=RequestContent mode=sed "s/: /::/g"
| dedup RequestContent | where RequestContent!="None" | eval RequestContent = "Method::".RequestContent
| rex field=RequestContent mode=sed "s/\n/#/g" | table RequestContent

 

Original Request
GET /favicon.ico HTTP/1.1\x0d\x0aHost: 1.1.1.1\x0d\x0aX-Real-IP: 2.2.2.2\x0d\x0aX-Forwarded-For: 185.1.1.1\x0d\x0aX-Forwarded-Proto: https\x0d\x0aX-Forwarded-Port: 443\x0d\x0aX-Forwarded-Host: 2.2.2.2\x0d\x0aAccept: image/webp,image/apng,image/*,*/*;q=0.8\x0d\x0aCookie: IO_id_NewSearch_90_84_245_165=778528c061e04a3facd579a51c1ec341; IO_idts_NewSearch_90_84_245_165=1591695485841; bb96b56e607644689f860e05a8e775ef=WyIzODcyOTc3OTMyIl0; IO_refts_NewSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; IO_idvc_NewSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138\x0d\x0aPragma: no-cache\x0d\x0aReferer: https://90.84.245.165/\x0d\x0aUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36\x0d\x0aCache-Control: no-cache\x0d\x0aSec-Fetch-Dest: image\x0d\x0aSec-Fetch-Mode: no-cors\x0d\x0aSec-Fetch-Site: same-origin\x0d\x0aAccept-Encoding: gzip, deflate, br\x0d\x0aAccept-Language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7\x0d\x0aVia: proxy A\x0d\x0a\x0d\x0a

Modified Request by SPL
Method::GET#URI:::HTTP/1.1#Host::1.1.1.1#X-Real-IP::2.2.2.2#X-Forwarded-For::2.2.2.2#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::1.1.1.1#Accept::text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9#Cookie::HW_id_NewSearch_90_84_245_165=778528c061e0a51c1ec341; IO_idts_NewSearch_90_84_245_165=15485841; bb96b5775ef=WyIzODcyOTc3OTMyIl0; IO_refts_HuaweiSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; HW_idvc_HuaweiSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138#Referer::http://www.more.org/showconfirmpage/?url=https://1.1.1.1#User-Agent::Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36#Sec-Fetch-Dest::document#Sec-Fetch-Mode::navigate#Sec-Fetch-Site::cross-site#Sec-Fetch-User::?1#Accept-Encoding::gzip, deflate, br#Accept-Language::en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7#Upgrade-Insecure-Requests::1#Via::proxy A#
Method::POST#URI::getNewList/v1 HTTP/1.1#Host::noname-dre.dt.noname.com#X-Real-IP::23.3.3.3#X-Forwarded-For::21.9.9.30#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::searchnews-dre.dt.noname.com#Content-Length::415#Authorization::SDK-HMAC-SHA256 Access=183b7bff5e48403c8c07e07, SignedHeaders=content-type;hmactoken;host;x-sdk-date, Signature=e73e171196bf221d08b7a2e365607b751d0f25f2e88d4d892#X-Sdk-Date::202T150331Z#hmacToken::VqP83hXcAq/TqRFOarchlCtFh5G+o=#Content-Type::application/json#Accept-Encoding::gzip#User-Agent::okhttp/3.12.0#Other Info::{"transId":"961b5d9720db4078b8349ec","version":"10.1.2.200","deviceId":"4ff2e1c83f3b43a693bee925146c5af4","userId":"5190064000024056394","serviceToken":"","lang":"zh-cn","phoneModel":"JNY","locale":"cn","net":"1","sysVer":"EmotionUI_10.1.0","ts":"159259","cmdId":"refresh","cmdVer":null,"userGrant":null,"channelId":"topNews","region":null,"newsCount":"10","pageNumber":1,"lastExposeItems":null}

Labels (2)
0 Karma
1 Solution

me74fhfd
Path Finder

No, that was not usefull
I had to build my own parser for this purpose and extract all all of the field manually

View solution in original post

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust
Check out the URL Parser (https://splunkbase.splunk.com/app/1545/) and URL Toolbox (https://splunkbase.splunk.com/app/2734) apps on splunkbase.
---
If this reply helps you, an upvote would be appreciated.
0 Karma

me74fhfd
Path Finder

No, that was not usefull
I had to build my own parser for this purpose and extract all all of the field manually

View solution in original post

Tags (1)
0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!