Parse decoded HTTP data

[me74fhfd]

Hi all, 

I am new to Splunk and trying here to parse decoded HTTP data 
to table with unique fields like "Method", "URI", "Host", "X-Forwarded-IP" etc
I order to achieve this I was thinking to set unique separators between fields
and values but this is as far as I've got. Any suggestions how to do this better and
elegant are welcome. One of the other issues is that not every request is gonna have
same set of fields so have in mind that it can be variable, however majority is gonna be the same.

Thanks

SPL:
index="index2" EventType=type2
| base64 field=RequestContent action=decode mode=replace suppress_error=True
| rex field=RequestContent mode=sed "s/\\\x0d\\\x0a/\n/g"
| rex field=RequestContent mode=sed "s/ \//\nURI::/g"
| rex field=RequestContent mode=sed "s/ HTTP Version\//\nHTTP::/g"
| rex field=RequestContent mode=sed "s/\n\n/\n/g"
| rex field=RequestContent mode=sed "s/\n{/\nOther Info::{/g"
| rex field=RequestContent mode=sed "s/\n</\nOther Info::</g"
| rex field=RequestContent mode=sed "s/: /::/g"
| dedup RequestContent | where RequestContent!="None" | eval RequestContent = "Method::".RequestContent
| rex field=RequestContent mode=sed "s/\n/#/g" | table RequestContent

Original Request
GET /favicon.ico HTTP/1.1\x0d\x0aHost: 1.1.1.1\x0d\x0aX-Real-IP: 2.2.2.2\x0d\x0aX-Forwarded-For: 185.1.1.1\x0d\x0aX-Forwarded-Proto: https\x0d\x0aX-Forwarded-Port: 443\x0d\x0aX-Forwarded-Host: 2.2.2.2\x0d\x0aAccept: image/webp,image/apng,image/*,*/*;q=0.8\x0d\x0aCookie: IO_id_NewSearch_90_84_245_165=778528c061e04a3facd579a51c1ec341; IO_idts_NewSearch_90_84_245_165=1591695485841; bb96b56e607644689f860e05a8e775ef=WyIzODcyOTc3OTMyIl0; IO_refts_NewSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; IO_idvc_NewSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138\x0d\x0aPragma: no-cache\x0d\x0aReferer: https://90.84.245.165/\x0d\x0aUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36\x0d\x0aCache-Control: no-cache\x0d\x0aSec-Fetch-Dest: image\x0d\x0aSec-Fetch-Mode: no-cors\x0d\x0aSec-Fetch-Site: same-origin\x0d\x0aAccept-Encoding: gzip, deflate, br\x0d\x0aAccept-Language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7\x0d\x0aVia: proxy A\x0d\x0a\x0d\x0a

Modified Request by SPL
Method::GET#URI:::HTTP/1.1#Host::1.1.1.1#X-Real-IP::2.2.2.2#X-Forwarded-For::2.2.2.2#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::1.1.1.1#Accept::text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9#Cookie::HW_id_NewSearch_90_84_245_165=778528c061e0a51c1ec341; IO_idts_NewSearch_90_84_245_165=15485841; bb96b5775ef=WyIzODcyOTc3OTMyIl0; IO_refts_HuaweiSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; HW_idvc_HuaweiSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138#Referer::http://www.more.org/showconfirmpage/?url=https://1.1.1.1#User-Agent::Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36#Sec-Fetch-Dest::document#Sec-Fetch-Mode::navigate#Sec-Fetch-Site::cross-site#Sec-Fetch-User::?1#Accept-Encoding::gzip, deflate, br#Accept-Language::en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7#Upgrade-Insecure-Requests::1#Via::proxy A#
Method::POST#URI::getNewList/v1 HTTP/1.1#Host::noname-dre.dt.noname.com#X-Real-IP::23.3.3.3#X-Forwarded-For::21.9.9.30#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::searchnews-dre.dt.noname.com#Content-Length::415#Authorization::SDK-HMAC-SHA256 Access=183b7bff5e48403c8c07e07, SignedHeaders=content-type;hmactoken;host;x-sdk-date, Signature=e73e171196bf221d08b7a2e365607b751d0f25f2e88d4d892#X-Sdk-Date::202T150331Z#hmacToken::VqP83hXcAq/TqRFOarchlCtFh5G+o=#Content-Type::application/json#Accept-Encoding::gzip#User-Agent::okhttp/3.12.0#Other Info::{"transId":"961b5d9720db4078b8349ec","version":"10.1.2.200","deviceId":"4ff2e1c83f3b43a693bee925146c5af4","userId":"5190064000024056394","serviceToken":"","lang":"zh-cn","phoneModel":"JNY","locale":"cn","net":"1","sysVer":"EmotionUI_10.1.0","ts":"159259","cmdId":"refresh","cmdVer":null,"userGrant":null,"channelId":"topNews","region":null,"newsCount":"10","pageNumber":1,"lastExposeItems":null}

[richgalloway]

Check out the URL Parser (https://splunkbase.splunk.com/app/1545/) and URL Toolbox (https://splunkbase.splunk.com/app/2734) apps on splunkbase.

[me74fhfd · Accepted solution]

No, that was not usefull
I had to build my own parser for this purpose and extract all all of the field manually