Splunk Search

Parse decoded HTTP data

me74fhfd
Path Finder

Hi all, 

I am new to Splunk and trying here to parse decoded HTTP data 
to table with unique fields like "Method", "URI", "Host", "X-Forwarded-IP" etc
I order to achieve this I was thinking to set unique separators between fields
and values but this is as far as I've got. Any suggestions how to do this better and
elegant are welcome. One of the other issues is that not every request is gonna have
same set of fields so have in mind that it can be variable, however majority is gonna be the same.

Thanks

SPL:
index="index2" EventType=type2
| base64 field=RequestContent action=decode mode=replace suppress_error=True
| rex field=RequestContent mode=sed "s/\\\x0d\\\x0a/\n/g"
| rex field=RequestContent mode=sed "s/ \//\nURI::/g"
| rex field=RequestContent mode=sed "s/ HTTP Version\//\nHTTP::/g"
| rex field=RequestContent mode=sed "s/\n\n/\n/g"
| rex field=RequestContent mode=sed "s/\n{/\nOther Info::{/g"
| rex field=RequestContent mode=sed "s/\n</\nOther Info::</g"
| rex field=RequestContent mode=sed "s/: /::/g"
| dedup RequestContent | where RequestContent!="None" | eval RequestContent = "Method::".RequestContent
| rex field=RequestContent mode=sed "s/\n/#/g" | table RequestContent

 

Original Request
GET /favicon.ico HTTP/1.1\x0d\x0aHost: 1.1.1.1\x0d\x0aX-Real-IP: 2.2.2.2\x0d\x0aX-Forwarded-For: 185.1.1.1\x0d\x0aX-Forwarded-Proto: https\x0d\x0aX-Forwarded-Port: 443\x0d\x0aX-Forwarded-Host: 2.2.2.2\x0d\x0aAccept: image/webp,image/apng,image/*,*/*;q=0.8\x0d\x0aCookie: IO_id_NewSearch_90_84_245_165=778528c061e04a3facd579a51c1ec341; IO_idts_NewSearch_90_84_245_165=1591695485841; bb96b56e607644689f860e05a8e775ef=WyIzODcyOTc3OTMyIl0; IO_refts_NewSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; IO_idvc_NewSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138\x0d\x0aPragma: no-cache\x0d\x0aReferer: https://90.84.245.165/\x0d\x0aUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36\x0d\x0aCache-Control: no-cache\x0d\x0aSec-Fetch-Dest: image\x0d\x0aSec-Fetch-Mode: no-cors\x0d\x0aSec-Fetch-Site: same-origin\x0d\x0aAccept-Encoding: gzip, deflate, br\x0d\x0aAccept-Language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7\x0d\x0aVia: proxy A\x0d\x0a\x0d\x0a

Modified Request by SPL
Method::GET#URI:::HTTP/1.1#Host::1.1.1.1#X-Real-IP::2.2.2.2#X-Forwarded-For::2.2.2.2#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::1.1.1.1#Accept::text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9#Cookie::HW_id_NewSearch_90_84_245_165=778528c061e0a51c1ec341; IO_idts_NewSearch_90_84_245_165=15485841; bb96b5775ef=WyIzODcyOTc3OTMyIl0; IO_refts_HuaweiSearch_90_84_245_165=1592407882547; APP_LANG=el-gr; APP_REGION=gr; HW_idvc_HuaweiSearch_90_84_245_165=16; IO_viewts__90_84_245_165=1592408854137; IO_viewts_NewSearch_90_84_245_165=1592408854138#Referer::http://www.more.org/showconfirmpage/?url=https://1.1.1.1#User-Agent::Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36#Sec-Fetch-Dest::document#Sec-Fetch-Mode::navigate#Sec-Fetch-Site::cross-site#Sec-Fetch-User::?1#Accept-Encoding::gzip, deflate, br#Accept-Language::en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7#Upgrade-Insecure-Requests::1#Via::proxy A#
Method::POST#URI::getNewList/v1 HTTP/1.1#Host::noname-dre.dt.noname.com#X-Real-IP::23.3.3.3#X-Forwarded-For::21.9.9.30#X-Forwarded-Proto::https#X-Forwarded-Port::443#X-Forwarded-Host::searchnews-dre.dt.noname.com#Content-Length::415#Authorization::SDK-HMAC-SHA256 Access=183b7bff5e48403c8c07e07, SignedHeaders=content-type;hmactoken;host;x-sdk-date, Signature=e73e171196bf221d08b7a2e365607b751d0f25f2e88d4d892#X-Sdk-Date::202T150331Z#hmacToken::VqP83hXcAq/TqRFOarchlCtFh5G+o=#Content-Type::application/json#Accept-Encoding::gzip#User-Agent::okhttp/3.12.0#Other Info::{"transId":"961b5d9720db4078b8349ec","version":"10.1.2.200","deviceId":"4ff2e1c83f3b43a693bee925146c5af4","userId":"5190064000024056394","serviceToken":"","lang":"zh-cn","phoneModel":"JNY","locale":"cn","net":"1","sysVer":"EmotionUI_10.1.0","ts":"159259","cmdId":"refresh","cmdVer":null,"userGrant":null,"channelId":"topNews","region":null,"newsCount":"10","pageNumber":1,"lastExposeItems":null}

Labels (2)
0 Karma
1 Solution

me74fhfd
Path Finder

No, that was not usefull
I had to build my own parser for this purpose and extract all all of the field manually

View solution in original post

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust
Check out the URL Parser (https://splunkbase.splunk.com/app/1545/) and URL Toolbox (https://splunkbase.splunk.com/app/2734) apps on splunkbase.
---
If this reply helps you, Karma would be appreciated.
0 Karma

me74fhfd
Path Finder

No, that was not usefull
I had to build my own parser for this purpose and extract all all of the field manually

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...