Getting Data In

Writing a parser for a difficult log entry

scottj1y
Path Finder

I have logs which are structure like such:

"There are no delimiters between
blocks since they are always 8-bytes
wide. The first byte following ^E
describes the field. Then the URI is
terminated by a ^E, and every field
begins with a letter"

In addition to that the log entries are URL Encoded and also contain key/value pairs which are separated with "=". Writing a props.conf entry for this is proving to be difficult. Any suggestions?

Edit: Here's an example of an entry

0afd165b5e8a6ea9000003e800000002/lite_klonk_ws/results_app_install?dp=kochava&ai=1189345596&mi=7270C021-C7E5-4F60-BF2F-C023815F8081&ar=validated_claim&arid=de632e12-7797-11ea-be9a-008cfa5b3d60-7fc7b21bb700\x{05}x0afd165b\x{05}Z4cc04\x{05}AB38975\x{01}A10.253.22.91\x{01}C10.253.22.91\x{01}D38975\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gApache-HttpClient/4.5.5 (Java/1.8.0_242)\x{05}Kapplication/json\x{05}mPOST 0ad2de265e8a6ea900000fa000000196/lite_klonk_ws/app-install?ai=sniper.honor.real3d.shooter.assassin.free.android&mi=da172421-d9bb-4bd0-9a96-fb203358c0cb&dp=adjust&id=a196563a53482b8b989229405cd97171-1586130599704&it=1586130599000&ua=an%3Dsniper.honor.real3d.shooter.assassin.free.android%3Bav%3D1.7.1%3Bon%3Dandroid%3Bov%3D9%3Bdo%3DGalaxyJ6&ip=130.193.195.36\x{05}x0ad2de26\x{05}Z4cc04\x{05}AB60156\x{01}A10.210.222.38\x{01}C10.210.222.38\x{01}D60156\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gYHC/1.0\x{05}Kapplication/json\x{05}mGET 0ad4f0325e8a6ea900000fa00000019c/lite_klonk_ws/in-app?a=8&.yp=10096341&dp=adjust&ai=deezer.android.app&mi=b3e39498-4cbc-4c51-ac40-a04e818233c6&js=no&ec=&ea=ActivatedApp&gv=0&gc=USD&id=efcd439004ca20026334173cfbdec4c9-1586130600022&et=1584361977000&ir=&ua=an%3Ddeezer.android.app%3Bav%3D6.1.21.66%3Bon%3Dandroid%3Bov%3D10%3Bdo%3DGalaxyS10%252B&ip=86.201.55.19\x{05}x0ad4f032\x{05}Z4cc04\x{05}AB42355\x{01}A10.212.240.50\x{01}C10.212.240.50\x{01}D42355\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gApache-HttpClient/4.5.5 (Java/1.8.0_242)\x{05}Kapplication/json\x{05}mGET 0ad2de265e8a6ea9000017700000016b/lite_klonk_ws/app-install?ai=530168168&mi=8AA4C40D-3D1E-4C66-A6AC-DCDCC8CB75D0&dp=kochava&id=0405235001YGZW3EM4GV751679890&it=1586130586000&ua=Mozilla%2F5.0+%28iPad%3B+CPU+OS+13_3_1+like+Mac+OS+X%29+AppleWebKit%2F605.1.15+%28KHTML%2C+like+Gecko%29+Mobile%2F15E148&ip=73.28.254.14\x{05}x0ad2de26\x{05}Z4cc04\x{05}AB60156\x{01}A10.210.222.38\x{01}C10.210.222.38\x{01}D60156\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gYHC/1.0\x{05}Kapplication/json\x{05}mGET
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

That's a hard data set to work with. I suggest starting with these props and then using URLToolbox at search time to get the rest.

[mysourcetype]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:GET|POST)()
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

That's a hard data set to work with. I suggest starting with these props and then using URLToolbox at search time to get the rest.

[mysourcetype]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:GET|POST)()
---
If this reply helps you, Karma would be appreciated.

scottj1y
Path Finder

Thanks Rich, that's at least a start.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Some sample data would be helpful.

---
If this reply helps you, Karma would be appreciated.
0 Karma

to4kawa
Ultra Champion

Do not quiz.
URL Encoded : use SEDCMD OR INGEST_EVAL
Writing a props.conf entry for this is proving to be difficult. : For you.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...