Writing a parser for a difficult log entry

scottj1y — Fri, 17 Apr 2020 22:51:38 GMT

I have logs which are structure like such:

"There are no delimiters between
blocks since they are always 8-bytes
wide. The first byte following ^E
describes the field. Then the URI is
terminated by a ^E, and every field
begins with a letter"

In addition to that the log entries are URL Encoded and also contain key/value pairs which are separated with "=". Writing a props.conf entry for this is proving to be difficult. Any suggestions?

Edit: Here's an example of an entry

0afd165b5e8a6ea9000003e800000002/lite_klonk_ws/results_app_install?dp=kochava&ai=1189345596&mi=7270C021-C7E5-4F60-BF2F-C023815F8081&ar=validated_claim&arid=de632e12-7797-11ea-be9a-008cfa5b3d60-7fc7b21bb700\x{05}x0afd165b\x{05}Z4cc04\x{05}AB38975\x{01}A10.253.22.91\x{01}C10.253.22.91\x{01}D38975\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gApache-HttpClient/4.5.5 (Java/1.8.0_242)\x{05}Kapplication/json\x{05}mPOST 0ad2de265e8a6ea900000fa000000196/lite_klonk_ws/app-install?ai=sniper.honor.real3d.shooter.assassin.free.android&mi=da172421-d9bb-4bd0-9a96-fb203358c0cb&dp=adjust&id=a196563a53482b8b989229405cd97171-1586130599704&it=1586130599000&ua=an%3Dsniper.honor.real3d.shooter.assassin.free.android%3Bav%3D1.7.1%3Bon%3Dandroid%3Bov%3D9%3Bdo%3DGalaxyJ6&ip=130.193.195.36\x{05}x0ad2de26\x{05}Z4cc04\x{05}AB60156\x{01}A10.210.222.38\x{01}C10.210.222.38\x{01}D60156\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gYHC/1.0\x{05}Kapplication/json\x{05}mGET 0ad4f0325e8a6ea900000fa00000019c/lite_klonk_ws/in-app?a=8&.yp=10096341&dp=adjust&ai=deezer.android.app&mi=b3e39498-4cbc-4c51-ac40-a04e818233c6&js=no&ec=&ea=ActivatedApp&gv=0&gc=USD&id=efcd439004ca20026334173cfbdec4c9-1586130600022&et=1584361977000&ir=&ua=an%3Ddeezer.android.app%3Bav%3D6.1.21.66%3Bon%3Dandroid%3Bov%3D10%3Bdo%3DGalaxyS10%252B&ip=86.201.55.19\x{05}x0ad4f032\x{05}Z4cc04\x{05}AB42355\x{01}A10.212.240.50\x{01}C10.212.240.50\x{01}D42355\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gApache-HttpClient/4.5.5 (Java/1.8.0_242)\x{05}Kapplication/json\x{05}mGET 0ad2de265e8a6ea9000017700000016b/lite_klonk_ws/app-install?ai=530168168&mi=8AA4C40D-3D1E-4C66-A6AC-DCDCC8CB75D0&dp=kochava&id=0405235001YGZW3EM4GV751679890&it=1586130586000&ua=Mozilla%2F5.0+%28iPad%3B+CPU+OS+13_3_1+like+Mac+OS+X%29+AppleWebKit%2F605.1.15+%28KHTML%2C+like+Gecko%29+Mobile%2F15E148&ip=73.28.254.14\x{05}x0ad2de26\x{05}Z4cc04\x{05}AB60156\x{01}A10.210.222.38\x{01}C10.210.222.38\x{01}D60156\x{01}E10.215.175.36\x{01}F4080\x{05}wapi.attribution.example.net\x{05}gYHC/1.0\x{05}Kapplication/json\x{05}mGET

Re: Writing a parser for a difficult log entry

to4kawa — Fri, 17 Apr 2020 23:01:21 GMT

Do not quiz.
URL Encoded : use SEDCMD OR INGEST_EVAL
Writing a props.conf entry for this is proving to be difficult. : For you.

Re: Writing a parser for a difficult log entry

richgalloway — Sat, 18 Apr 2020 12:31:28 GMT

Some sample data would be helpful.

Re: Writing a parser for a difficult log entry

richgalloway — Wed, 22 Apr 2020 12:24:36 GMT

That's a hard data set to work with. I suggest starting with these props and then using URLToolbox at search time to get the rest.

[mysourcetype]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:GET|POST)()

Re: Writing a parser for a difficult log entry

scottj1y — Wed, 22 Apr 2020 14:28:28 GMT

Thanks Rich, that's at least a start.

topic Re: Writing a parser for a difficult log entry in Getting Data In

Writing a parser for a difficult log entry

Re: Writing a parser for a difficult log entry

Re: Writing a parser for a difficult log entry

Re: Writing a parser for a difficult log entry

Re: Writing a parser for a difficult log entry