How to filter initial data from a file and process it as a JSON file?

bkumarm — Tue, 16 Feb 2016 11:46:33 GMT

I am reading an input from a TCP input that is coming as below. However, each event is having a set of data prepended which is causing the spath command to fail.
I wan to filter out "<01>- hostname " and provide the remaining part as input for spath command.
I need help in getting a regular expression to do it.

The sample input:

<01>- hostname {"name":"DefaultProfile","version":"1.0","isonjkpFormat":"yyyy-MM-dd'T'HH:mm:ss.SSSZ","type":"Event","category":"RT_FLOW_SESSION_CREATE_LS","protocolID":"6","sev":"1","src":"192.168.1.10","dst":"192.168.1.110","srcPort":"25253","dstPort":"80","ujnhbgtrf":"1","credibility":"1","startnjkpEaqer":"1455617975494","startnjkpISO":"2014-01-19T01:29:30.494-03:00","hnbjkiunjkpEaqer":"1455617975494","hnbjkiunjkpISO":"2014-01-19T04:19:35.494-06:00","mkiophirhbYH":"12abc22z-b812-22h7-234f-551016123jal","devnjkpEaqer":"1455639575000","iopHuimeNJI":"2014-01-19T10:19:35.000-06:00","qweLjctNAT":"192.168.1.210","pinBistYHG":"192.168.1.110","srcGyhNBHAsdn":"1","dstGyhNBHAsdn":"1","qweLjctNATPort":"-1234","pinBistYHGPort":"80","hasIdentity":"false","njukoan":"<01>2 2014-01-19T10:17:45.679Z abcd-efg2-9876z WT_FLOW - WT_FLOW_MJHGTYH_CREATE_BS [helps@2222.2.2.2.2.22 localis-compute-zxcv=\"ABC1\" lokmnb-qazwsxe=\"192.168.1.10\" gtfvbj-qwer=\"25253\" qwertyujhnq-address=\"192.168.1.110\" qwertyujhnq-port=\"80\" compute-zxcv=\"kudoi-http\" bat-mnbvcx-address=\"192.168.1.210\" bat-mnbvcx-port=\"78781\" bat-qwertyujhnq-address=\"192.168.1.110\" batqwertyujhnq-port=\"80\" abc-lkjhgfd-name=\"VX-TYU9-TYU-OPT-2\" pkt-lkjhgfd-name=\"None\" protocol-id=\"9\" asdfgh-name=\"JV-X-P786-QWER-BHV\" source-NMLP-name=\"Z-B768-QWER\" qwertyujhnq-NMLP-name=\"M-B890-POIU\" session-id-32=\"098765432\" username=\"N/A\" roles=\"N/A\" MLKIOP-MNJYHGTS-ASDRNJUPJ=\"NJKI3.9876\" application=\"UNKNOWN\" MKUNJI-application=\"UNKNOWN\" mnbhyujgt=\"UNKNOWN\"]","bgasbnJuh":"1","mnbIPOUN":"other","absIPPOL":"other","basghlothp":"false","muqaloID":"9","qwmnpName":"VV_MJOK_SESSION_AWBHNJ_PK","qwaszxmnlkpotyfh":"Session Nhjuyt","wrtyJhyblascvfght":"Access","eventDescription":"A nyhbgtfr basftio bhg inmuytr.","azcvftghbvgt":"acb","qweTgvfrt":"minj-bag6-7856ab-Hnqasui","abcPecpokk":"och-00-145-987.Net_11_4_5_6","mnbJhbpoiu":"other","oijDgfhbnIjkm":"Poijhgb PKM Mnbgftr Asdervhj Thbgfra","nhjRkyhcfBhytf":"MKI-PLO-ASW","thuHyrtfcQhbnjuytfv":"192.168.1.11"}

Re: How to filter initial data from a file and process it as a JSON file?

somesoni2 — Tue, 16 Feb 2016 18:03:22 GMT

You can use following props.conf configuration in your Indexer/Heavy Forwarder, to remove that prefix at index time

props.conf

[YourSourceType]
...other configurations...
SEDCMD-removeprefix = s/^([^\{]+)(\{.*)/2/g

To do this in your search in-line, try something like this

your base search | rex mode=sed "s/^([^\{]+)(\{.*)/2/g" | spath

Re: How to filter initial data from a file and process it as a JSON file?

bkumarm — Wed, 17 Feb 2016 07:05:42 GMT

The final Query that solved my requirement is

base search | rex field=_raw mode=sed "s/\<\d+\>\-[^\{]+//g"

this Query, even without spath command provides afield value exactly the way we need for analytics.

topic How to filter initial data from a file and process it as a JSON file? in Getting Data In

How to filter initial data from a file and process it as a JSON file?

Re: How to filter initial data from a file and process it as a JSON file?

Re: How to filter initial data from a file and process it as a JSON file?