Getting Data In

How to filter initial data from a file and process it as a JSON file?

bkumarm
Contributor

HI

I am reading an input from a TCP input that is coming as below. However, each event is having a set of data prepended which is causing the spath command to fail.
I wan to filter out "<01>- hostname " and provide the remaining part as input for spath command.
I need help in getting a regular expression to do it.

The sample input:

<01>- hostname {"name":"DefaultProfile","version":"1.0","isonjkpFormat":"yyyy-MM-dd'T'HH:mm:ss.SSSZ","type":"Event","category":"RT_FLOW_SESSION_CREATE_LS","protocolID":"6","sev":"1","src":"192.168.1.10","dst":"192.168.1.110","srcPort":"25253","dstPort":"80","ujnhbgtrf":"1","credibility":"1","startnjkpEaqer":"1455617975494","startnjkpISO":"2014-01-19T01:29:30.494-03:00","hnbjkiunjkpEaqer":"1455617975494","hnbjkiunjkpISO":"2014-01-19T04:19:35.494-06:00","mkiophirhbYH":"12abc22z-b812-22h7-234f-551016123jal","devnjkpEaqer":"1455639575000","iopHuimeNJI":"2014-01-19T10:19:35.000-06:00","qweLjctNAT":"192.168.1.210","pinBistYHG":"192.168.1.110","srcGyhNBHAsdn":"1","dstGyhNBHAsdn":"1","qweLjctNATPort":"-1234","pinBistYHGPort":"80","hasIdentity":"false","njukoan":"<01>2 2014-01-19T10:17:45.679Z abcd-efg2-9876z WT_FLOW - WT_FLOW_MJHGTYH_CREATE_BS [helps@2222.2.2.2.2.22 localis-compute-zxcv=\"ABC1\" lokmnb-qazwsxe=\"192.168.1.10\" gtfvbj-qwer=\"25253\" qwertyujhnq-address=\"192.168.1.110\" qwertyujhnq-port=\"80\" compute-zxcv=\"kudoi-http\" bat-mnbvcx-address=\"192.168.1.210\" bat-mnbvcx-port=\"78781\" bat-qwertyujhnq-address=\"192.168.1.110\" batqwertyujhnq-port=\"80\" abc-lkjhgfd-name=\"VX-TYU9-TYU-OPT-2\" pkt-lkjhgfd-name=\"None\" protocol-id=\"9\" asdfgh-name=\"JV-X-P786-QWER-BHV\" source-NMLP-name=\"Z-B768-QWER\" qwertyujhnq-NMLP-name=\"M-B890-POIU\" session-id-32=\"098765432\" username=\"N/A\" roles=\"N/A\" MLKIOP-MNJYHGTS-ASDRNJUPJ=\"NJKI3.9876\" application=\"UNKNOWN\" MKUNJI-application=\"UNKNOWN\" mnbhyujgt=\"UNKNOWN\"]","bgasbnJuh":"1","mnbIPOUN":"other","absIPPOL":"other","basghlothp":"false","muqaloID":"9","qwmnpName":"VV_MJOK_SESSION_AWBHNJ_PK","qwaszxmnlkpotyfh":"Session Nhjuyt","wrtyJhyblascvfght":"Access","eventDescription":"A nyhbgtfr basftio bhg inmuytr.","azcvftghbvgt":"acb","qweTgvfrt":"minj-bag6-7856ab-Hnqasui","abcPecpokk":"och-00-145-987.Net_11_4_5_6","mnbJhbpoiu":"other","oijDgfhbnIjkm":"Poijhgb PKM Mnbgftr Asdervhj Thbgfra","nhjRkyhcfBhytf":"MKI-PLO-ASW","thuHyrtfcQhbnjuytfv":"192.168.1.11"}
0 Karma
1 Solution

somesoni2
Revered Legend

You can use following props.conf configuration in your Indexer/Heavy Forwarder, to remove that prefix at index time

props.conf

[YourSourceType]
...other configurations...
SEDCMD-removeprefix = s/^([^\{]+)(\{.*)/2/g

To do this in your search in-line, try something like this

your base search | rex mode=sed "s/^([^\{]+)(\{.*)/2/g" | spath

View solution in original post

somesoni2
Revered Legend

You can use following props.conf configuration in your Indexer/Heavy Forwarder, to remove that prefix at index time

props.conf

[YourSourceType]
...other configurations...
SEDCMD-removeprefix = s/^([^\{]+)(\{.*)/2/g

To do this in your search in-line, try something like this

your base search | rex mode=sed "s/^([^\{]+)(\{.*)/2/g" | spath

bkumarm
Contributor

The final Query that solved my requirement is

base search | rex field=_raw mode=sed "s/\<\d+\>\-[^\{]+//g" 

this Query, even without spath command provides afield value exactly the way we need for analytics.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...