topic Re: Extract Json from a log in Getting Data In

Extract Json from a log

gcusello — Fri, 03 Apr 2020 15:02:48 GMT

Hi at all,
I'm finding problems extracting fields from a json log using spath, I cannot use regexes because I have to use these fields in the Zimperium App Datamodel.
I already extracted json, but I don't know why I'm finding problems.

This is a sample:

<14>1 04 02 2020 17:02:22 UTC zconsole-xxxxxxxxxx-xxx44 {"system_token": "company-uat", "severity": 1, "event_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "forensics": {"zdid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "event_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "os": 1, "attack_time": {"$date": 1585846942000}, "general": [{"name": "Threat Type", "val": "DORMANT"}, {"name": "Action Triggered", "val": ""}], "threat_uuid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "type": 100}, "mitigated": false, "location": null, "eventtimestamp": "04 02 2020 17:02:22 UTC", "user_info": {"employee_name": "User03 Test", "user_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "user_role": "End User", "user_email": "test.user03@company.com", "user_group": "__MTD_UAT"}, "device_info": {"tag1": "", "device_time": "03 30 2020 17:01:31 UTC", "app_version": "10.5.1.0.52R", "zdid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "tag2": "", "os": "Android", "app": "MobileIron", "jailbroken": false, "operator": null, "os_version": "9", "mdm_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "imei": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "model": "SM-A530F", "device_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "type": "jackpotltexx", "zapp_instance_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx"}, "threat": {"story": "Inactive Device", "name": "Inactive Device", "general": {"action_triggered": "", "threat_type": "DORMANT"}}}

if I use the spath command I have an additional field called "14" containing all the event.

Problems started from the ingestion, because this log isn't recognized al json the guided ingestion.

Can anyone give me an idea how to do this?

Thank you in advance.

Ciao.
Giuseppe

Re: Extract Json from a log

gcusello — Fri, 03 Apr 2020 15:21:03 GMT

I reached to extract json using a regex:

index="mtd" sourcetype="mtd" 
| rex "[^\{]*(?<all>.*)"
| spath input=all

But i cannot use this search in Datamodel.

Ciao.
Giuseppe

Re: Extract Json from a log

to4kawa — Fri, 03 Apr 2020 23:35:05 GMT

| makeresults
| eval _raw="<14>1 04 02 2020 17:02:22 UTC zconsole-xxxxxxxxxx-xxx44 {\"system_token\": \"company-uat\", \"severity\": 1, \"event_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"forensics\": {\"zdid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"event_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"os\": 1, \"attack_time\": {\"$date\": 1585846942000}, \"general\": [{\"name\": \"Threat Type\", \"val\": \"DORMANT\"}, {\"name\": \"Action Triggered\", \"val\": \"\"}], \"threat_uuid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"type\": 100}, \"mitigated\": false, \"location\": null, \"eventtimestamp\": \"04 02 2020 17:02:22 UTC\", \"user_info\": {\"employee_name\": \"User03 Test\", \"user_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"user_role\": \"End User\", \"user_email\": \"test.user03@company.com\", \"user_group\": \"__MTD_UAT\"}, \"device_info\": {\"tag1\": \"\", \"device_time\": \"03 30 2020 17:01:31 UTC\", \"app_version\": \"10.5.1.0.52R\", \"zdid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"tag2\": \"\", \"os\": \"Android\", \"app\": \"MobileIron\", \"jailbroken\": false, \"operator\": null, \"os_version\": \"9\", \"mdm_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"imei\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"model\": \"SM-A530F\", \"device_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"type\": \"jackpotltexx\", \"zapp_instance_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\"}, \"threat\": {\"story\": \"Inactive Device\", \"name\": \"Inactive Device\", \"general\": {\"action_triggered\": \"\", \"threat_type\": \"DORMANT\"}}}"
| rex mode=sed "s/.*UTC \S+ //g"
| spath

props.conf

TIME_PREFIX = \<\d+\>\d\s
TIME_FORMAT = %m %d %Y %T %Z
SEDCMD-trim = s/.*UTC \S+ //g
INDEXED_EXTRACTIONS = json
KV_MODE = none

Hi, @gcusello
I don't know if it's a data model, but usually this is fine.