Extract Json from a log

[gcusello]

Hi at all,
I'm finding problems extracting fields from a json log using spath, I cannot use regexes because I have to use these fields in the Zimperium App Datamodel.
I already extracted json, but I don't know why I'm finding problems.

This is a sample:

<14>1 04 02 2020 17:02:22 UTC zconsole-xxxxxxxxxx-xxx44 {"system_token": "company-uat", "severity": 1, "event_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "forensics": {"zdid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "event_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "os": 1, "attack_time": {"$date": 1585846942000}, "general": [{"name": "Threat Type", "val": "DORMANT"}, {"name": "Action Triggered", "val": ""}], "threat_uuid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "type": 100}, "mitigated": false, "location": null, "eventtimestamp": "04 02 2020 17:02:22 UTC", "user_info": {"employee_name": "User03 Test", "user_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "user_role": "End User", "user_email": "[email protected]", "user_group": "__MTD_UAT"}, "device_info": {"tag1": "", "device_time": "03 30 2020 17:01:31 UTC", "app_version": "10.5.1.0.52R", "zdid": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "tag2": "", "os": "Android", "app": "MobileIron", "jailbroken": false, "operator": null, "os_version": "9", "mdm_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "imei": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "model": "SM-A530F", "device_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx", "type": "jackpotltexx", "zapp_instance_id": "xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx"}, "threat": {"story": "Inactive Device", "name": "Inactive Device", "general": {"action_triggered": "", "threat_type": "DORMANT"}}}

if I use the spath command I have an additional field called "14" containing all the event.

Problems started from the ingestion, because this log isn't recognized al json the guided ingestion.

Can anyone give me an idea how to do this?

Thank you in advance.

Ciao.
Giuseppe

[gcusello]

I reached to extract json using a regex:

index="mtd" sourcetype="mtd" 
| rex "[^\{]*(?<all>.*)"
| spath input=all

But i cannot use this search in Datamodel.

Ciao.
Giuseppe

[to4kawa]

| makeresults
| eval _raw="<14>1 04 02 2020 17:02:22 UTC zconsole-xxxxxxxxxx-xxx44 {\"system_token\": \"company-uat\", \"severity\": 1, \"event_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"forensics\": {\"zdid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"event_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"os\": 1, \"attack_time\": {\"$date\": 1585846942000}, \"general\": [{\"name\": \"Threat Type\", \"val\": \"DORMANT\"}, {\"name\": \"Action Triggered\", \"val\": \"\"}], \"threat_uuid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"type\": 100}, \"mitigated\": false, \"location\": null, \"eventtimestamp\": \"04 02 2020 17:02:22 UTC\", \"user_info\": {\"employee_name\": \"User03 Test\", \"user_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"user_role\": \"End User\", \"user_email\": \"[email protected]\", \"user_group\": \"__MTD_UAT\"}, \"device_info\": {\"tag1\": \"\", \"device_time\": \"03 30 2020 17:01:31 UTC\", \"app_version\": \"10.5.1.0.52R\", \"zdid\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"tag2\": \"\", \"os\": \"Android\", \"app\": \"MobileIron\", \"jailbroken\": false, \"operator\": null, \"os_version\": \"9\", \"mdm_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"imei\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"model\": \"SM-A530F\", \"device_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\", \"type\": \"jackpotltexx\", \"zapp_instance_id\": \"xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx\"}, \"threat\": {\"story\": \"Inactive Device\", \"name\": \"Inactive Device\", \"general\": {\"action_triggered\": \"\", \"threat_type\": \"DORMANT\"}}}"
| rex mode=sed "s/.*UTC \S+ //g"
| spath

props.conf

TIME_PREFIX = \<\d+\>\d\s
TIME_FORMAT = %m %d %Y %T %Z
SEDCMD-trim = s/.*UTC \S+ //g
INDEXED_EXTRACTIONS = json
KV_MODE = none

Hi, @gcusello
I don't know if it's a data model, but usually this is fine.