Field Extraction using Regex

[alexspunkshell]

Hi There,

In my logs, the specific field "Other Parameters" contains a lot of logs. I want it to extract the logs and make a separate field for the logs.

Here I don' have access to Props & transforms. conf

I want to do by field extraction. Please help here.

I want new fields like - md5, pid, ppid, full_path, name, 

Sample Logs in the "Other Parameter" field

"Other Parameters":"payload={\"config_id\":5,\"config_rev_id\":13,\"finding\":{\"system_info\":{\"bits\":64,\"build_number\":\"19042\",\"os\":\"Microsoft Windows 10 Enterprise\",\"patch_level\":\"10.0.19042.0.0\",\"platform\":\"Windows\"}},\"intel_id\":209,\"match\":{\"contexts\":[{\"event\":{},\"process\":{\"uniqueEventId\":\"72057594037975619\",\"uniqueProcessId\":\"-7264562598978448809\"}},{\"event\":{},\"process\":{\"uniqueEventId\":\"72057594037975619\",\"uniqueProcessId\":\"-7264562598978448809\"}},{\"event\":{},\"process\":{\"uniqueEventId\":\"72057594037975619\",\"uniqueProcessId\":\"-7264562598978448809\"}}],\"hash\":2108229220,\"properties\":{\"args\":\"C:\\\\Windows\\\\Microsoft.NET\\\\Framework\\\\v4.0.30319\\\\regasm \\/codebase \\\"C:\\\\Windows\\\\SPEOFIPLAN\\\\Softland.CapitalHumano.Bridge.v701.dll\\\" \\/tlb:\\\"C:\\\\Windows\\\\SPEOFIPLAN\\\\Softland.CapitalHumano.Bridge.v701.tlb\\\"\",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\Microsoft.NET\\\\Framework\\\\v4.0.30319\\\\RegAsm.exe\",\"md5\":\"0d5df43af2916f47d00c1573797c1a13\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\Microsoft.NET\\\\Framework\\\\v4.0.30319\\\\RegAsm.exe\",\"parent\":{\"args\":\"\\\"C:\\\\WINDOWS\\\\System32\\\\cmd.exe\\\" \\/C \\\"C:\\\\Windows\\\\SPEOFIPLAN\\\\RegSCHv4.cmd\\\" \",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\System32\\\\cmd.exe\",\"md5\":\"8a2122e8162dbef04694b9c3e0b6cdee\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\System32\\\\cmd.exe\",\"parent\":{\"args\":\"C:\\\\WINDOWS\\\\Explorer.EXE\",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\explorer.exe\",\"md5\":\"5ea66ff5ae5612f921bc9da23bac95f7\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\explorer.exe\",\"parent\":{\"args\":\"C:\\\\Windows\\\\system32\\\\userinit.exe\",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\System32\\\\userinit.exe\",\"md5\":\"582a919ca5f944aa83895a5c633c122c\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\System32\\\\userinit.exe\",\"parent\":{\"args\":\"winlogon.exe\",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\System32\\\\winlogon.exe\",\"md5\":\"a987b43e6a8e8f894b98a3df022db518\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\System32\\\\winlogon.exe\",\"parent\":{\"args\":\"\\\\SystemRoot\\\\System32\\\\smss.exe 000000e4 00000084 \",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\System32\\\\smss.exe\",\"md5\":\"2c3f91bb4c0994a7b36ed0b6b14ec9c7\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\System32\\\\smss.exe\",\"parent\":{\"args\":\"\\\\SystemRoot\\\\System32\\\\smss.exe\",\"cwd\":null,\"file\":{\"fullpath\":\"C:\\\\Windows\\\\System32\\\\smss.exe\",\"md5\":\"2c3f91bb4c0994a7b36ed0b6b14ec9c7\",\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"C:\\\\Windows\\\\System32\\\\smss.exe\",\"parent\":{\"args\":\"\",\"cwd\":null,\"file\":{\"fullpath\":\"System\",\"md5\":null,\"sha1\":null,\"sha256\":null,\"size\":null},\"name\":\"System\",\"parent\":{\"pid\":null},\"pid\":4,\"ppid\":null,\"recorder_table_id\":null,\"recorder_unique_id\":\"3510362775707909737\",\"start_time\":\"2021-09-04T15:50:24Z\",\"user\":\"NT AUTHORITY\\\\SYSTEM\"},\"pid\":456,\"ppid\":4,\"recorder_table_id\":null,\"recorder_unique_id\":\"-6091327008692918503\",\"start_time\":\"2021-09-04T15:50:26Z\",\"user\":\"NT AUTHORITY\\\\SYSTEM\"},\"pid\":9032,\"ppid\":456,\"recorder_table_id\":null,\"recorder_unique_id\":\"-2820547760412538774\",\"start_time\":\"2021-09-04T15:58:34Z\",\"user\":\"NT AUTHORITY\\\\SYSTEM\"},\"pid\":14432,\"ppid\":9032,\"recorder_table_id\":null,\"recorder_unique_id\":\"4059013987027248497\",\"start_time\":\"2021-09-04T15:58:34Z\",\"user\":\"NT AUTHORITY\\\\SYSTEM\"},\"pid\":12896,\"ppid\":14432,\"recorder_table_id\":null,\"recorder_unique_id\":\"7583898902682956175\",\"start_time\":\"2021-09-04T15:59:19Z\",\"user\":\"Weir\\\\221924\"},\"pid\":1772,\"ppid\":12896,\"recorder_table_id\":null,\"recorder_unique_id\":\"5896339469594690517\",\"start_time\":\"2021-09-04T15:59:20Z\",\"user\":\"Weir\\\\221924\"},\"pid\":2032,\"ppid\":1772,\"recorder_table_id\":null,\"recorder_unique_id\":\"-594781336782834181\",\"start_time\":\"2021-09-04T17:03:14Z\",\"user\":\"Weir\\\\123121q\"},\"pid\":15172,\"ppid\":2032,\"recorder_table_id\":null,\"recorder_unique_id\":\"-7264562598978448809\",\"start_time\":\"2021-09-04T17:03:14Z\",\"user\":\"Weir\\\\123121q\"},\"source\":\"signals\",\"type\":\"process\",\"version\":1},\"service_id\":\"b909f782-b0ed-4979-aa9e-ee6fbe4ba5b2\"}"}
Show syntax highlighted

[ITWhisperer]

Try something like this

| makeresults
| eval _raw="{\"Other Parameters\":\"payload={\\\"config_id\\\":5,\\\"config_rev_id\\\":13,\\\"finding\\\":{\\\"system_info\\\":{\\\"bits\\\":64,\\\"build_number\\\":\\\"19042\\\",\\\"os\\\":\\\"Microsoft Windows 10 Enterprise\\\",\\\"patch_level\\\":\\\"10.0.19042.0.0\\\",\\\"platform\\\":\\\"Windows\\\"}},\\\"intel_id\\\":209,\\\"match\\\":{\\\"contexts\\\":[{\\\"event\\\":{},\\\"process\\\":{\\\"uniqueEventId\\\":\\\"72057594037975619\\\",\\\"uniqueProcessId\\\":\\\"-7264562598978448809\\\"}},{\\\"event\\\":{},\\\"process\\\":{\\\"uniqueEventId\\\":\\\"72057594037975619\\\",\\\"uniqueProcessId\\\":\\\"-7264562598978448809\\\"}},{\\\"event\\\":{},\\\"process\\\":{\\\"uniqueEventId\\\":\\\"72057594037975619\\\",\\\"uniqueProcessId\\\":\\\"-7264562598978448809\\\"}}],\\\"hash\\\":2108229220,\\\"properties\\\":{\\\"args\\\":\\\"C:\\\\\\\\Windows\\\\\\\\Microsoft.NET\\\\\\\\Framework\\\\\\\\v4.0.30319\\\\\\\\regasm \\\\/codebase \\\\\\\"C:\\\\\\\\Windows\\\\\\\\SPEOFIPLAN\\\\\\\\Softland.CapitalHumano.Bridge.v701.dll\\\\\\\" \\\\/tlb:\\\\\\\"C:\\\\\\\\Windows\\\\\\\\SPEOFIPLAN\\\\\\\\Softland.CapitalHumano.Bridge.v701.tlb\\\\\\\"\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\Microsoft.NET\\\\\\\\Framework\\\\\\\\v4.0.30319\\\\\\\\RegAsm.exe\\\",\\\"md5\\\":\\\"0d5df43af2916f47d00c1573797c1a13\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\Microsoft.NET\\\\\\\\Framework\\\\\\\\v4.0.30319\\\\\\\\RegAsm.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"\\\\\\\"C:\\\\\\\\WINDOWS\\\\\\\\System32\\\\\\\\cmd.exe\\\\\\\" \\\\/C \\\\\\\"C:\\\\\\\\Windows\\\\\\\\SPEOFIPLAN\\\\\\\\RegSCHv4.cmd\\\\\\\" \\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\cmd.exe\\\",\\\"md5\\\":\\\"8a2122e8162dbef04694b9c3e0b6cdee\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\cmd.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"C:\\\\\\\\WINDOWS\\\\\\\\Explorer.EXE\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\explorer.exe\\\",\\\"md5\\\":\\\"5ea66ff5ae5612f921bc9da23bac95f7\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\explorer.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"C:\\\\\\\\Windows\\\\\\\\system32\\\\\\\\userinit.exe\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\userinit.exe\\\",\\\"md5\\\":\\\"582a919ca5f944aa83895a5c633c122c\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\userinit.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"winlogon.exe\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\winlogon.exe\\\",\\\"md5\\\":\\\"a987b43e6a8e8f894b98a3df022db518\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\winlogon.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"\\\\\\\\SystemRoot\\\\\\\\System32\\\\\\\\smss.exe 000000e4 00000084 \\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\smss.exe\\\",\\\"md5\\\":\\\"2c3f91bb4c0994a7b36ed0b6b14ec9c7\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\smss.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"\\\\\\\\SystemRoot\\\\\\\\System32\\\\\\\\smss.exe\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\smss.exe\\\",\\\"md5\\\":\\\"2c3f91bb4c0994a7b36ed0b6b14ec9c7\\\",\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"C:\\\\\\\\Windows\\\\\\\\System32\\\\\\\\smss.exe\\\",\\\"parent\\\":{\\\"args\\\":\\\"\\\",\\\"cwd\\\":null,\\\"file\\\":{\\\"fullpath\\\":\\\"System\\\",\\\"md5\\\":null,\\\"sha1\\\":null,\\\"sha256\\\":null,\\\"size\\\":null},\\\"name\\\":\\\"System\\\",\\\"parent\\\":{\\\"pid\\\":null},\\\"pid\\\":4,\\\"ppid\\\":null,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"3510362775707909737\\\",\\\"start_time\\\":\\\"2021-09-04T15:50:24Z\\\",\\\"user\\\":\\\"NT AUTHORITY\\\\\\\\SYSTEM\\\"},\\\"pid\\\":456,\\\"ppid\\\":4,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"-6091327008692918503\\\",\\\"start_time\\\":\\\"2021-09-04T15:50:26Z\\\",\\\"user\\\":\\\"NT AUTHORITY\\\\\\\\SYSTEM\\\"},\\\"pid\\\":9032,\\\"ppid\\\":456,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"-2820547760412538774\\\",\\\"start_time\\\":\\\"2021-09-04T15:58:34Z\\\",\\\"user\\\":\\\"NT AUTHORITY\\\\\\\\SYSTEM\\\"},\\\"pid\\\":14432,\\\"ppid\\\":9032,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"4059013987027248497\\\",\\\"start_time\\\":\\\"2021-09-04T15:58:34Z\\\",\\\"user\\\":\\\"NT AUTHORITY\\\\\\\\SYSTEM\\\"},\\\"pid\\\":12896,\\\"ppid\\\":14432,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"7583898902682956175\\\",\\\"start_time\\\":\\\"2021-09-04T15:59:19Z\\\",\\\"user\\\":\\\"Weir\\\\\\\\221924\\\"},\\\"pid\\\":1772,\\\"ppid\\\":12896,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"5896339469594690517\\\",\\\"start_time\\\":\\\"2021-09-04T15:59:20Z\\\",\\\"user\\\":\\\"Weir\\\\\\\\221924\\\"},\\\"pid\\\":2032,\\\"ppid\\\":1772,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"-594781336782834181\\\",\\\"start_time\\\":\\\"2021-09-04T17:03:14Z\\\",\\\"user\\\":\\\"Weir\\\\\\\\123121q\\\"},\\\"pid\\\":15172,\\\"ppid\\\":2032,\\\"recorder_table_id\\\":null,\\\"recorder_unique_id\\\":\\\"-7264562598978448809\\\",\\\"start_time\\\":\\\"2021-09-04T17:03:14Z\\\",\\\"user\\\":\\\"Weir\\\\\\\\123121q\\\"},\\\"source\\\":\\\"signals\\\",\\\"type\\\":\\\"process\\\",\\\"version\\\":1},\\\"service_id\\\":\\\"b909f782-b0ed-4979-aa9e-ee6fbe4ba5b2\\\"}\"}"


| spath "Other Parameters"
| eval _raw='Other Parameters'
| extract
| spath input=payload

[alexspunkshell]

@ITWhisperer Thanks for your reply.

The above query gives the same raw log results. 

But I want to get a new field to be created for the raw log "Other Parameter" in interesting fields.

Example 

In "Other Parameter " field there are results for md5, pid,ppid 

I want a new field created like md5, pid,ppid 

Could you please help?

[ITWhisperer]

If you look at the text that I assigned to _raw, you will notice that I inserted an opening brace which wasn't in the text that you posted. If this was there in your original event and you have successfully extracted 'Other parameters', which the extracted fields would seem to suggest, then you could start with the eval _raw= to copy the 'Other parameters' field to _raw and use the extract command (which only works on the _raw field). You then should have a payload field, which you can use spath on to extract all the fields or specify the path you want for parts of the JSON string. (You may want to copy _raw to another field first if keeping the original _raw is important for the rest of the search.)