How to parse the Docker json logs?

[jackin]

Hi,

This is the log sent from Docker

("log":"[21:52:02] [/home/a143519/.local/share/code-server/extensions/ms-toolsai.jupyter-2021.9.1303320346]: Extension is not compatible with Code 1.66.2. Extension requires: 1.72.0.\n","stream":"stderr","time":"2023-03-06T21:52:02.2194402152"}{"log":"[21:52:02] [/home/a15 3509/.local/share/code-server/extensions/ms-python.vscode-pylance-2023. 1.10]: Extension is not compatible with Code 1.66.2. Extension req uires: 1.67.0.\n", "stream":"stderr","time": "2023-03-06T21:52:02.219891147Z")("log": "[21:52:02] [\u009cunknown\u009e][80d9f7e6][Extension HostConnection] New connection established.\n","stream":"stdout","time":"2023-03-06T21:52:02.604222684Z"){"log":"[21:52:02] [\u009cunknow n\u009e][80d9f7e6][ExtensionHostConnection] \u003c1453\u009e Launched Extension Host Process. \n","stream":"stdout","time":"2023-03-06T21: 52:02.617643295Z"]["log": "[IPC Library: Pty Host] INFO Persistent process "1": Replaying 505 chars and 1 size events\n","stream":"stdo ut", "time":"2023-03-06T21:52:06.9270320622"} ["log":"[IPC Library: Pty Host] WARN Shell integration cannot be enabled for executable \"/b in/bash and args undefined\n", "stream":"stdout","time":"2023-03-06T21:52:56.754368802Z"}{ log":"[21:57:00] [\u009cunknown\u009e][laf3f4 9a][ExtensionHostConnection] \u007c766\u007e Extension Host Process exited with code: 0, signal: null.\n","stream"stdout", "time":"2023- 03-06T21:57:00 839878031Z"}"log" [02:12:50] [\u009cunknown\u009e][abc26d01][ManagementConnection] The client has disconnected, will wai t for reconnection 3h before disposing...\n","stream":"stdout, "time":"2023-03-07T04:12:50. 7892655182")("log":"[05:12:59] [\u007cunknown \u007e][abf26c01][ManagementConnection] The reconnection grace time of 3h has expired, so the connection will be disposed. \n", "stream":"s tdout","time":"2023-03-07T05:12:59.567198587Z" log":[13:16:53] [\u003cunknown\u003e][adf26d01][ManagementConnection] Unknown reconnect ion token (seen before) \n","stream":"stderr","time":"2023-03-07T13:17:53 2951627292")("log":"[14:16:53] [\u003cunknown\u003e][90d9f9e6] [ExtensionHostConnection] The client has reconnected. \n","stream":"stdout", "time": "2023-03-07T13: 16:53.453120386Z")

Here is my props.conf :

auto learned

SHOULD LINEMERGE=false

LINE BREAKER=([\n\r]+)\s*("log":"{\n

NO BINARY CHECK-true

TIME PREFIX="time"

MAX TIMESTAMP LOOKAHEAD=48

TIME FORMAT=%Y-%m-%dT%H:%M:%S.9N%z

TRUNCATE=999999

CHARSET=UTF-8

KV MODE=json

ANNOTATE POINT=false

I have tried many different props.conf. Configurations but no luck.

Any help would be greatly appreciated!

[yeahnah]

Hi @jackin 

That's some messed up log output.  It looks like it should be JSON but is invalid (check here https://jsonlint.com/ ) for multiple reasons.

As a start, maybe look at the docker source that is producing the log output and fix it up so the output is in proper JSON format, then Splunk will just eat it up.

Otherwise, if you cannot change it, then I suggest you try and normalise the log output to look like JSON using some SEDCMD in props.conf first.  This should occur before line breaking so you can then have a generic rule once the log format is correct.

Hope this helps