Getting Data In

How to parse the Docker json logs?

jackin
Path Finder

Hi,

This is the log sent from Docker

("log":"[21:52:02] [/home/a143519/.local/share/code-server/extensions/ms-toolsai.jupyter-2021.9.1303320346]: Extension is not compatible with Code 1.66.2. Extension requires: 1.72.0.\n","stream":"stderr","time":"2023-03-06T21:52:02.2194402152"}{"log":"[21:52:02] [/home/a15 3509/.local/share/code-server/extensions/ms-python.vscode-pylance-2023. 1.10]: Extension is not compatible with Code 1.66.2. Extension req uires: 1.67.0.\n", "stream":"stderr","time": "2023-03-06T21:52:02.219891147Z")("log": "[21:52:02] [\u009cunknown\u009e][80d9f7e6][Extension HostConnection] New connection established.\n","stream":"stdout","time":"2023-03-06T21:52:02.604222684Z"){"log":"[21:52:02] [\u009cunknow n\u009e][80d9f7e6][ExtensionHostConnection] \u003c1453\u009e Launched Extension Host Process. \n","stream":"stdout","time":"2023-03-06T21: 52:02.617643295Z"]["log": "[IPC Library: Pty Host] INFO Persistent process "1": Replaying 505 chars and 1 size events\n","stream":"stdo ut", "time":"2023-03-06T21:52:06.9270320622"} ["log":"[IPC Library: Pty Host] WARN Shell integration cannot be enabled for executable \"/b in/bash and args undefined\n", "stream":"stdout","time":"2023-03-06T21:52:56.754368802Z"}{ log":"[21:57:00] [\u009cunknown\u009e][laf3f4 9a][ExtensionHostConnection] \u007c766\u007e Extension Host Process exited with code: 0, signal: null.\n","stream"stdout", "time":"2023- 03-06T21:57:00 839878031Z"}"log" [02:12:50] [\u009cunknown\u009e][abc26d01][ManagementConnection] The client has disconnected, will wai t for reconnection 3h before disposing...\n","stream":"stdout, "time":"2023-03-07T04:12:50. 7892655182")("log":"[05:12:59] [\u007cunknown \u007e][abf26c01][ManagementConnection] The reconnection grace time of 3h has expired, so the connection will be disposed. \n", "stream":"s tdout","time":"2023-03-07T05:12:59.567198587Z" log":[13:16:53] [\u003cunknown\u003e][adf26d01][ManagementConnection] Unknown reconnect ion token (seen before) \n","stream":"stderr","time":"2023-03-07T13:17:53 2951627292")("log":"[14:16:53] [\u003cunknown\u003e][90d9f9e6] [ExtensionHostConnection] The client has reconnected. \n","stream":"stdout", "time": "2023-03-07T13: 16:53.453120386Z")

Here is my props.conf :

 

auto learned

SHOULD LINEMERGE=false

LINE BREAKER=([\n\r]+)\s*("log":"{\n

NO BINARY CHECK-true

TIME PREFIX="time"

MAX TIMESTAMP LOOKAHEAD=48

TIME FORMAT=%Y-%m-%dT%H:%M:%S.9N%z

TRUNCATE=999999

CHARSET=UTF-8

KV MODE=json

ANNOTATE POINT=false

 

I have tried many different props.conf. Configurations but no luck.

Any help would be greatly appreciated!

Labels (4)
0 Karma

yeahnah
Motivator

Hi @jackin 

That's some messed up log output.  It looks like it should be JSON but is invalid (check here https://jsonlint.com/ ) for multiple reasons.

As a start, maybe look at the docker source that is producing the log output and fix it up so the output is in proper JSON format, then Splunk will just eat it up.

Otherwise, if you cannot change it, then I suggest you try and normalise the log output to look like JSON using some SEDCMD in props.conf first.  This should occur before line breaking so you can then have a generic rule once the log format is correct.

Hope this helps 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...