Getting Data In

How to parse the Docker json logs?

jackin
Path Finder

Hi,

This is the log sent from Docker

("log":"[21:52:02] [/home/a143519/.local/share/code-server/extensions/ms-toolsai.jupyter-2021.9.1303320346]: Extension is not compatible with Code 1.66.2. Extension requires: 1.72.0.\n","stream":"stderr","time":"2023-03-06T21:52:02.2194402152"}{"log":"[21:52:02] [/home/a15 3509/.local/share/code-server/extensions/ms-python.vscode-pylance-2023. 1.10]: Extension is not compatible with Code 1.66.2. Extension req uires: 1.67.0.\n", "stream":"stderr","time": "2023-03-06T21:52:02.219891147Z")("log": "[21:52:02] [\u009cunknown\u009e][80d9f7e6][Extension HostConnection] New connection established.\n","stream":"stdout","time":"2023-03-06T21:52:02.604222684Z"){"log":"[21:52:02] [\u009cunknow n\u009e][80d9f7e6][ExtensionHostConnection] \u003c1453\u009e Launched Extension Host Process. \n","stream":"stdout","time":"2023-03-06T21: 52:02.617643295Z"]["log": "[IPC Library: Pty Host] INFO Persistent process "1": Replaying 505 chars and 1 size events\n","stream":"stdo ut", "time":"2023-03-06T21:52:06.9270320622"} ["log":"[IPC Library: Pty Host] WARN Shell integration cannot be enabled for executable \"/b in/bash and args undefined\n", "stream":"stdout","time":"2023-03-06T21:52:56.754368802Z"}{ log":"[21:57:00] [\u009cunknown\u009e][laf3f4 9a][ExtensionHostConnection] \u007c766\u007e Extension Host Process exited with code: 0, signal: null.\n","stream"stdout", "time":"2023- 03-06T21:57:00 839878031Z"}"log" [02:12:50] [\u009cunknown\u009e][abc26d01][ManagementConnection] The client has disconnected, will wai t for reconnection 3h before disposing...\n","stream":"stdout, "time":"2023-03-07T04:12:50. 7892655182")("log":"[05:12:59] [\u007cunknown \u007e][abf26c01][ManagementConnection] The reconnection grace time of 3h has expired, so the connection will be disposed. \n", "stream":"s tdout","time":"2023-03-07T05:12:59.567198587Z" log":[13:16:53] [\u003cunknown\u003e][adf26d01][ManagementConnection] Unknown reconnect ion token (seen before) \n","stream":"stderr","time":"2023-03-07T13:17:53 2951627292")("log":"[14:16:53] [\u003cunknown\u003e][90d9f9e6] [ExtensionHostConnection] The client has reconnected. \n","stream":"stdout", "time": "2023-03-07T13: 16:53.453120386Z")

Here is my props.conf :

 

auto learned

SHOULD LINEMERGE=false

LINE BREAKER=([\n\r]+)\s*("log":"{\n

NO BINARY CHECK-true

TIME PREFIX="time"

MAX TIMESTAMP LOOKAHEAD=48

TIME FORMAT=%Y-%m-%dT%H:%M:%S.9N%z

TRUNCATE=999999

CHARSET=UTF-8

KV MODE=json

ANNOTATE POINT=false

 

I have tried many different props.conf. Configurations but no luck.

Any help would be greatly appreciated!

Labels (4)
0 Karma

yeahnah
Motivator

Hi @jackin 

That's some messed up log output.  It looks like it should be JSON but is invalid (check here https://jsonlint.com/ ) for multiple reasons.

As a start, maybe look at the docker source that is producing the log output and fix it up so the output is in proper JSON format, then Splunk will just eat it up.

Otherwise, if you cannot change it, then I suggest you try and normalise the log output to look like JSON using some SEDCMD in props.conf first.  This should occur before line breaking so you can then have a generic rule once the log format is correct.

Hope this helps 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...