Getting Data In

Sending rsyslog JSON format

Engager

Hello,

I have tried today to integrate Splunk with Rsyslog that Contains JSON.
The issue is that rsyslog is sending the information with timestamp, hostname, and application name before the JSON message that my application made.

0 Karma
1 Solution

Engager

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

View solution in original post

0 Karma

Path Finder

I did this with a similar situation where a syslog header was added to the front of XML events. In props.conf, make a LINE_BREAKER setting for the sourcetype (or whatever) and add the regex for the syslog header, and end with the opening of your JSON (XML in my example):

`[sam:xml]
# Strips syslog header and makes events pure XML
LINE_BREAKER = (\d\s\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\s[\w_-]+\s[\w-_]+\s\d+\s-\s-\s)
SHOULD_LINEMERGE = false`

Obviously, your regex will vary. Post a sample event if you need help with the regex.

Engager

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

JSON and Syslog are different formats and dont particularly play well together due to the nature of the format of syslog. There are some custom rules you can compile for Rsyslog that will help with JSON.

BUT.. since you're using Splunk, why are you trying to ingest these with rsyslog? Why not use a Splunk Universal Forwarder and ingest the JSON files directly into Splunk. Splunk understands JSON format with no problems..

Revered Legend

Can you post some sample log entries?

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!