topic Re: Sending rsyslog JSON format in Getting Data In

Sending rsyslog JSON format

shaharl — Tue, 13 Oct 2015 21:38:41 GMT

Hello,

I have tried today to integrate Splunk with Rsyslog that Contains JSON.
The issue is that rsyslog is sending the information with timestamp, hostname, and application name before the JSON message that my application made.

Re: Sending rsyslog JSON format

somesoni2 — Tue, 13 Oct 2015 21:55:02 GMT

Can you post some sample log entries?

Re: Sending rsyslog JSON format

esix_splunk — Wed, 14 Oct 2015 04:57:42 GMT

JSON and Syslog are different formats and dont particularly play well together due to the nature of the format of syslog. There are some custom rules you can compile for Rsyslog that will help with JSON.

BUT.. since you're using Splunk, why are you trying to ingest these with rsyslog? Why not use a Splunk Universal Forwarder and ingest the JSON files directly into Splunk. Splunk understands JSON format with no problems..

Re: Sending rsyslog JSON format

shaharl — Tue, 29 Sep 2020 07:33:58 GMT

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

Re: Sending rsyslog JSON format

stmyers7941 — Wed, 14 Oct 2015 12:42:13 GMT

I did this with a similar situation where a syslog header was added to the front of XML events. In props.conf, make a LINE_BREAKER setting for the sourcetype (or whatever) and add the regex for the syslog header, and end with the opening of your JSON (XML in my example):

`[sam:xml]
# Strips syslog header and makes events pure XML
LINE_BREAKER = (\d\s\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\s[\w_-]+\s[\w-_]+\s\d+\s-\s-\s)
SHOULD_LINEMERGE = false`

Obviously, your regex will vary. Post a sample event if you need help with the regex.