Getting Data In

Sending rsyslog JSON format

shaharl
Engager

Hello,

I have tried today to integrate Splunk with Rsyslog that Contains JSON.
The issue is that rsyslog is sending the information with timestamp, hostname, and application name before the JSON message that my application made.

0 Karma
1 Solution

shaharl
Engager

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

View solution in original post

0 Karma

stmyers7941
Path Finder

I did this with a similar situation where a syslog header was added to the front of XML events. In props.conf, make a LINE_BREAKER setting for the sourcetype (or whatever) and add the regex for the syslog header, and end with the opening of your JSON (XML in my example):

`[sam:xml]
# Strips syslog header and makes events pure XML
LINE_BREAKER = (\d\s\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\s[\w_-]+\s[\w-_]+\s\d+\s-\s-\s)
SHOULD_LINEMERGE = false`

Obviously, your regex will vary. Post a sample event if you need help with the regex.

shaharl
Engager

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

JSON and Syslog are different formats and dont particularly play well together due to the nature of the format of syslog. There are some custom rules you can compile for Rsyslog that will help with JSON.

BUT.. since you're using Splunk, why are you trying to ingest these with rsyslog? Why not use a Splunk Universal Forwarder and ingest the JSON files directly into Splunk. Splunk understands JSON format with no problems..

somesoni2
SplunkTrust
SplunkTrust

Can you post some sample log entries?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...