Hello,
I have tried today to integrate Splunk with Rsyslog that Contains JSON.
The issue is that rsyslog is sending the information with timestamp, hostname, and application name before the JSON message that my application made.
my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.
rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG
so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)
By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate
And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp
I did this with a similar situation where a syslog header was added to the front of XML events. In props.conf
, make a LINE_BREAKER
setting for the sourcetype (or whatever) and add the regex for the syslog header, and end with the opening of your JSON (XML in my example):
`[sam:xml] # Strips syslog header and makes events pure XML LINE_BREAKER = (\d\s\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\s[\w_-]+\s[\w-_]+\s\d+\s-\s-\s) SHOULD_LINEMERGE = false`
Obviously, your regex will vary. Post a sample event if you need help with the regex.
my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.
rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG
so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)
By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate
And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp
JSON and Syslog are different formats and dont particularly play well together due to the nature of the format of syslog. There are some custom rules you can compile for Rsyslog that will help with JSON.
BUT.. since you're using Splunk, why are you trying to ingest these with rsyslog? Why not use a Splunk Universal Forwarder and ingest the JSON files directly into Splunk. Splunk understands JSON format with no problems..
Can you post some sample log entries?