I've added the following to etc/system/local/limits.conf
[udp://514]
no_priority_stripping = true
no_appending_timestamp = true
My interest is to retrieve the facility and severity (loglevel) from the incoming syslog events. However now each event is prefixed with <137> which means nothing to me. Here's an example:
<137>Sep 22 15:52:30 host...
Facility is set at local1 and level is alert. Per rfc3164 that'd be facility=17 and severity=1.
My questions:
1. What is <137> (it wasn't there
before, and does show up in _raw)?
2. How do I retrieve facility / severity? I'd like them to be indexed fields if possible, to make searching, sorting and alerting easier.
After loads of searching, I think I've finally found the answer. Apparently splunk is using a matrix (which I suspected, but couldn't identify or confirm) that seems to match the following:
Severity 0 1 2 3 4 5 6 7
Facility
kernel 0 0 1 2 3 4 5 6 7
user 1 8 9 10 11 12 13 14 15
mail 2 16 17 18 19 20 21 22 23
system 3 24 25 26 27 28 29 30 31
security 4 32 33 34 35 36 37 38 39
syslog 5 40 41 42 43 44 45 46 47
lpd 6 48 49 50 51 52 53 54 55
nntp 7 56 57 58 59 60 61 62 63
uucp 8 64 65 66 67 68 69 70 71
time 9 72 73 74 75 76 77 78 79
security 10 80 81 82 83 84 85 86 87
ftpd 11 88 89 90 91 92 93 94 95
ntpd 12 96 97 98 99 100 101 102 103
logaudit 13 104 105 106 107 108 109 110 111
logalert 14 112 113 114 115 116 117 118 119
clock 15 120 121 122 123 124 125 126 127
local0 16 128 129 130 131 132 133 134 135
local1 17 136 137 138 139 140 141 142 143
local2 18 144 145 146 147 148 149 150 151
local3 19 152 153 154 155 156 157 158 159
local4 20 160 161 162 163 164 165 166 167
local5 21 168 169 170 171 172 173 174 175
local6 22 176 177 178 179 180 181 182 183
local7 23 184 185 186 187 188 189 190 191
Source of Matrix:
http://chris-mccafferty.blogspot.com/2010/12/syslog-priority-matrix.html
Here's a perl script to sort it out for you:
#!/usr/bin/perl -w
use strict;
# http://splunk-base.splunk.com/answers/31036/syslog-facility-and-severity-loglevel
my @facilities = qw(Kernel User Mail System Security Syslog Lpd Nntp Uucp Time
Security Ftpd Ntpd Logaudit Logalert Clock Local0
Local1 Local2 Local3 Local4 Local5 Local6 Local7);
my @severities = qw(Emergency Alert Critical Error Warning Notice Info Debug);
my $count = 0;
foreach my $facility (@facilities) {
foreach my $severity (@severities) {
print("$count,$facility.$severity\n");
$count++;
}
}
If you are using UDP directly in to Splunk (not our best practice, which is to use a separate syslog daemon and write to files, and have Splunk read the files)
and have no_priority_stripping
turned on, then try my app.
http://splunk-base.splunk.com/apps/28634/syslog-priority-field-decoder-lookup
It will use a lookup to automatically create the syslog severity and facility names for you at search time.
great app @Jason. Still very useful after 4+ years
you're welcome - seems like syslog hasn't changed very much over the years
Thanks for creating this app - the config files worked great for me.
You're welcome. This reminded to update the listing to show it works with the latest versions of Splunk.
I founded it.
In newest version you have to edit inputs.conf
source: http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports?r=searchtip
Hi
I am using the last version of Splunk and I can not activate the priority level on my logs. I edited all limit.conf files with your code.
Regards,
In RFC3164 priority (i.e. the required PRI part of the syslog packet (before the HEADER and MSG) is calculated by multiplying the facility by 8, then adding the severity. So per the RFC, where local1 = 17, therefore 17*8 = 136. Adding to that a 1 for the severity = alert, you get the 137 mentioned in the original post. The <137> is just on spec for a proper syslog message. As for extraction, you need to reverse the math. This explains it really, really well:
https://gist.github.com/1017480
As for field extraction, I think there must be a quick piece of code that can do that vs a table/matrix search.
Pretty print a table of this data in a nice "human friendly" format:
http://www.digitalprognosis.com/opensource/scripts/syslog-priorities.py
Example output:
emergency alert critical error warning notice info debug
kernel 0 1 2 3 4 5 6 7
user 8 9 10 11 12 13 14 15
mail 16 17 18 19 20 21 22 23
system 24 25 26 27 28 29 30 31
security 32 33 34 35 36 37 38 39
syslog 40 41 42 43 44 45 46 47
lpd 48 49 50 51 52 53 54 55
nntp 56 57 58 59 60 61 62 63
uucp 64 65 66 67 68 69 70 71
time 72 73 74 75 76 77 78 79
security 80 81 82 83 84 85 86 87
ftpd 88 89 90 91 92 93 94 95
ntpd 96 97 98 99 100 101 102 103
logaudit 104 105 106 107 108 109 110 111
logalert 112 113 114 115 116 117 118 119
clock 120 121 122 123 124 125 126 127
local0 128 129 130 131 132 133 134 135
local1 136 137 138 139 140 141 142 143
local2 144 145 146 147 148 149 150 151
local3 152 153 154 155 156 157 158 159
local4 160 161 162 163 164 165 166 167
local5 168 169 170 171 172 173 174 175
local6 176 177 178 179 180 181 182 183
local7 184 185 186 187 188 189 190 191
This should be easy enough to hack up.
One idea that comes to mind for making use of this data would be to transform this matrix into a CSV and use it as part of automatic lookup definition in order to provide additional context/knowledge about these events. Using the code (in your case 137) as the input field, and providing Facility and Severity as output fields.
I'm thinking columns like:
Code,Facility,Severity
1,kernel,0
2,kernel,1
...
18,mail,1
137,local1,0
http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups
this was exactly what I was looking for
After loads of searching, I think I've finally found the answer. Apparently splunk is using a matrix (which I suspected, but couldn't identify or confirm) that seems to match the following:
Severity 0 1 2 3 4 5 6 7
Facility
kernel 0 0 1 2 3 4 5 6 7
user 1 8 9 10 11 12 13 14 15
mail 2 16 17 18 19 20 21 22 23
system 3 24 25 26 27 28 29 30 31
security 4 32 33 34 35 36 37 38 39
syslog 5 40 41 42 43 44 45 46 47
lpd 6 48 49 50 51 52 53 54 55
nntp 7 56 57 58 59 60 61 62 63
uucp 8 64 65 66 67 68 69 70 71
time 9 72 73 74 75 76 77 78 79
security 10 80 81 82 83 84 85 86 87
ftpd 11 88 89 90 91 92 93 94 95
ntpd 12 96 97 98 99 100 101 102 103
logaudit 13 104 105 106 107 108 109 110 111
logalert 14 112 113 114 115 116 117 118 119
clock 15 120 121 122 123 124 125 126 127
local0 16 128 129 130 131 132 133 134 135
local1 17 136 137 138 139 140 141 142 143
local2 18 144 145 146 147 148 149 150 151
local3 19 152 153 154 155 156 157 158 159
local4 20 160 161 162 163 164 165 166 167
local5 21 168 169 170 171 172 173 174 175
local6 22 176 177 178 179 180 181 182 183
local7 23 184 185 186 187 188 189 190 191
Source of Matrix:
http://chris-mccafferty.blogspot.com/2010/12/syslog-priority-matrix.html
Here's a perl script to sort it out for you:
#!/usr/bin/perl -w
use strict;
# http://splunk-base.splunk.com/answers/31036/syslog-facility-and-severity-loglevel
my @facilities = qw(Kernel User Mail System Security Syslog Lpd Nntp Uucp Time
Security Ftpd Ntpd Logaudit Logalert Clock Local0
Local1 Local2 Local3 Local4 Local5 Local6 Local7);
my @severities = qw(Emergency Alert Critical Error Warning Notice Info Debug);
my $count = 0;
foreach my $facility (@facilities) {
foreach my $severity (@severities) {
print("$count,$facility.$severity\n");
$count++;
}
}
The way I implemented this was to create a new App that contained my lookup csv as well as the tranforms.conf that references the lookup file. We'll call this "App-syslog-lookup". The permissions on that app were set to allow the files to be shared with all other Splunk apps. This allows me to reference the same lookup file from multiple apps.
I then edited the props.conf inside each app that I want to do a lookup with. As long as you have "App-syslog-lookup" shared with other apps, you can reference the stanza, syslog_facility_severity_codes, from your props.conf and do a lookup.
You must already be extracting the syslog code from your events as a field called "syslog_code". I added the following lines to my files to make that happen:
PROPS.conf
- Under the stanza for my sourcetype:
REPORT-ExtractSyslogCode = extract_syslog_code
TRANSFORMS.conf
- New stanza called [extract_syslog_code]
[extract_syslog_code]
REGEX = ^<(\d+)>
FORMAT = syslog_code::$1
[syslog_facility_severity_codes]
filename = syslog-codes.csv
LOOKUP-SyslogCode = syslog_facility_severity_codes code AS syslog_code OUTPUTNEW facility AS facility, severity AS severity
There is definitely a more "fancy" way of doing this via a scripted lookup that does a matrix search, but I didn't want to spend the time doing that...so here is what I did...
code,facility,severity
0,kernel,emergency
1,kernel,alert
2,kernel,critical
You've only answered Q1 here. What about Q2? 2. How do I retrieve facility / severity? I'd like them to be indexed fields if possible, to make searching, sorting and alerting easier.