syslog facility and severity (loglevel)

[telecity]

I've added the following to etc/system/local/limits.conf

[udp://514]
no_priority_stripping = true
no_appending_timestamp = true

My interest is to retrieve the facility and severity (loglevel) from the incoming syslog events. However now each event is prefixed with <137> which means nothing to me. Here's an example:

<137>Sep 22 15:52:30 host...

Facility is set at local1 and level is alert. Per rfc3164 that'd be facility=17 and severity=1.

My questions:
1. What is <137> (it wasn't there
before, and does show up in _raw)?
2. How do I retrieve facility / severity? I'd like them to be indexed fields if possible, to make searching, sorting and alerting easier.

[Jason]

If you are using UDP directly in to Splunk (not our best practice, which is to use a separate syslog daemon and write to files, and have Splunk read the files)

and have no_priority_stripping turned on, then try my app.

http://splunk-base.splunk.com/apps/28634/syslog-priority-field-decoder-lookup

It will use a lookup to automatically create the syslog severity and facility names for you at search time.

[koshyk]

great app @Jason. Still very useful after 4+ years

[Jason]

you're welcome - seems like syslog hasn't changed very much over the years

[_smp_]

Thanks for creating this app - the config files worked great for me.

[Jason]

You're welcome. This reminded to update the listing to show it works with the latest versions of Splunk.

[are0002]

I founded it.

In newest version you have to edit inputs.conf

source: http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports?r=searchtip

[are0002]

Hi

I am using the last version of Splunk and I can not activate the priority level on my logs. I edited all limit.conf files with your code.

Regards,

[billshinn]

In RFC3164 priority (i.e. the required PRI part of the syslog packet (before the HEADER and MSG) is calculated by multiplying the facility by 8, then adding the severity. So per the RFC, where local1 = 17, therefore 17*8 = 136. Adding to that a 1 for the severity = alert, you get the 137 mentioned in the original post. The <137> is just on spec for a proper syslog message. As for extraction, you need to reverse the math. This explains it really, really well:

https://gist.github.com/1017480

As for field extraction, I think there must be a quick piece of code that can do that vs a table/matrix search.

[SEJeff]

Pretty print a table of this data in a nice "human friendly" format:
http://www.digitalprognosis.com/opensource/scripts/syslog-priorities.py

Example output:

            emergency   alert   critical   error   warning   notice   info   debug
kernel              0       1          2       3         4        5      6       7
user                8       9         10      11        12       13     14      15
mail               16      17         18      19        20       21     22      23
system             24      25         26      27        28       29     30      31
security           32      33         34      35        36       37     38      39
syslog             40      41         42      43        44       45     46      47
lpd                48      49         50      51        52       53     54      55
nntp               56      57         58      59        60       61     62      63
uucp               64      65         66      67        68       69     70      71
time               72      73         74      75        76       77     78      79
security           80      81         82      83        84       85     86      87
ftpd               88      89         90      91        92       93     94      95
ntpd               96      97         98      99       100      101    102     103
logaudit          104     105        106     107       108      109    110     111
logalert          112     113        114     115       116      117    118     119
clock             120     121        122     123       124      125    126     127
local0            128     129        130     131       132      133    134     135
local1            136     137        138     139       140      141    142     143
local2            144     145        146     147       148      149    150     151
local3            152     153        154     155       156      157    158     159
local4            160     161        162     163       164      165    166     167
local5            168     169        170     171       172      173    174     175
local6            176     177        178     179       180      181    182     183
local7            184     185        186     187       188      189    190     191

This should be easy enough to hack up.

[MillerTime]

One idea that comes to mind for making use of this data would be to transform this matrix into a CSV and use it as part of automatic lookup definition in order to provide additional context/knowledge about these events. Using the code (in your case 137) as the input field, and providing Facility and Severity as output fields.

I'm thinking columns like:

Code,Facility,Severity
1,kernel,0
2,kernel,1
...
18,mail,1
137,local1,0

http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups

[infinitiguy]

this was exactly what I was looking for

[telecity · Accepted solution]

After loads of searching, I think I've finally found the answer. Apparently splunk is using a matrix (which I suspected, but couldn't identify or confirm) that seems to match the following:

Severity        0   1   2   3   4   5   6   7
Facility
kernel  0   0   1   2   3   4   5   6   7
user    1   8   9   10  11  12  13  14  15
mail    2   16  17  18  19  20  21  22  23
system  3   24  25  26  27  28  29  30  31
security    4   32  33  34  35  36  37  38  39
syslog  5   40  41  42  43  44  45  46  47
lpd 6   48  49  50  51  52  53  54  55
nntp    7   56  57  58  59  60  61  62  63
uucp    8   64  65  66  67  68  69  70  71
time    9   72  73  74  75  76  77  78  79
security    10  80  81  82  83  84  85  86  87
ftpd    11  88  89  90  91  92  93  94  95
ntpd    12  96  97  98  99  100 101 102 103
logaudit    13  104 105 106 107 108 109 110 111
logalert    14  112 113 114 115 116 117 118 119
clock   15  120 121 122 123 124 125 126 127
local0  16  128 129 130 131 132 133 134 135
local1  17  136 137 138 139 140 141 142 143
local2  18  144 145 146 147 148 149 150 151
local3  19  152 153 154 155 156 157 158 159
local4  20  160 161 162 163 164 165 166 167
local5  21  168 169 170 171 172 173 174 175
local6  22  176 177 178 179 180 181 182 183
local7  23  184 185 186 187 188 189 190 191

Source of Matrix:
http://chris-mccafferty.blogspot.com/2010/12/syslog-priority-matrix.html

Here's a perl script to sort it out for you:

#!/usr/bin/perl -w
use strict;
# http://splunk-base.splunk.com/answers/31036/syslog-facility-and-severity-loglevel

my @facilities = qw(Kernel User Mail System Security Syslog Lpd Nntp Uucp Time
                        Security Ftpd Ntpd Logaudit Logalert Clock Local0 
                        Local1 Local2 Local3 Local4 Local5 Local6 Local7);
my @severities = qw(Emergency Alert Critical Error Warning Notice Info Debug);
my $count = 0;

foreach my $facility (@facilities) {
        foreach my $severity (@severities) {
                print("$count,$facility.$severity\n");
                $count++;
        }
}

[jchampagne]

The way I implemented this was to create a new App that contained my lookup csv as well as the tranforms.conf that references the lookup file. We'll call this "App-syslog-lookup". The permissions on that app were set to allow the files to be shared with all other Splunk apps. This allows me to reference the same lookup file from multiple apps.

I then edited the props.conf inside each app that I want to do a lookup with. As long as you have "App-syslog-lookup" shared with other apps, you can reference the stanza, syslog_facility_severity_codes, from your props.conf and do a lookup.

[jchampagne]

You must already be extracting the syslog code from your events as a field called "syslog_code". I added the following lines to my files to make that happen:

PROPS.conf
- Under the stanza for my sourcetype:
REPORT-ExtractSyslogCode = extract_syslog_code

TRANSFORMS.conf
- New stanza called [extract_syslog_code]
[extract_syslog_code]
REGEX = ^<(\d+)>
FORMAT = syslog_code::$1

[jchampagne]

1. Create a tranforms.conf that references your lookup:

[syslog_facility_severity_codes]
filename = syslog-codes.csv

1. In props.conf, add the following line to the stanza of any sourcetype that you want to do the lookup for:

LOOKUP-SyslogCode = syslog_facility_severity_codes code AS syslog_code OUTPUTNEW facility AS facility, severity AS severity

[jchampagne]

There is definitely a more "fancy" way of doing this via a scripted lookup that does a matrix search, but I didn't want to spend the time doing that...so here is what I did...

1. I created a CSV that contains all of the Syslog codes as well as two corresponding columns, one for the facility, and one for the severity. I'll see if I can post the whole thing in another message, but this is basically what it looks like:

code,facility,severity
0,kernel,emergency
1,kernel,alert
2,kernel,critical

[SyntecSeb]

You've only answered Q1 here. What about Q2? 2. How do I retrieve facility / severity? I'd like them to be indexed fields if possible, to make searching, sorting and alerting easier.