Splunk is truncating the last key/value pair on line if the value contains a space

[andrewcg]

We are ingesting Aruba CearPass logs. The ClearPass Appliances send their syslog to a syslog server that writes the logs to disk and then reads those log lines into Splunk. The log lines look like:

 <143>2016-03-07 18:04:57,504 yyy.yyy.yyy.yyy CPPM_Dashboard_Summary 35249531 1 0 session_id=R022d8e6d-04-56de08db,req_source=RADIUS,user_name=user@local.domain,service_name=WIRELESS_LOCAL,alerts_present=0,nas_ip=xxx.xxx.xxx.xxx,nas_port=0,conn_status=Unknown,login_status=ACCEPT,error_code=0,mac_address=abcdef123456,timestamp=2016-03-07 18:03:55-05,write_timestamp=2016-03-07 18:03:56.93952-05

The last key value pair is "write_timestamp=2016-03-07 18:03:56.93952-05", but Splunk records it as "write_timestamp=2016-03-07" this affects other field as well:

<143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,timestamp=2016-03-07 18:08:07.536779-05

"timestamp=2016-03-07 18:08:07.536779-05" becomes "timestamp=2016-03-07"

[somesoni2 · Accepted solution]

Space is one of the pair delimiter and if the string with spaces is not enclosed within double quotes. If possible have the data source send the logs with values enclosed within double quotes OR process in syslog to add double quotes. If none of that is possible, another option is to use transforms to do the field extractions. See this https://answers.splunk.com/answers/1377/field-extraction-with-kv-extract.html

[andrewcg]

I was hoping this would not be the case. I could either update rsyslog to use something like this:

http://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

$template doublequotelastfield,"%rawmsg:R,ERE,1,ZERO:(.*)=([^=,]+$)--end%=\"%rawmsg:R,ERE,2,ZERO:(.*)=([^=,]+$)--end%\"\n"

Which normal results in this:

<143>2016-03-08 14:58:30,800 136.167.0.15 CPPM_Proc_Stats 170 1 0 id=4540039,process_id=17,cpu_usage=0,res_mem_usage=4540,virt_mem_usage=185984,timestamp="2016-03-08 14:58:08.158684-05"

There are two side affects:

1. It will add ="0" to the end of any line that does not have an equals sign in it. This is extremely unlikely, I am searching for this kind of event in the old data.

2. Any log line that is truncated abnormally would have the

   <143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,times

Would become

<143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,times="0"

but that is a garbage line anyway.

Or we could update the SQL that ClearPass uses to generate the syslog data. The rsyslog seems to be the better option as all the attempts to add the quotes with concat to the SQL statements failed.