Getting Data In

Splunk is truncating the last key/value pair on line if the value contains a space

andrewcg
Path Finder

We are ingesting Aruba CearPass logs. The ClearPass Appliances send their syslog to a syslog server that writes the logs to disk and then reads those log lines into Splunk. The log lines look like:

 <143>2016-03-07 18:04:57,504 yyy.yyy.yyy.yyy CPPM_Dashboard_Summary 35249531 1 0 session_id=R022d8e6d-04-56de08db,req_source=RADIUS,user_name=user@local.domain,service_name=WIRELESS_LOCAL,alerts_present=0,nas_ip=xxx.xxx.xxx.xxx,nas_port=0,conn_status=Unknown,login_status=ACCEPT,error_code=0,mac_address=abcdef123456,timestamp=2016-03-07 18:03:55-05,write_timestamp=2016-03-07 18:03:56.93952-05

The last key value pair is "write_timestamp=2016-03-07 18:03:56.93952-05", but Splunk records it as "write_timestamp=2016-03-07" this affects other field as well:

<143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,timestamp=2016-03-07 18:08:07.536779-05

"timestamp=2016-03-07 18:08:07.536779-05" becomes "timestamp=2016-03-07"

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Space is one of the pair delimiter and if the string with spaces is not enclosed within double quotes. If possible have the data source send the logs with values enclosed within double quotes OR process in syslog to add double quotes. If none of that is possible, another option is to use transforms to do the field extractions. See this https://answers.splunk.com/answers/1377/field-extraction-with-kv-extract.html

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Space is one of the pair delimiter and if the string with spaces is not enclosed within double quotes. If possible have the data source send the logs with values enclosed within double quotes OR process in syslog to add double quotes. If none of that is possible, another option is to use transforms to do the field extractions. See this https://answers.splunk.com/answers/1377/field-extraction-with-kv-extract.html

0 Karma

andrewcg
Path Finder

I was hoping this would not be the case. I could either update rsyslog to use something like this:

http://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

$template doublequotelastfield,"%rawmsg:R,ERE,1,ZERO:(.*)=([^=,]+$)--end%=\"%rawmsg:R,ERE,2,ZERO:(.*)=([^=,]+$)--end%\"\n"

Which normal results in this:

<143>2016-03-08 14:58:30,800 136.167.0.15 CPPM_Proc_Stats 170 1 0 id=4540039,process_id=17,cpu_usage=0,res_mem_usage=4540,virt_mem_usage=185984,timestamp="2016-03-08 14:58:08.158684-05"

There are two side affects:

  1. It will add ="0" to the end of any line that does not have an equals sign in it. This is extremely unlikely, I am searching for this kind of event in the old data.
  2. Any log line that is truncated abnormally would have the

    <143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,times

Would become

<143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,times="0"

but that is a garbage line anyway.

Or we could update the SQL that ClearPass uses to generate the syslog data. The rsyslog seems to be the better option as all the attempts to add the quotes with concat to the SQL statements failed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...