Random behaviour from splunk

[MikeKulls]

I'm sending syslog messages through to Splunk in field/value pairs. When the field value contains spaces or certain other characters we put double quotes around the value. When the value contains quotes we add a backslash before the quote, eg

Message=Closed
Message="It was closed"
Message="The dir \"datavol3\" was closed"

The problem is with the third item. Sometimes it will process the \" correctly as being an embedded quote and sometimes it will think it is the end of the string. So completely at random the value of the message field will be one of these:

The dir "datavol3" was closed     <----- correct value
The dir \                         <----- string cut short

I can send the same log through to Splunk 20 times and it parses the message correctly pretty much 50% of the time. Can anyone suggest why this might happen?

[MikeKulls · Accepted solution]

It appears the reason was the half the servers have different config on them. Being a large company it was difficult to get anyone to fix it so I just strip out the embedded quotes before sending the message across to Splunk. Not ideal but sometimes you just have to go with the simplest solution 🙂