How to set and configure the sourcetype to format events written to Splunk's HTTP event collector?

[nickcromwell87]

I'm having issues when writing events to Splunk's HTTP event collector. We have a good amount of existing queries that may need to be rewritten if this cannot be successful. The problem occurs when the POST content is in the following format:

{"event": "Tenant=\"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A\"\r\nDevice=\"1000\""}

The result is an escaped string in Splunk, which you can see in the attached photo. We need Splunk to unescape the escaped characters and insert line breaks where the Windows new line appears.

I believe this can be done by setting the sourcetype of the event and configuring it properly, but I haven't had any success thus far and I feel I've tried dozens of combinations of LINE_BREAKER and KV_MODE values.

Thanks

EDIT:

Here's the full Http POST request's content.

{
    "host":"TestHost",
    "sourcetype": "TestSourceType",
    "source": "TestSource",
    "event": "Tenant=\"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A\"\r\nDevice=\"1000\""
}

Attached 'TestSourceType' configuration below. Result is still the same with escape characters visible in the splunk event.

[gblock_splunk]

nickcromwell87 this is a known issue and has been fixed in cloud and in our next Splunk Enterprise release. One work around for now is you can send a JSON object as the "event" value i.e.

 {"event": {"tenant":"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A", "device":"1000"}}

Using this approach will also allow quoted strings in the field values i.e.

{"event": {"tenant":"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A", "device":"1000", "message":"\"quotes\""}}

This is an interim solution which as I mentioned has been fixed.

[gustavomichels]

What specific Splunk Enterprise version this is supposed to be fixed? Just tried with 6.3.3 and got the same escaped quotes issue, and I am trying to avoid using any workarounds.

[gblock_splunk]

It is fixed in the next version of Enterprise, not what is currently available.

[nickcromwell87 · Accepted solution]

The solution I've found to removing the escape characters is to use SEDCMD. In this case,

SEDCMD=s/(\\")/"/g

will remove the escape character before the double quote.

[jplumsdaine22]

Thanks for posting the additional information! I misunderstood what you were doing before.

Do you have any control over the script that is sending data to the event collector? I guess it is passing the string "Tenant=\"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A\"\r\nDevice=\"1000\"" to whatever function is making the calls to Splunk? by that I mean your code is like

myEvent= '"Tenant=\"FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A\"\r\nDevice=\"1000\"" '
postRequest(myEvent)

If that's the case just pass the string {"Tenant": "FA1248BC-FC3C-48CF-BC1A-AC07518BAD5A", "Device":"1000"} instead, and for the sourcetype just use json.

[gblock_splunk]

Don't use _json if you are sending via HTTP Event Collector as the sourcetype is configured to do JSON field extractions at index-time which does not work with HEC. If you leave the sourcetype as the default or use something that is not specifying JSON field extraction at index-time, then results will be as expected when you search / fields will get extracted at search time.

[jplumsdaine22]

Post your current props.conf for that sourcetype. Post the full HTTP POST request that you are adding data with.

Cheers

[nickcromwell87]

thanks, i updated it with them