Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

[gary_richardson]

Hello!

I have some json data being generated by a client-side tool:

{
    "name": "open_sockets",
    "hostIdentifier": "ip-172-30-1-242.ec2.internal",
    "calendarTime": "Tue May 24 10:37:31 2016 UTC",
    "unixTime": "1464086251",
    "columns": {
        "family": "2",
        "fd": "6",
        "local_address": "172.30.1.242",
        "local_port": "32886",
        "path": "",
        "pid": "547",
        "protocol": "17",
        "remote_address": "4.53.160.75",
        "remote_port": "123",
        "socket": "52263"
    },
    "action": "added"
}

When this data is dropped into a flat file on the client then picked up by the Splunk Universal Forwarder, the field extractions using the _json sourcetype work perfectly. I've since reconfigured the tool to push the data into Amazon S3 via Firehose, and the field extractions are no longer work using the _json sourcetype.

The data is unchanged. I've examined the raw logs in the S3 management console and they are the same structure as the previously indexed flat file with no additional data or formatting as far as I can tell.

I've tried a variety of regex in the BREAK_ONLY_BEFORE, BREAK_ONLY_BEFORE_DATE, MUST_BREAK_AFTER, no effect.

I currently have two near identical clients forwarding this information: one using the Splunk UF and one using AWS Firehose, both with the _json sourcetype, the first works fine, the second does not!

I am editing sourcetypes using the GUI; we are imminently moving to Splunk Cloud, and I am training myself to cope with no shell access!

Thanks