Getting Data In
Highlighted

Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

Hello!

I have some json data being generated by a client-side tool:

{
    "name": "open_sockets",
    "hostIdentifier": "ip-172-30-1-242.ec2.internal",
    "calendarTime": "Tue May 24 10:37:31 2016 UTC",
    "unixTime": "1464086251",
    "columns": {
        "family": "2",
        "fd": "6",
        "local_address": "172.30.1.242",
        "local_port": "32886",
        "path": "",
        "pid": "547",
        "protocol": "17",
        "remote_address": "4.53.160.75",
        "remote_port": "123",
        "socket": "52263"
    },
    "action": "added"
}

When this data is dropped into a flat file on the client then picked up by the Splunk Universal Forwarder, the field extractions using the _json sourcetype work perfectly. I've since reconfigured the tool to push the data into Amazon S3 via Firehose, and the field extractions are no longer work using the _json sourcetype.

The data is unchanged. I've examined the raw logs in the S3 management console and they are the same structure as the previously indexed flat file with no additional data or formatting as far as I can tell.

I've tried a variety of regex in the BREAKONLYBEFORE, BREAKONLYBEFOREDATE, MUSTBREAK_AFTER, no effect.

I currently have two near identical clients forwarding this information: one using the Splunk UF and one using AWS Firehose, both with the _json sourcetype, the first works fine, the second does not!

I am editing sourcetypes using the GUI; we are imminently moving to Splunk Cloud, and I am training myself to cope with no shell access!

Thanks

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

As an update, I have created another sourcetype with the below in the SplunkTAaws app:

[osq2]
DATETIME_CONFIG = 
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
TRUNCATE = 0
category = Structured
pulldown_type = 1
BREAK_ONLY_BEFORE = (\{\"name\")/g
disabled = false

Still not getting event breaking. Suggestions welcomed!

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

SplunkTrust
SplunkTrust

JSON should linemerge, and I know you said you tried the _json sourcetype, but this is a copy of it i'd like you to try instead:

[osq2]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
category=Structured
disabled=false
pulldown_type=true
0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

Didn't work sadly...

I have discovered a difference between the two sources:

  • The flat file on disk, each json object begins on it's own line.

  • The AWS S3 source, all json events occur on the same line.

So, need a way to break events from a single line, where each json object begins with {"name":

I thought my regex would have done this, but clearly not.

Thanks.

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

SplunkTrust
SplunkTrust

Have you tried this regex instead?

'{"name"'

surrounded by single quotes...?
or even '\{"name"'

again surrounded by single quotes but escaping the {

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

Thanks, tried both but still not breaking.

Am I right in thinking the SHOULDLINEMERGE directive could be causing Splunk to assume that the entire block of data is a single event? In that case, shouldn't a matching regex in BREAKONLY_BEFORE override that and define the individual events?

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

SplunkTrust
SplunkTrust

Oh sorry you just hit the nail on the head

Your thinking is correct but let's try removing break only before, setting should line merge equal to false and use our regex as LINE_BREAKER instead.

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

SplunkTrust
SplunkTrust

Should linemerge = true and the break only here and there's are for tcp/udp inputs mainly. Line breakers are for when you don't have the standard carriage returns / line feeds. Now we might still have issues with indexed extractions and may need to use kv mode instead... Let's see. Sorry for the bad syntax I'm replying from a phone.

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

Thanks, attempting to force SHOULDLINEMERGE=false via the GUI keeps defaulting to "true" and adding a BREAKONLYBEFORE directive, which is annoying... have no console access at present to edit the props.conf, will do this tomorrow back in the office and let you know.

0 Karma
Highlighted

Re: Json event breaking no longer working since forwarding method changed from using a universal forwarder to AWS Firehose

Path Finder

So, currently running with the below in system/local/props.conf

[osq]
NOBINARYCHECK = true
disabled = false
KVMODE = none
SHOULD
LINEMERGE = false
LINE_BREAKER = {\"name/g

And still no breaking... Regex validated using http://www.regextester.com/:

alt text

alt text

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.