topic Trouble setting host from a hostname field in a json datastructure. in Getting Data In

Trouble setting host from a hostname field in a json datastructure.

gryz — Mon, 28 Sep 2020 12:30:13 GMT

Here is our props.conf:

[aristajson]
TIME_PREFIX = hosttime": "
MAX_TIMESTAMP_LOOKAHEAD = 22
BREAK_ONLY_BEFORE = {{"hostname
KV_MODE = json
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 90000
pulldown_type = 1
TRANSFORMS-larry = aristahostname

Here is our transforms.conf:

I've tried it with and without the host in <> . I've also tried to indicate the space after the : with a \s

[aristahostname]
REGEX = "hostname": "([a-zA-Z0-9-_]+)"
FORMAT = host::$1
DEST_KEY = MetaData:Host

Here is a snippet of our data that comes in via tcp. There is only one cr at the very end of the event:

{{"hostname": "nyaristalab-2"}{"hosttime": "2012-09-19 18:58:58"}{"neighbors": {"Ethernet3": {"2": {"sysName": "nyaristalab-1", ..... }

Here is what it kinda looks like in search. Each of the + are drill downable. So it is all good except for it not using what is in hostname as the host field.

1 » 9/25/12
10:25:59.000 PM
{[-]
hostname : "nyaristalab-2",
hosttime : "2012-09-25 22:25:59",
interfaces : {[+]},
neighbors : {[+]},
routing : {[+]}
}

Thanks for looking!

Re: Trouble setting host from a hostname field in a json datastructure.

kristian_kolb — Wed, 26 Sep 2012 07:33:08 GMT

In transforms.conf you should not specify <host> inside the capturing group. Have you tried;

[aristahome]
REGEX = \"hostname\":\s+\"([^"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

On a side note, your combination of SHOULD_LINEMERGE=false and BREAK_ONLY_BEFORE=... is invalid. BREAK_ONLY_BEFORE/AFTER, MUST_(NOT_)BREAK_BEFORE/AFTER etc will only work with SHOULD_LINEMERGE=true

If you have SHOULD_LINEMERGE=false, you set your event splitting with LINE_BREAKER= regex. The default value for LINE_BREAKER is one or more newlines, so for single-line events, you do not need to specify this.

If your event is multilined, you should probably set SHOULD_LINEMERGE = true and BREAK_ONLY_BEFORE_DATE = true. If this breaks events in a strange manner, because there are several date/timestamps in the events, you might have to set SHOULD_LINEMERGE = false, and use a LINE_BREAKER regex like

([\r\n]+)\d+/\d+/\d+\s+\d+:\d+:\d+

which should match a m/d/y h:m:s style timestamp after a newline

Hope this helps,

Kristian

Re: Trouble setting host from a hostname field in a json datastructure.

gryz — Mon, 28 Sep 2020 12:31:51 GMT

Thanks for the clearer explanation of the SHOULD_LINEMERGE and BREAK_ONLY_BEFORE. I got rid of those.

However the regex suggestions did not work. As I had noted in my original post, I had tried without the .

I checked both your and my regexes with various regex checkers and they both should work.

Any other ideas?