topic Re: Setting event time and host metadata from key/value pairs in Getting Data In

Setting event time and host metadata from key/value pairs

brent_weaver — Wed, 15 Feb 2017 17:45:09 GMT

I have this nice JSON event that has all the information I need in it, most namely timestamp and hostname of transaction. How do I rewrite the event to update metadata host and time? This cannot be that hard, the docs show REGEX but I would be able to just use fields.?!?!?!

Any help is much appreciated!

Re: Setting event time and host metadata from key/value pairs

jkat54 — Wed, 15 Feb 2017 18:06:11 GMT

Do this at index time (aka on forwarders & indexers)

Props.conf:

 [yourSourcetype]
TRANSFORMS-datetime=datetimeTransform
TRANSFORMS-hostname=hostnameTransform

Transforms.conf:

[datetimeTransform]
SOURCE_KEY = _raw
REGEX = some regex with a capture group that captures the datetime stamp
DEST_KEY = _time

[hostnameTransform]
SOURCE_KEY = _raw
REGEX = some regex with a capture group that captures host
DEST_KEY = MetaData:Host

See transforms.conf for a list of DEST_KEYs

Re: Setting event time and host metadata from key/value pairs

brent_weaver — Wed, 15 Feb 2017 18:22:43 GMT

Thanks for the response, do I have to use a REGEX to get the value, can't I just use the key from the key value pair?

Re: Setting event time and host metadata from key/value pairs

jkat54 — Wed, 15 Feb 2017 19:31:53 GMT

I suppose you could and if you absolutely have to have regex= to work... Just do (.*) as your regex.

Re: Setting event time and host metadata from key/value pairs

brent_weaver — Sun, 19 Feb 2017 16:07:03 GMT

Thank for the response. I guesss the question is, do I absolutely need to have REGEX? What I am contending with is a format where I have a time stand followed buy host IP address, the a fully intact JSON string. So In my JSON string I have the true time of the event as a field called timestamp.

My props is as follows:

[bv]
KV_MODE = json
INDEXTED_EXTRACTIONS = json
TRANSFORMS-extract = json_extraction
FIELDALIAS-conn_id = protocol.session_id AS conn_id

transforms.conf:

[json_extraction]
SOURCE_KEY = _raw
DEST_KEY = _raw
REGEX = ^([^{]+)({.+})$
FORMAT = $2

As you see I carve out the first two elements and only keep the JSON string. So this now leaves me with all the metadata I would need to. So how do I use these kv pairs to set this?