Help parsing out events - need to get timestamp, host and sourcetype to rewrite metadata

[brent_weaver]

I have the following event:

{   [-] 
     ident:  vcap.cloud_controller_ng   
     message:     {"timestamp":1489461920.4637804,"message":"(0.000343s) SELECT * FROM `spaces` WHERE `id` = 479","log_level":"info","source":"cc.db","data":{"request_guid":"27de1815-57c0-41a5-63e1-614c44dfcac7::ef9ddb89-8aaf-46da-8843-902a91f95b44"},"thread_id":47430748867200,"fiber_id":47430746457820,"process_id":5696,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.29.0/lib/sequel/database/logging.rb","lineno":70,"method":"block in log_each"}  
     orig_host:  10.72.134.207  
     pid:    job=api_z2 index=1 
     pri:    14 
}

I need to use the ident field as the source type, get the timestamp out of the message line and set host as org_host field. I would also like to parse out the message field and make it kv pairs in son format. I have tried numerous things to no avail. How would you all approach this?

Any help is much appreciated!

[brent_weaver]

I solved this by the following:

props:

[test]
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = json
TRANSFORMS-updateMetaData = autosource, autohost, autoparse, fixsource

transforms:

[autosource]
SOURCE_KEY = field:ident
REGEX = (.*)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[autohost]
SOURCE_KEY = field:orig_host
REGEX = (.*)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[autoparse]
SOURCE_KEY = field:message
REGEX = (.*)
FORMAT = $1
DEST_KEY = _raw

[fixsource]
SOURCE_KEY = field:source
REGEX = (.*)
FORMAT = source::$1
DEST_KEY = MetaData:Source

I will admit the fixsource does not work but the rest does. I was missing the INDEXED_EXTRACTIONS = json

[woodcock]

Give this a try (there are SEVERAL differences from the other answer):

input.conf (on forwarder monitoring the file)

[monitor...
.....
sourcetype=someGenericName

props.conf (on heavy forwarder/Indexer)

[someGenericName]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?ms)(((?:^|[\r\n\s]+}[\r\n\s]+){[^\r\n]+[\r\n\s]+)|[\r\n\s]+})
TIME_PREFIX = {\"timestamp\"\:
TIME_FORMAT = %s.%N
MAX_TIMESTAMP_LOOKAHEAD = 18
TRANSFORMS-changest = set_sourcetype_my_log

transforms.conf (on heavy forwarder/Indexer)

[set_sourcetype_my_log]
REGEX = ident\s*:\s*(\S+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf (on Search Head)

[host::YourHostHere]
REPORT-extractcv = first_level_kvps, second_level_kvps, third_level_kvps

transforms.conf (on Search Head)

[first_level_kvps]
REGEX = (?ms)\s+([^\r\n:]+)\s*:\s*{?([^\r\n]+)(?<![}\s])
FORMAT = $1::$2

[second_level_kvps]
SOURCE_KEY = message
REGEX = (?ms)"([^"]+)":([^,]+)
FORMAT = $1::$2

[third_level_kvps]
SOURCE_KEY = data
REGEX = (?ms)"([^"]+)":([^,]+)
FORMAT = $1::$2

[somesoni2]

Give this a try

input.conf (on forwarder monitoring the file)

[monitor...
.....
sourcetype=someGenericName

props.conf (on heavy forwarder/Indexer)

[someGenericName]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\{\s*[\r\n]*ident:)
TIME_PREFIX = timestamp\"\:
TIME_FORMAT = %s.%N
MAX_TIMESTAMP_LOOKAHEAD = 18
TRANSFORMS-changest = set_sourcetype_my_log

transforms.conf (on heavy forwarder/Indexer)

[set_sourcetype_my_log]
REGEX = ident\:\s*(\S+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf (on Search Head)

[someGenericName]
REPORT-extractcv = colon_delim_fields

transforms.conf (on Search Head)

[colon_delim_fields]
DELIMS = ",", ":"

[brent_weaver]

Hey thanks for the info but it did not work. I am using a standalone instance in a VM on my laptop. But I did combine transforms and props since this is an indexer and sh.

Any other thoughts?

[brent_weaver]

*parsing not passing