topic Re: Updated : JSON event break for multiple source type and single event in multiple lines. in Getting Data In

Updated : JSON event break for multiple source type and single event in multiple lines.

AKG1_old1 — Mon, 21 May 2018 17:43:40 GMT

Hi,

I am uploading json file having 2 different source type and single event is split multiple lines.

I am very close to solution but having issue with a small problem.

Problem: I am not be able to use these together in props.conf. only 1 is working at a time.

1.
SEDCMD-remove_bracket = s/\[|\]//g
SEDCMD-remove_coma = s/\},/}\n/g
2.
MUST_BREAK_AFTER = \}

JSON logs after removing extra charaters(" [ , ] , }, ") to convert it in correct Json format

  {
      "name" : "CounterpartyNotional",
      "type" : "RiskBreakdown",
      "duration" : 20848,
      "count" : 1,
      "average" : 20848.0
    }
 {
      "name" : "CounterpartyPreSettlement",
      "type" : "RiskBreakdown",
      "duration" : 15370,
      "count" : 1,
      "average" : 15370.0
    }
     {
      "name" : "TraderCurrency",
      "type" : "Formula",
      "duration" : 344,
      "count" : 1,
      "average" : 344.0
    }

transform.conf

[reset_profiler_formula]
DEST_KEY = MetaData:Sourcetype
REGEX = \"type\" : \"RiskBreakdown\"
FORMAT = sourcetype::reset_profiler:risks

[reset_profiler_risks]
DEST_KEY = MetaData:Sourcetype
REGEX = \"type\" : \"Formula\"
FORMAT = sourcetype::reset_profiler:formula

prop.conf

[reset_profiler]
SEDCMD-remove_bracket = s/\[|\]//g
SEDCMD-remove_coma = s/\},/}\n/g
KV_MODE = json
NO_BINARY_CHECK = true
MUST_BREAK_AFTER = \}
TRANSFORMS-sourcetye_routing = reset_profiler_formula,reset_profiler_risks

[reset_profiler:formula]
KV_MODE= json

[reset_profiler:risks]
KV_MODE= json

Thanks

Re: Updated : JSON event break for multiple source type and single event in multiple lines.

somesoni2 — Mon, 21 May 2018 18:25:30 GMT

Give this a try

props.conf (on indexer/heavy forwarder whichever comes first)

[reset_profiler]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\r\n]+)(?=\[\s*\{)
DATETIME_CONFIG = CURRENT
TRANSFORMS-sourcetye_routing = reset_profiler_st_override

transforms.conf (on indexer/heavy forwarder whichever comes first)

[reset_profiler_st_override]
DEST_KEY = MetaData:Sourcetype
 REGEX =\"type\"\s*\:\s*\"([^\"]+)\"
 FORMAT = sourcetype::reset_profiler:$1

props.conf (on search head)

[reset_profiler:formula]
 KV_MODE= json

 [reset_profiler:risks]
 KV_MODE= json

Re: Updated : JSON event break for multiple source type and single event in multiple lines.

AKG1_old1 — Mon, 21 May 2018 21:46:56 GMT

@somesoni : Thanks for reply but now all events in one file taken as single event.