Getting Data In

Why is my props and transforms configuration not filtering out events from json data as expected?

New Member

I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else

Sample log data:

2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}

The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.

Props.conf

[json_mixed] 
NO_BINARY_CHECK = true
category = Custom 
disabled = false
pulldown_type = true 
TRANSFORMS-set = extractjson,setnull

Transforms.conf

[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw

[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue

I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.

To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.

With TRANSFORMS-set = extractjson,setnull the preview pane is empty.

With TRANSFORMS-set = setnull, extractjson the preview pane gives an error saying no events were found.

I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.

What am I missing here?

0 Karma

Contributor

Could you change the order as below:

TRANSFORMS-set = setnull, extractjson

From Document
the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

0 Karma

New Member

Still does not work. Instead of a blank preview pane I get the message "No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file."

0 Karma

Contributor

Wondering after changes, if you have added more json files in monitored folder. Changes will apply to new events only.

0 Karma

New Member

This is using the 'Upload files from my computer' feature, so it always uses new events.

0 Karma