Getting Data In

Why is my props and transforms configuration not filtering out events from json data as expected?

kittle
New Member

I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else

Sample log data:

2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}

The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.

Props.conf

[json_mixed] 
NO_BINARY_CHECK = true
category = Custom 
disabled = false
pulldown_type = true 
TRANSFORMS-set = extractjson,setnull

Transforms.conf

[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw

[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue

I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.

To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.

With TRANSFORMS-set = extractjson,setnull the preview pane is empty.

With TRANSFORMS-set = setnull, extractjson the preview pane gives an error saying no events were found.

I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.

What am I missing here?

0 Karma

sanjay_shrestha
Contributor

Could you change the order as below:

TRANSFORMS-set = setnull, extractjson

From Document
the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

0 Karma

kittle
New Member

Still does not work. Instead of a blank preview pane I get the message "No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file."

0 Karma

sanjay_shrestha
Contributor

Wondering after changes, if you have added more json files in monitored folder. Changes will apply to new events only.

0 Karma

kittle
New Member

This is using the 'Upload files from my computer' feature, so it always uses new events.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...