Getting Data In

Why is my props and transforms configuration not filtering out events from json data as expected?

kittle
New Member

I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else

Sample log data:

2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}

The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.

Props.conf

[json_mixed] 
NO_BINARY_CHECK = true
category = Custom 
disabled = false
pulldown_type = true 
TRANSFORMS-set = extractjson,setnull

Transforms.conf

[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw

[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue

I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.

To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.

With TRANSFORMS-set = extractjson,setnull the preview pane is empty.

With TRANSFORMS-set = setnull, extractjson the preview pane gives an error saying no events were found.

I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.

What am I missing here?

0 Karma

sanjay_shrestha
Contributor

Could you change the order as below:

TRANSFORMS-set = setnull, extractjson

From Document
the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

0 Karma

kittle
New Member

Still does not work. Instead of a blank preview pane I get the message "No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file."

0 Karma

sanjay_shrestha
Contributor

Wondering after changes, if you have added more json files in monitored folder. Changes will apply to new events only.

0 Karma

kittle
New Member

This is using the 'Upload files from my computer' feature, so it always uses new events.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...