topic Why is my props and transforms configuration not filtering out events from json data as expected? in Getting Data In

Why is my props and transforms configuration not filtering out events from json data as expected?

kittle — Thu, 05 Feb 2015 21:28:09 GMT

I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else

Sample log data:

2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}

The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.

Props.conf

[json_mixed] 
NO_BINARY_CHECK = true
category = Custom 
disabled = false
pulldown_type = true 
TRANSFORMS-set = extractjson,setnull

Transforms.conf

[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw

[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue

I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.

To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.

With TRANSFORMS-set = extractjson,setnull the preview pane is empty.

With TRANSFORMS-set = setnull, extractjson the preview pane gives an error saying no events were found.

I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.

What am I missing here?

Re: Why is my props and transforms configuration not filtering out events from json data as expected?

sanjay_shrestha — Thu, 05 Feb 2015 21:31:50 GMT

Could you change the order as below:

TRANSFORMS-set = setnull, extractjson

From Document
the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

Re: Why is my props and transforms configuration not filtering out events from json data as expected?

kittle — Thu, 05 Feb 2015 21:40:18 GMT

Still does not work. Instead of a blank preview pane I get the message "No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file."

Re: Why is my props and transforms configuration not filtering out events from json data as expected?

sanjay_shrestha — Thu, 05 Feb 2015 21:45:45 GMT

Wondering after changes, if you have added more json files in monitored folder. Changes will apply to new events only.

Re: Why is my props and transforms configuration not filtering out events from json data as expected?

kittle — Thu, 05 Feb 2015 21:49:46 GMT

This is using the 'Upload files from my computer' feature, so it always uses new events.