Need help splitting an event at the Response line

[mninansplunk]

Hello,

I am new too Splunk and am needing to split an Event at the Response Line.  Below is an example of an Event.

Request : August 17, 2021, 4:50 pm
Data: {"requestNode":"Item","updatedBy":"WebServices_User","elements":{"typeOfItem":"Stock","country":"1",""baseUnitOfMeasure":"EA","IsItASerializedProduct":false,"currencyCode":"1","freezeCodeCorpLevel":98,"fractionalInventory":false,"isItADirectShippedProduct":false,"globalHold":false,"replacementCost":9.6,"productForm":"Non-Hazardous\/Transferrable","PrimaryVendor":"V9723","landedProduct":true,"standardCost":11.425,"status":"Inactive","priceGroup":"1N","invoiceCost":0,"listCost":11.99,"ueType":"Nursery","ueLine":"CNCO","ueDepartment":"EUONYMUS","taxCategory":"07"}}
Response: {"success":false,"message":"No valid Item exists","code":"205"}

The purpose is, I need to create Fields for each parameter in the Response Line, and with this line being a part of the Data portion of the Event, which has varying number of fields, we can't get the regex working.  Support said we needed to break out the Response line, but wouldn't offer any recommendation on which line breaker I should be using.

I've tried adding a BREAK_ONLY_BEFORE to the sourcetype in props.conf, but after a Splunk restart, we stop seeing events for this sourcetype.

Below is what the sourcetype looks like in props.conf.

[webservices_log-too_small]
BREAK_ONLY_BEFORE = ^[a-zA-Z](?:[_-]?\w)*:\s+\{"[a-zA-Z](?:[_-]?\w)*":
PREFIX_SOURCETYPE = True
is_valid = True
maxDist = 9999

Any help on this would be awesome, I really appreciate it.

Thanks,
Tom

[ITWhisperer]

From the example you have shown, it looks like the application is logging the request and response in the same event (not particularly useful if you want to know how long a request took as you have only one timestamp), so rather than trying to split the event, perhaps you could work on fixing the regex to extract the fields you want from the event.

What have you tried in this respect?

[mninansplunk]

Hello,

Thank you very much for the help, Support and I tried using the Field Extraction page and they said we need to use a Line Breaker since the Data portion of the event always has different # of fields and the regex will not work.

I am not a regex pro, so I honestly am not familiar with this. I did read that I might be able to achieve field extraction by configuring search time extraction. But, I see there's 3 types of these extractions, and I couldn't tell which one I could possibly use or the regex format for the 3 fields within the Response portion of the Event.

Thanks again,
Tom

[s2_splunk · Accepted solution]

Tom,

one more: If you can't get the log format changed at the source to something a bit more convenient, you can do something like this at search time:

<yourSearchForEvents> | rex "Response: (?<responseComponent>.+)" | spath input=responseComponent

Assuming your response message always starts with "Response: ", there is only one such string in each event and the Response is everything from there to the end of the line, it will pull out the whole response message and store it in a field called responseComponent. The following spath command will then take every element of that responseComponent (valid JSON) and convert it into a field/value pair. Field names will be the element names from the response. So, you should have three fields (given your example message):

If the response contains more than three elements, you'll see the additional fields as well.

[mninansplunk]

Thank you very much for the help.  This did the trick for me.  : )

[s2_splunk]

Let's assume you are in fact using LINE_BREAKER to split this into two individual events, how are you going to correlate a given response event to the correct request event at search time? I do agree that your better path here is to try to ensure the source application logs requests and responses separately and includes a unique requestId in both log messages, so that you can correlate requests and responses properly.

The other issue I see here is that the request and response log message is basically two separate (and invalid) JSON messages. Your life may get a bit simpler if it would be one valid JSON object and you can utilize builtin functionality to parse that properly.

[s2_splunk]

Here's a quick example of a valid JSON log message that should parse properly out of the (proverbial) box:

{
	"Request": {
		"timestamp": "August 17, 2021, 4:50 pm",

		"Data": {
			"requestNode": "Item",
			"updatedBy": "WebServices_User",
			"elements": {
				"typeOfItem": "Stock",
				"globalHold": false,
				".....omittted lines.....": "",
				"taxCategory": "07"
			}
		}
	},
	"Response": {
		"success": false,
		"message": "No valid Item exists",
		"code": "205"
	}
}

If you have the opportunity to effect this log format change, I would also recommend logging timestamps in xsd:dateTime format (including timezone). 😉