topic Re: CSV file with last 2 fields XML payloads in Getting Data In

CSV file with last 2 fields XML payloads

odigokid — Mon, 23 Apr 2018 20:43:45 GMT

Need help with the following CSV (everything I am trying, the XML fields are getting parsed incorrectly)

so I have a CSV file with a header line and then data record

The last two fields - FullRequest, and FullResponse - are SOAP payloads which have \n and ',' in the payload - so splunk is treating the newline as a new event, and it's also chopping at the comma because that's the delimiter.

The other fields before these are what I would call your standard CSV fields in "","","","" - but as you can see some fields can be empty (i.e. ,"",)

so looking for approaches to parsing this log file.

Re: CSV file with last 2 fields XML payloads

ssadanala1 — Mon, 23 Apr 2018 20:47:55 GMT

posting a sample event will help

Re: CSV file with last 2 fields XML payloads

odigokid — Tue, 29 Sep 2020 19:12:09 GMT

I tried to attach but stated I don't have enough karma points - let me paste here. (I have not put all the data in the payloads due to customer data - but I have put a line there that has , in the data. and you see the "newline's" in the payloads.

LogType(v1.0),RootLogId,SubLogId,TransactionId,Instance,Operation,Status,User,Hostname,Protocol,Target,StartTime,ExecuteTime,ResponseCode,FullRequest,FullResponse
"southbound","PLP1EM01PL61804231005392658CAI3G1_2","/1/1/1","","","PGW_Create","SUCCESSFUL","","PLP1EM01PL6","SOAP","PGW-SNQ","2018-04-23 10.05.39.892","00 00:00:00.843","0","

 #### more tags and data - data can have comma's (ex. below)
<serviceType>serviceTypeId=0,OU=SERVICE,OU=UMA,NE=MOBILE_DATA_SERVER</serviceType>

 #### more tags and data - data can have comma's (ex. below)
<serviceType>serviceTypeId=0,OU=SERVICE,OU=UMA,NE=MOBILE_DATA_SERVER</serviceType>

Re: CSV file with last 2 fields XML payloads

woodcock — Mon, 23 Apr 2018 21:02:23 GMT

I generally use INDEXED_EXTRACTIONS which should work fine for your data:
http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Extractfieldsfromfileswithstructureddata

Re: CSV file with last 2 fields XML payloads

odigokid — Tue, 29 Sep 2020 19:12:13 GMT

Hi - this is my current props.conf which is not working

DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Image of what I am seeing on search head - the xml is getting broken on the newlines

Re: CSV file with last 2 fields XML payloads

odigokid — Mon, 23 Apr 2018 21:29:06 GMT

Image link - link text

Re: CSV file with last 2 fields XML payloads

odigokid — Mon, 23 Apr 2018 21:31:09 GMT

sorry link
http://www.screencast.com/t/dmOS7eCi6H