topic Log ingested having xml tags that are not having fields extracted and named as per the tags? in Dashboards & Visualizations

Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Tue, 07 Jan 2020 23:25:12 GMT

• Attached you will see a sample of the log I am working with. <-- tried to attach it but not enough points. Haha!

• You will see in the body of the log row there are xml tags. Like CorrelationId, MessageId, DateTime, Status, Action, Message.

• I would think that Splunk would at least attempt to recognized these "xml tagged" field and name them.

• Is it not recognizing the xml tags because the FULL log is not xml tagged & the beginning few positions aren't xml tagged?

• Any thoughts or suggestions appreciated.

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

richgalloway — Wed, 08 Jan 2020 00:56:39 GMT

Please copy-and-paste some sample data.
Please also share the props.conf file settings for the sourcetype.

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Wed, 08 Jan 2020 01:06:40 GMT

Hi Rich - I will send it first thing tomorrow morning. Thank you for your attention to my question. Tim

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Wed, 30 Sep 2020 03:36:28 GMT

Log row....

000 2 00 0000 2020-01-08 13:52:48.771 (PickupStatusListener:41) Message payload: 4dba36a5-076b-4dc6-bde5-04366a4dcb78bf6cd527-fb0b-4f08-8920-f1c01d498764c4b3cc79-c4bb-4512-a3b7-f1527cf034911570a3c5-fab3-4ad3-ad13-5ed8831717122020-01-08T13:52:48.717-05:002020-01-08T13:52:27-05:00ReadReceiptRequest to remove pickup from manifest was received by the scanner.BRK_APPTransferRequestRemove314

Props.conf for sourcetype...

-bash-4.2$ cat props.conf
[kedex:cpc:log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{3}\s+\d{1}\s+\d{2}\s+\d{4}
NO_BINARY_CHECK=true
disabled=false
TIME_PREFIX=\s+\d{4}\s+
TIME_FORMAT=%F %T.%3N
TRUNCATE=999999
MAX_TIMESTAMP_LOOKAHEAD=35

[kedex:cpc:gc]
SHOULD_LINEMERGE=false
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=25
TRUNCATE=999999

[kedex:cpc:error]
SHOULD_LINEMERGE=false
TIME_PREFIX=^\d+\s+\d+\s+\d+\s+\d+\s+
LINE_BREAKER=([\r\n]+)\d+\s+\d+\s+\d+\s+\d+\s+\d{4}-\d{2}-\d{2}
TIME_FORMAT=%F %T.%3N
MAX_TIMESTAMP_LOOKAHEAD=25
TRUNCATE=999999

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

richgalloway — Mon, 13 Jan 2020 03:27:46 GMT

There are no XML tags in the sample log row.

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Tue, 14 Jan 2020 13:27:30 GMT

I am unable to upload an image of this log row - AND - when I copy and paste the text of the log row it strips out my xml tags. How else can I get the text or an image of this log row to you.

For some reason they say I do not have enough "points" to upload a small image of the log row.

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Tue, 14 Jan 2020 13:29:34 GMT

For some reason when I copy / paste the log row this field strips out my xml tags.
Also when I try to upload a small image of the log row this site tells me I do not have enough points to upload the file.
Therefore this site and any interaction thru the SPLUNK provided field for pasting data is virtually worthless!

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Tue, 14 Jan 2020 13:30:06 GMT

Is there ANY OTHER way I can get this image of the log row to you?

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Tue, 14 Jan 2020 13:31:00 GMT

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

richgalloway — Tue, 14 Jan 2020 13:39:35 GMT

To avoid losing XML tags and other formatting characters, enclose the text within backtics (`) or highlight it and click the code button (101010).

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Wed, 15 Jan 2020 11:38:57 GMT

Thanks Rich! I will remember that!! Here is the log row....

000 2 00 0000 2020-01-08 13:52:48.771 (PickupStatusListener:41) Message payload: <?xml version="1.0" encoding="UTF-8"?> <ns0:PickupStatus xmlns:ns0="http://cpc.ground.fedex.com/pickup/status/v1"><CorrelationId>4dba36a5-076b-4dc6-bde5-04366a4dcb78</CorrelationId><MessageId>bf6cd527-fb0b-4f08-8920-f1c01d498764</MessageId><ParentMessageId>c4b3cc79-c4bb-4512-a3b7-f1527cf03491</ParentMessageId><StatusForMessageId>1570a3c5-fab3-4ad3-ad13-5ed883171712</StatusForMessageId><DateTime>2020-01-08T13:52:48.717-05:00</DateTime><SourceSentDateTime>2020-01-08T13:52:27-05:00</SourceSentDateTime><Status>ReadReceipt</Status><Message>Request to remove pickup from manifest was received by the scanner.</Message><UserId>BRK_APP</UserId><RequestType>TransferRequest</RequestType><Action>Remove</Action><TerminalNumber>314</TerminalNumber></ns0:PickupStatus>

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Wed, 15 Jan 2020 11:40:29 GMT

 `   000 2 00 0000 2020-01-08 13:52:48.771 (PickupStatusListener:41) Message payload: <?xml version="1.0" encoding="UTF-8"?> <ns0:PickupStatus xmlns:ns0="http://cpc.ground.fedex.com/pickup/status/v1"><CorrelationId>4dba36a5-076b-4dc6-bde5-04366a4dcb78</CorrelationId><MessageId>bf6cd527-fb0b-4f08-8920-f1c01d498764</MessageId><ParentMessageId>c4b3cc79-c4bb-4512-a3b7-f1527cf03491</ParentMessageId><StatusForMessageId>1570a3c5-fab3-4ad3-ad13-5ed883171712</StatusForMessageId><DateTime>2020-01-08T13:52:48.717-05:00</DateTime><SourceSentDateTime>2020-01-08T13:52:27-05:00</SourceSentDateTime><Status>ReadReceipt</Status><Message>Request to remove pickup from manifest was received by the scanner.</Message><UserId>BRK_APP</UserId><RequestType>TransferRequest</RequestType><Action>Remove</Action><TerminalNumber>314</TerminalNumber></ns0:PickupStatus> `

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

timothytruax — Wed, 15 Jan 2020 11:44:15 GMT

What appears to be happening is the timestamp fields can vary in length and appearance; so when we attempt to use the extractor to create as field that is past one of those varying timestamps we get unmatched rows in the extractor.

Re: Log ingested having xml tags that are not having fields extracted and named as per the tags?

to4kawa — Wed, 15 Jan 2020 12:33:37 GMT

| makeresults 
| eval _raw="000 2 00 0000 2020-01-08 13:52:48.771 (PickupStatusListener:41) Message payload: <?xml version=\"1.0\" encoding=\"UTF-8\"?> <ns0:PickupStatus xmlns:ns0=\"http://cpc.ground.fedex.com/pickup/status/v1\"><CorrelationId>4dba36a5-076b-4dc6-bde5-04366a4dcb78</CorrelationId><MessageId>bf6cd527-fb0b-4f08-8920-f1c01d498764</MessageId><ParentMessageId>c4b3cc79-c4bb-4512-a3b7-f1527cf03491</ParentMessageId><StatusForMessageId>1570a3c5-fab3-4ad3-ad13-5ed883171712</StatusForMessageId><DateTime>2020-01-08T13:52:48.717-05:00</DateTime><SourceSentDateTime>2020-01-08T13:52:27-05:00</SourceSentDateTime><Status>ReadReceipt</Status><Message>Request to remove pickup from manifest was received by the scanner.</Message><UserId>BRK_APP</UserId><RequestType>TransferRequest</RequestType><Action>Remove</Action><TerminalNumber>314</TerminalNumber></ns0:PickupStatus>" 
| rex "(?<xml_data>(?=\<).+)"
| rex "(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3})" 
| spath input=xml_data
| fields - _* xml_data
| eval _time=strptime(time,"%F %T.%3Q")

Hi, @timothytruax
If there is data in _raw , it can be extracted like this.