topic Re: Question on Line Breaker for XML log in Getting Data In

Question on Line Breaker for XML log

kkossery — Tue, 29 Sep 2020 09:47:44 GMT

Hi Experts,

We are trying to ingest a XML log file to splunk with the following data

2016-05-26 10:14:37 | R.R.services.section_B_C.save.saveSection_B_C | [I] | <?xml version="1.0"?>
<subsidary_id>1014</subsidary_id>
<is_first_filling>false</is_first_filling>
<system_id>8323</system_id>
<updated_by>username@email.com</updated_by>
<created_by>username@email.com</created_by>
2016-05-26 10:14:49 | R.R.services.section_A_K_N.save.saveSection_A_K_N | [I] | <?xml version="1.0"?>
<soa_cable_id>4913</soa_cable_id>
<accounting_period_id>1008</accounting_period_id>
<gross_recipts>8233</gross_recipts>
<contact_person_id>1095</contact_person_id>
<user_name>user@email.com</user_name>
<form_type_id>1079</form_type_id>
<form_type_value>short</form_type_value>
2016-05-26 10:14:58 | R.R.services.section_D.save.saveSection_D | [I] | <?xml version="1.0"?>
<community>
  <soa_city_id/>
  <soa_cable_id/>
  <is_first_community>true</is_first_community>
 ..
 ..

We are trying to see if we can extract the first block (savesection_B_C) and then the second block (savesection_A_K_N) etc.. and we think Line breaking this till these blocks will give us a way to build a report on Splunk.
Let us know if this makes sense.

Re: Question on Line Breaker for XML log

somesoni2 — Tue, 29 Sep 2020 09:49:08 GMT

Not sure if I understand the requirement completely. Do you want to have block with saveSection_B_C and saveSection_A_K_N to appear as one event?

Re: Question on Line Breaker for XML log

woodcock — Fri, 27 May 2016 21:09:24 GMT

Try this:

[YourSourcetypeHere]
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:M:%S

Re: Question on Line Breaker for XML log

kkossery — Tue, 29 Sep 2020 09:50:12 GMT

Thanks! I have a variation to the above question as we found out this further complicates matters on extraction. From the below code, can we force splunk to ignore the timestamps and select the whole block (start to end)? We would then run a xmlkv | table soa_cable_id etc.. to make a report.

2016-05-26 10:14:37 | R.R.services.section_B_C.save.saveSection_B_C | [I] | <?xml version="1.0"?>
<subsidary_id>1014</subsidary_id>
<is_first_filling>false</is_first_filling>
<system_id>8323</system_id>
<updated_by>user@email.com</updated_by>
<created_by>user@email.com</created_by>
2016-05-26 10:14:49 | R.R.services.section_A_K_N.save.saveSection_A_K_N | [I] | <?xml version="1.0"?>
<soa_cable_id>4917</soa_cable_id>
<accounting_period_id>1008</accounting_period_id>
<gross_recipts>8233</gross_recipts>
<contact_person_id>1095</contact_person_id>
<user_name>user@email.com</user_name>
<form_type_id>1079</form_type_id>
<form_type_value>short</form_type_value>
2016-05-26 10:14:58 | R.R.services.section_D.save.saveSection_D | [I] | <?xml version="1.0"?>
<community>
  <soa_city_id/>
  <soa_cable_id/>
  <is_first_community>true</is_first_community>
  <soa_city_name>vienna</soa_city_name>
  <county></county>
  <state_id>1001</state_id>
  <state>
    <state_id/>
    <state_name/>
    <country>
      <country_id/>
      <country_name/>
      <updated_by/>
      <updated_date/>
      <created_by/>
      <created_date/>
    </country>
    <country_id/>
    <updated_by/>
    <updated_date/>
    <created_by/>
    <created_date/>
  </state>
  <created_by/>
</community>
<soa_cable_id>4917</soa_cable_id>
<user_name>user@email.com</user_name>
2016-05-26 10:15:03 | R.R.services.section_E.save.saveSection_E | [I] | <?xml version="1.0"?>
<soa_cable_id>4917</soa_cable_id>
<user_name>user@email.com</user_name>
<secTransmissionTV>
  <res_svc_first_set_no_subscribers>8</res_svc_first_set_no_subscribers>
  <res_svc_first_set_no_rate>3243</res_svc_first_set_no_rate>
  <res_svc_addl_sets_no_subscribers></res_svc_addl_sets_no_subscribers>
  <res_svc_addl_sets_no_rate></res_svc_addl_sets_no_rate>
 <res_fm_no_subscribers></res_fm_no_subscribers>
  <res_fm_no_rate></res_fm_no_rate>
  <motel_hotel_subscribers></motel_hotel_subscribers>
  <motel_hotel_rate></motel_hotel_rate>
  <commercial_subscribers></commercial_subscribers>
  <commercial_rate></commercial_rate>
  <converter_subscribers/>
  <converter_rate/>
  <converter_resid_subscribers></converter_resid_subscribers>
  <converter_resid_rate></converter_resid_rate>
  <converter_non_resid_subscriber></converter_non_resid_subscriber>
  <converter_non_resid_rate></converter_non_resid_rate>
  <updated_by/>
  <updated_date/>
  <created_by/>
  <created_date/>
</secTransmissionTV>
2016-05-26 10:15:05 | R.R.services.section_F.save.saveSection_F | [I] | <?xml version="1.0"?>
<soa_cable_id>4917</soa_cable_id>
<user_name>user@email.com</user_name>
<otherSecTransmissionTV>
  <soa_cable_id/>
  <cont_svcs_pay_cable></cont_svcs_pay_cable>
  <cont_svcs_pay_cablepay_cable_addl_channel></cont_svcs_pay_cablepay_cable_addl_channel>
  <cont_svcs_fire_protection></cont_svcs_fire_protection>
  <cont_svcs_burglar_protection></cont_svcs_burglar_protection>
  <install_resi_first_set></install_resi_first_set>
  <install_resi_addl_set></install_resi_addl_set>
  <install_resi_fm_radio></install_resi_fm_radio>
  <install_resi_converter></install_resi_converter>
  <install_non_resi_motel></install_non_resi_motel>
  <install_non_resi_commercial></install_non_resi_commercial>
  <install_non_resi_pay_cable></install_non_resi_pay_cable>
  <install_non_resi_pay_cable_addl_channel></install_non_resi_pay_cable_addl_channel>
  <install_non_resi_fire_protection></install_non_resi_fire_protection>
  <install_non_resi_burglar_protection></install_non_resi_burglar_protection>
  <othr_svcs_reconnect></othr_svcs_reconnect>
  <othr_svcs_disconnect></othr_svcs_disconnect>
  <othr_svcs_outlet_relocation></othr_svcs_outlet_relocation>
  <othr_svcs_move_to_new_addr></othr_svcs_move_to_new_addr>
  <updated_by/>
  <updated_date/>
  <created_by/>
  <created_date/>
</otherSecTransmissionTV>
2016-05-26 10:15:19 | R.R.services.section_I.save.saveSection_I | [I] | <?xml version="1.0"?>
<soa_user_id>4917</soa_user_id>
<user_name>user@email.com</user_name>
<non_network_substitute_channels>false</non_network_substitute_channels>
2016-05-26 10:15:21 | R.R.services.section_L.save.saveSection_L | [I] | <?xml version="1.0"?>
<soa_user_id>4917</soa_user_id>
<user_name>user@email.com</user_name>
<royaltyFee>
  <soa_cable_id>4917</soa_cable_id>
  <accounting_period_id>1008</accounting_period_id>
  <gross_receipts>8233.00</gross_receipts>
  <fee>67.0</fee>
  <interest_amount>0.00</interest_amount>
  <filing_fee>15.00</filing_fee>
  <total_royalty_fee>67.00</total_royalty_fee>
  <form_type_id>1079</form_type_id>
  <is_partial_distant></is_partial_distant>
  <has_distant></has_distant>
  <submitted_date/>
  <shortform>
    <block1_1>52.00</block1_1>
    <block1_2/>
    <block2_1/>
    <block2_3/>
    <block2_5/>
    <block2_6/>
    <block2_7/>
    <block2_8/>
    <block2_9/>
    <block3_2/>
    <block3_3/>
    <block3_4/>
    <block3_5/>
    <block3_6/>
    <block3_7/>
  </shortform>
  <longform/>
  <interestAssesment>
    <late_payment_amount>52.00</late_payment_amount>
    <interest_rate></interest_rate>
    <no_of_days_late>0</no_of_days_late>
    <sectionQ_line2>0.00</sectionQ_line2>
    <sectionQ_line3>0.00</sectionQ_line3>
    <interest_amount>0.00</interest_amount>
  </interestAssesment>
</royaltyFee>
2016-05-26 10:15:25 | R.R.services.section_M.save.saveSection_M | [I] | <?xml version="1.0"?>
<soa_user_id>4917</soa_user_id>
<total_number_of_channels>1</total_number_of_channels>
<activated_channel_count>2</activated_channel_count>
<user_name>user@email.com</user_name>
2016-05-26 10:15:25 | R.R.services.section_M.save.saveSection_M | [I] | <?xml version="1.0"?>
<soa_user_id>4917</soa_user_id>
<total_number_of_channels>1</total_number_of_channels>
<activated_channel_count>2</activated_channel_count>
<user_name>user@email.com</user_name>
2016-05-26 10:15:27 | R.R.services.section_P.save.saveSection_P | [I] | <?xml version="1.0"?>
<soa_user_id>4917</soa_user_id>
<is_secondary_transmission_satellite>false</is_secondary_transmission_satellite>
<satellite_gross_recipts></satellite_gross_recipts>
<user_name>user@email.com</user_name>
2016-05-26 10:15:34 | R.R.services.section_O.save.saveSection_O | [I] | <?xml version="1.0"?>
<certified>
  <certified_by_type_id>1041</certified_by_type_id>
  <user_id>1095</user_id>
  <user_name>user@email.com</user_name>
  <soa_cable_id>4917</soa_cable_id>
</certified>

Re: Question on Line Breaker for XML log

kkossery — Wed, 01 Jun 2016 11:22:12 GMT

Thank you for your response. We are now looking at selecting the whole block of XML code instead of breaking it by time stamp. See my response below to woodcock. Thank you again.

Re: Question on Line Breaker for XML log

alemarzu — Wed, 01 Jun 2016 12:41:48 GMT

Passing a false value to TIME_PREFIX parameter could work, splunk wont be able to find the timestamp, hence one big event with the current timestamp.

With the given sample data, this worked for me.

[<SOURCETYPE NAME> ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
TIME_PREFIX=asdasd

Hope it helps.

Re: Question on Line Breaker for XML log

woodcock — Wed, 01 Jun 2016 13:21:35 GMT

Whatever you are doing merged, can surely also be done with properly line-broken events. If this answer properly breaks the lines, let's work from there.

Re: Question on Line Breaker for XML log

kkossery — Tue, 29 Sep 2020 09:50:18 GMT

Thanks for your response. In your opinion what is the best way to break this XML file so that we can construct a table with

[Date]              [SOA_Cable_ID]            [Created_by]

The problem with the XML is there are multiple tags for Soa_cable_id, dates etc.. that we are not able to discern how to create the report.

Re: Question on Line Breaker for XML log

kkossery — Wed, 01 Jun 2016 15:03:16 GMT

Thanks! This is a good idea. I will follow this if nothing else works.

Re: Question on Line Breaker for XML log

woodcock — Wed, 01 Jun 2016 23:43:07 GMT

If properly linebroken, then you can get the first one with SOA_Cable_ID=mvindex(SOA_Cable_ID,0), etc.