Why Universal Forwarder not sending entire file to splunk instance?

[adityaanand]

Hi,

I am monitoring the directory which contains xml files through universal forwarder. Before forwarding the data, a python script run and add some data in file then forwarder send the file to indexer on some interval.
But data receives at receiver side i.e. splunk instance in chunks i.e. not entire file.

My XML file look like:

<?xml version="1.0" ?><ABC xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <SiteDetail>
    <CustomerName>Govind Imperial Hospitale</CustomerName>
    <HITPCSN>HIT001</HITPCSN>
    <WorkstationCount>7</WorkstationCount>
    <Location>
      <Region>AMERICA</Region>
      <Country>KANADA</Country>
      <State>XYZ</State>    
      <City>ABC</City>
    </Location>
    </SiteDetail>
   <TargetMachines>
    <TargetMachine IPAddress="10.0.0.1" IsManuallyInstalled="true" Name="Demo_Machine1" Status="NoCommunication" StatusMessage="">
      <Tasks>
        <Task IsCancelled="false" IsDeleted="false" PackageID="ID_Sample_EXE_Success_New" PackageName="Sample Task New" PackageVersion="2.0.0.0" Status="InProgress" StatusMessage="TimeLogger30: Executing" TaskSer="12"/>
      </Tasks>
    </TargetMachine>
    <TargetMachine IPAddress="10.0.0.4" IsManuallyInstalled="true" Name="Demo_Machine4" Status="NoCommunication" StatusMessage="">
      <Tasks>
        <Task IsCancelled="false" IsDeleted="false" PackageID="ID_Sample_EXE_Success_New" PackageName="Sample Task New" PackageVersion="2.0.0.0" Status="InProgress" StatusMessage="TimeLogger38: Extracting files" TaskSer="16"/>
      </Tasks>
    </TargetMachine>
</TargetMachines>
</ABC>

Suppose script run and add a <date>...</date> as child of <SiteDetail> i.e. between </Location> and </SiteDetail>.
First time when script run it
send the data into two parts.
1. Below from <Date> tag including <date> tag and 2. Above <Date> tag.

When the script run further it sends only above the <Date> tag.

If I changed the position of <Date> tag it follow same sequence. i.e. 1. Below from <Date> including <date> tag tag and 2. Above <Date> tag.

But i need that forwarder sends entire file not in chunks.

My script look like:

from xml.dom.minidom import parse
import xml.dom.minidom
import time
xmlFile = xml.dom.minidom.parse("C:\\Data Analytics\file1.xml")
localtime = time.asctime(time.localtime(time.time()))
#date_element = xmlFile.getElementsByTagName(""date"")
if len(xmlFile.getElementsByTagName("date")) != 1:
   date = xmlFile.createElement("date")
   parent = xmlFile.getElementsByTagName("CustomerName")[0]
   value = xmlFile.createTextNode( localtime)
   date.appendChild(value)
   parent.appendChild(date)
print xmlFile.toprettyxml()
fo = open("C:\\Data Analytics\file1.xmll","wb") 
fo.write(xmlFile.toxml())
fo.close()

Please help.

Regards,
Aditya

[richgalloway]

I wonder if the forwarder is reading the file while your script is still processing it. Can you change things around so the script and the forwarder monitor different directories? Have the script read from one directory and write to the one the forwarder monitors.