Why is my current file monitor configuration always missing the first line of a CSV file that has no headers?

[JeremyHagan]

I've got a file monitor set up for a headerless CSV file which I generate on a periodic basis. I've noticed that the monitor is always ignoring the first line of the file. I am not using CHECK_FOR_HEADER and from what I can tell this is turned off by default.

Anyone seen this before? Here is the config:

Inputs.conf:

[monitor://C:\ePOExport\Threat]
disabled = 0
index = unclassified
sourcetype = epo:threat
followTail = 0
recursive = false
crcSalt = < SOURCE >

Props.conf

[epo:threat]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
REPORT-epo:threat = epo:threat:report

Transforms.conf

# ------------------------------------
# McAfee ePO Threat Events Fields
# ------------------------------------
[epo:threat:report]
DELIMS = ","
FIELDS = timestamp,signature,threat_type,signature_id,category,severity_id,event_description,detected_timestamp,file_name,detection_method,vendor_action,threat_handled,logon_user,user,dest_nt_domain,dest_dns,dest_nt_host,fqdn,dest_ip,dest_netmask,dest_mac,os,sp,os_version,os_build,timezone,src_dns,src_ip,src_mac,process,url,source_logon_user,is_laptop,product,product_version,engine_version,dat_version,vse_dat_version,vse_engine64_version,vse_engine_version,vse_hotfix,vse_product_version,vse_sp

[gcusello]

Hi JeremyHagan,
I don't know why your props.conf doesn'r read the first line of your csv file, but I have many csv files extractions and I used a different way to extract fields, try something like this:

Inputs.conf:

[monitor://C:\ePOExport\Threat]
disabled = 0
index = unclassified
sourcetype = epo:threat
crcSalt = <SOURCE>

Props.conf

[epo:threat]
INDEXED_EXTRACTIONS = csv
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = true
disabled = false
TIMESTAMP_FIELDS = timestamp
FIELD_NAMES = timestamp,signature,threat_type,signature_id,category,severity_id,event_description,detected_timestamp,file_name,detection_method,vendor_action,threat_handled,logon_user,user,dest_nt_domain,dest_dns,dest_nt_host,fqdn,dest_ip,dest_netmask,dest_mac,os,sp,os_version,os_build,timezone,src_dns,src_ip,src_mac,process,url,source_logon_user,is_laptop,product,product_version,engine_version,dat_version,vse_dat_version,vse_engine64_version,vse_engine_version,vse_hotfix,vse_product_version,vse_sp

Beware that you must insert props.conf both on your Indexers and on your forwarders!

Bye.
Giuseppe

[JeremyHagan]

I tried completely changing the export script to use key/value pairs instead of CSV, set up a new monitor on a different folder into a temporary index and have observed the same issue. It eliminates the specific parts of the config on the indexer side of the props and tranforms section relating specifically to this source type as being the issue.

What I can't understand is that I have a second export from the same forwarder using a different select statement which isn't affected by this issue.

[JeremyHagan]

Note that the spaces in < SOURCE > are not in the actual inputs.conf. I have inserted them here because the editor was removing the statement without them.