Problem indexing .sdf logs

[splunkIT]

Our enviroment consist of splunk light forwarder > intermediate forwarder > indexer.

I'm trying to index the .sdf logs. Our reporting software exports performance (.pdf) log every hour, and we have a script that moves file to correct folder.

This destination folder is monitored by splunk light forwarder (which forwards to the intermediate forwarder, then to the indexer), and each monitored folder has own source and sourcetype.

[monitor://C:\PLET\Splunk\192.9-Kemira-SAP_SAN_2-FC-8G-PUS\*.sdf]
disabled=0
index=dwdm
source=Kemira-SAP_SAN_2-FC-8G-PUS
sourcetype=dwdm-perf
crcSalt = <SOURCE>

However, only some performance logs are coming to indexer. On indexer, I can mostly see three first rows of each file, but other data is left out. I'm suspecting that the reason for this behaviour is that each file's three first rows are identical and the light forwarder suspects that the file is already indexed. I have tried to add crcSalt and modified Check_Method property but these doesn't seems to work.

[splunkIT]

Thanks very much, Yann!

[yannK · Accepted solution]

The parsing happens on the indexer and any heavy forwarder.
Not on UF or LWF.

I you have to use heavy forwarders, I recommend to have the necessary props/transforms on every indexers and heavy forwarders.

[cmendiola]

Posting on behalf of splunkIT:

Ayn: I think you may be correct about the timestamp issue. This are some of the DateParserVerbose errors from the intermediate forwarder:

10-22-2012 07:29:53.181 +0300 WARN DateParserVerbose - Time parsed (Mon Oct 22 00:45:00 2012) is too far away from the previous event's time (Mon Oct 22 22:10:12 2012) to be accepted. If this is a correct time, MAX_DIFF_SECS_AGO (3600) or MAX_DIFF_SECS_HENCE (604800) may be overly restrictive.

Why is the timestamp parsing occurring at the intermediate forwarder, and not at the indexer? I have the props.conf stanzas for all these sourcetypes on the indexer. Did I mis-configure something here?

[splunkIT]

yannK: these are .sdf files. They are essentially just .csv files.

[Ayn]

Could also be a timestamp issue, i.e. logs are coming in but get a different timestamp than what you expect and therefore falls out of the time window you're searching in. Have you looked for events over all time?

[yannK]

sdf or pdf ?