Not getting all the files from forwarders

[carlyleadmin]

Hi,

I know there are lot of questions under the same topic,but i am stuck.i have an application server which forwards the logs to splunk.The way logs are written is that are on random selection.i will share that information as well. So, when there is a process and being written into the log it picks a random one from all the logs and appends to it.even the log date modified is ,lets say today, when i open up the log it might start with a date and a process written onto that log from 3 months ago and at the end of that log i can see the latest process from today ,and when another process happens it writes it to another log and that is the cycle.

here is my inputs.conf

[default]
host = xxxxxx

[monitor://D:\y\Log Files]
disabled = 0
index=z
followTail = 0
sourcetype=Data Import
ignoreOlderThan = 30d

Here are the screenshots
alt text

could post the last screenshot but it is showing the end of the same log i posted with today's date.

My question is,i am not getting all the log files form that location.not sure how long this has been happening for but i jut found out about this couple days ago.Lets say i have 15 log files from yesterday,i only got 3 of them.To troubleshoot the issue i tried looking at the splunkd but that did not give me much.

this is the latest entry on splunkd

01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-wmi.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 10000000000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-MonitorNoHandle.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-admon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-netmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-perfmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: run once
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe --ps2
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-regmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winevtlog.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winprintmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.041 -0500 INFO PipelineComponent - Launching the pipelines for set 0.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - TailWatcher initializing...
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk...stash_new.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\license_usage_summary.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\metrics.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\y\Log Files.
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\y\Log Files.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: tailreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting tailreader0 thread
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: batchreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting batchreader0 thread
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 3333 sockets
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 1365 threads
01-09-2018 12:21:39.710 -0500 INFO WatchedFile - Will begin reading at offset=988394 for file='D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:21:39.726 -0500 INFO WatchedFile - Will begin reading at offset=3402522 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:21:39.804 -0500 INFO TcpOutputProc - Connected to idx=10.14.0.246:9997, pset=0, reuse=0.
01-09-2018 12:21:52.876 -0500 INFO WatchedFile - Will begin reading at offset=344718 for file=''D:\y\Log Files\DataImport-62-[5712].log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\splunkd_ui_access.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - Will begin reading at offset=50885 for file='D:\splunk\var\log\splunk\splunkd-utility.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\searchhistory.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\scheduler.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\remote_searches.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\mongod.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=12261005 for file='D:\splunk\var\log\splunk\metrics.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage_summary.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=11480 for file='D:\splunk\var\log\splunk\conf.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=77366 for file='D:\splunk\var\log\splunk\audit.log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.099 -0500 INFO WatchedFile - Will begin reading at offset=24999075 for file='D:\splunk\var\log\splunk\metrics.log.1'.

i deleted the splunkd and restarted the splunk service and check to see if i was getting the missing logs and that worked for a day.and whenever i made a change to the log it was being captured and sent to indexer.But today,it is the same behavior.i am missing log files in splunk.

i hope this is not too complicated .i am kind of stuck and need second set of eyes to tell me that i missing something.Any help is appreciated.

Thanks

[carlyleadmin · Accepted solution]

I just uninstall/install the forwarder and cleaned up the log files from the location and started from scratch to be exact.so far so good and i am getting all the log files.it is highly unlikely that forwarder was the culprit but it's been 4 days since i installed the latest version and it is working

thanks for all the help

[micahkemp]

Does the splunk service have permission to read the files in question? Do you see the files listed when you run:

splunk list monitor

[carlyleadmin]

yes i can see them. like i said i am missing some of the files and cant find the pattern.Though i am trying to eliminate all the possibilities and i have other sources that are being monitored.i have iis logs,i have another application log file that i monitor and when i check those and go thru the folders i am not missing a single file on those.i am suspecting that it has to do with the log itself.otherwise i wouldve probably seen the same behavior on those as well,right?i am using the exact config files thru out my environment as follows

disabled = 0
index=yyyy
followTail = 0
sourcetype=xxxx
ignoreOlderThan = 30d

[mayurr98]

add below in inputs.conf

  crcSalt = <SOURCE>

http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Inputsconf
Let me know if this helps!

[micahkemp]

crcSalt = <SOURCE> is almost always a bad idea, and the wrong solution. I would not recommend it unless you know exactly why you need it, and what the side effects may be.

[carlyleadmin]

hey Mayury,

Thanks for the reply. i've made changes to the inputs file and yes it did help in some manner.but still,results are sporadic.i am still missing a file or two when i post a new process and i can see the log file being created in the folder but it is not showing up in splunk.

i even changed my limits.conf to

[thruput]
maxKBps = 512

[inputproc]
max_fd = 1000

[mayurr98]

can you try making maxKBps infinite for a time being to see if this is causing a problem? it may index a huge amount of data but you can adjust it anytime.

[carlyleadmin]

yeah that did not help either.

[mayurr98]

try making crcSalt = string I do know the impact but just trial and error method you can apply
you're looking to reindex a file thats already been indexed i mean those missing files ever indexed?

[carlyleadmin]

Still the same

[mayurr98]

https://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html