Deployment Architecture

Not getting all the files from forwarders

carlyleadmin
Contributor

Hi,

I know there are lot of questions under the same topic,but i am stuck.i have an application server which forwards the logs to splunk.The way logs are written is that are on random selection.i will share that information as well. So, when there is a process and being written into the log it picks a random one from all the logs and appends to it.even the log date modified is ,lets say today, when i open up the log it might start with a date and a process written onto that log from 3 months ago and at the end of that log i can see the latest process from today ,and when another process happens it writes it to another log and that is the cycle.

here is my inputs.conf

[default]
host = xxxxxx

[monitor://D:\y\Log Files]
disabled = 0
index=z
followTail = 0
sourcetype=Data Import
ignoreOlderThan = 30d

Here are the screenshots
alt text

could post the last screenshot but it is showing the end of the same log i posted with today's date.

My question is,i am not getting all the log files form that location.not sure how long this has been happening for but i jut found out about this couple days ago.Lets say i have 15 log files from yesterday,i only got 3 of them.To troubleshoot the issue i tried looking at the splunkd but that did not give me much.

this is the latest entry on splunkd

01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-wmi.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 10000000000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-MonitorNoHandle.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-admon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-netmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-perfmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: run once
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe --ps2
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-regmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winevtlog.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winprintmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.041 -0500 INFO PipelineComponent - Launching the pipelines for set 0.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - TailWatcher initializing...
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk...stash_new.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\license_usage_summary.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\metrics.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\y\Log Files.
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\y\Log Files.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: tailreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting tailreader0 thread
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: batchreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting batchreader0 thread
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 3333 sockets
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 1365 threads
01-09-2018 12:21:39.710 -0500 INFO WatchedFile - Will begin reading at offset=988394 for file='D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:21:39.726 -0500 INFO WatchedFile - Will begin reading at offset=3402522 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:21:39.804 -0500 INFO TcpOutputProc - Connected to idx=10.14.0.246:9997, pset=0, reuse=0.
01-09-2018 12:21:52.876 -0500 INFO WatchedFile - Will begin reading at offset=344718 for file=''D:\y\Log Files\DataImport-62-[5712].log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\splunkd_ui_access.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - Will begin reading at offset=50885 for file='D:\splunk\var\log\splunk\splunkd-utility.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\searchhistory.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\scheduler.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\remote_searches.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\mongod.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=12261005 for file='D:\splunk\var\log\splunk\metrics.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage_summary.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=11480 for file='D:\splunk\var\log\splunk\conf.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=77366 for file='D:\splunk\var\log\splunk\audit.log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.099 -0500 INFO WatchedFile - Will begin reading at offset=24999075 for file='D:\splunk\var\log\splunk\metrics.log.1'.

i deleted the splunkd and restarted the splunk service and check to see if i was getting the missing logs and that worked for a day.and whenever i made a change to the log it was being captured and sent to indexer.But today,it is the same behavior.i am missing log files in splunk.

i hope this is not too complicated .i am kind of stuck and need second set of eyes to tell me that i missing something.Any help is appreciated.

Thanks

Tags (1)
0 Karma
1 Solution

carlyleadmin
Contributor

I just uninstall/install the forwarder and cleaned up the log files from the location and started from scratch to be exact.so far so good and i am getting all the log files.it is highly unlikely that forwarder was the culprit but it's been 4 days since i installed the latest version and it is working

thanks for all the help

View solution in original post

carlyleadmin
Contributor

I just uninstall/install the forwarder and cleaned up the log files from the location and started from scratch to be exact.so far so good and i am getting all the log files.it is highly unlikely that forwarder was the culprit but it's been 4 days since i installed the latest version and it is working

thanks for all the help

micahkemp
Champion

Does the splunk service have permission to read the files in question? Do you see the files listed when you run:

splunk list monitor
0 Karma

carlyleadmin
Contributor

yes i can see them. like i said i am missing some of the files and cant find the pattern.Though i am trying to eliminate all the possibilities and i have other sources that are being monitored.i have iis logs,i have another application log file that i monitor and when i check those and go thru the folders i am not missing a single file on those.i am suspecting that it has to do with the log itself.otherwise i wouldve probably seen the same behavior on those as well,right?i am using the exact config files thru out my environment as follows

disabled = 0
index=yyyy
followTail = 0
sourcetype=xxxx
ignoreOlderThan = 30d

0 Karma

mayurr98
Super Champion

add below in inputs.conf

  crcSalt = <SOURCE>

http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Inputsconf
Let me know if this helps!

0 Karma

micahkemp
Champion

crcSalt = <SOURCE> is almost always a bad idea, and the wrong solution. I would not recommend it unless you know exactly why you need it, and what the side effects may be.

0 Karma

carlyleadmin
Contributor

hey Mayury,

Thanks for the reply. i've made changes to the inputs file and yes it did help in some manner.but still,results are sporadic.i am still missing a file or two when i post a new process and i can see the log file being created in the folder but it is not showing up in splunk.

i even changed my limits.conf to

[thruput]
maxKBps = 512

[inputproc]
max_fd = 1000

0 Karma

mayurr98
Super Champion

can you try making maxKBps infinite for a time being to see if this is causing a problem? it may index a huge amount of data but you can adjust it anytime.

0 Karma

carlyleadmin
Contributor

yeah that did not help either.

0 Karma

mayurr98
Super Champion

try making crcSalt = string I do know the impact but just trial and error method you can apply
you're looking to reindex a file thats already been indexed i mean those missing files ever indexed?

0 Karma

carlyleadmin
Contributor

Still the same

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...