Hello Mikael! Thank you for your Add-on!
I have installed an universal forwarder on my TMG 2010 Server, and I have configured to input log files from a local directory on this server. The logs are writting to this local folder by TMG constantly.
I see fresh events in my splunk web-interface. But some of events are corrupted and the consists of part of a line.
I think the root cause of this issue in indexing the events before TMG has finished to write event completely.
What setting I should change to fix this problem?
1) yes, I have copied it to apps folder and I have create "local" folder with inputs.conf
## Forefront TMG Firewall logs ## Modify paths to fit your needs [monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_FWS_*.w3c] sourcetype = microsoft:forefront:tmg:fw disabled=0 index=forefront_tmg ## Forefront TMG Proxy logs ## Modify paths to fit your needs [monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_WEB_*.w3c] sourcetype = microsoft:forefront:tmg:proxy disabled=0 index=forefront_tmg
2) yes, I have copied it to apps folder. with default settings
I have another test deployment and for a test I have inputed the copied big log file of the yesterday to my test splunk instance. And all events are correct and field extraction works fine.
In my main installation which is indexes hot log files from TMG it is about 99.5% of events are correct and field extraction work. But there are 0.5 % of events which are partialy indexed and field extraction not work correctly.
How can I get 100% of events will be indexed correct on my main instance?
Ok, thanks for checking that out.
What Splunk version are you on?
Regarding the partially indexed events - are you sure these are not just the headers or the #Fields line from the beginning of the file?
You may also try to use MonitorNoHandle:// instead of monitor://
In this case you must specify a single file, not a path nor wildcards.
You may also test the "time_before_close" parameter in your inputs.conf: https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Inputsconf
There are probably a few other things to try as well. Maybe someone else will chime in with some tips.
I haven't seen this before.
Both are a requirement.