All Apps and Add-ons

The app is indexing event, before the TMG has write out the line of the event completely

Path Finder

Hello Mikael! Thank you for your Add-on!
Hello Guys,
I have installed an universal forwarder on my TMG 2010 Server, and I have configured to input log files from a local directory on this server. The logs are writting to this local folder by TMG constantly.
I see fresh events in my splunk web-interface. But some of events are corrupted and the consists of part of a line.
I think the root cause of this issue in indexing the events before TMG has finished to write event completely.
What setting I should change to fix this problem?

0 Karma
1 Solution

Esteemed Legend

Esteemed Legend

Path Finder

Hi, woodcock!
Thank you for your knowledge!

0 Karma

Path Finder

Hi,
1) yes, I have copied it to apps folder and I have create "local" folder with inputs.conf

## Forefront TMG Firewall logs
## Modify paths to fit your needs 
[monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_FWS_*.w3c] 
sourcetype = microsoft:forefront:tmg:fw 
disabled=0 
index=forefront_tmg

## Forefront TMG Proxy logs
## Modify paths to fit your needs
[monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_WEB_*.w3c] 
sourcetype = microsoft:forefront:tmg:proxy 
disabled=0
index=forefront_tmg

2) yes, I have copied it to apps folder. with default settings

I have another test deployment and for a test I have inputed the copied big log file of the yesterday to my test splunk instance. And all events are correct and field extraction works fine.

In my main installation which is indexes hot log files from TMG it is about 99.5% of events are correct and field extraction work. But there are 0.5 % of events which are partialy indexed and field extraction not work correctly.
How can I get 100% of events will be indexed correct on my main instance?

0 Karma

Motivator

Ok, thanks for checking that out.

What Splunk version are you on?

Regarding the partially indexed events - are you sure these are not just the headers or the #Fields line from the beginning of the file?

You may also try to use MonitorNoHandle:// instead of monitor://
In this case you must specify a single file, not a path nor wildcards.

You may also test the "time_before_close" parameter in your inputs.conf: https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Inputsconf

There are probably a few other things to try as well. Maybe someone else will chime in with some tips.

Path Finder

Hi, Mikael,
Fantastic! It Works!
I have added time_before_close=60 into inputs.conf and now all events are indexing correctly!
Thank you for your time!

0 Karma

Motivator

Hi,

I haven't seen this before.

  1. Did you push the TA-Microsoft_Forefront_TMG add-on to the Forwarder in question?
  2. Did you install the TA-Microsoft_Forefront_TMG add-on on your Indexers?

Both are a requirement.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!