Hi,
After a lot of searching, trying and bashing my head, i will drop my problem here. I would like to index the logfiles of a windows smtp service. I thought this would be easy, however I can't get it to work.
Splunk is installed on a debian along with some other tools. So to index Windows files, I need to install the universal forwarder. I installed version 6, so the Splunk_TA_windows is installed too. During the installation I cannot browse to c:\windows\System32\logfiles
. It's not there. I can browse to the folder in Windows explorer... Ok, no problem, I select a directory to monitor and i will change it in inputs.conf.
I understand that i should edit
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf.
It does not matter what i try, if I add c:\windows\System32\logfiles
, the folder is not getting indexed. Any other folder will be indexed.
What is so special about this folder?
I see that Splunk also indexes the eventlog (system, security and application) how can I remove these? During the installation i made sure nothing was selected. I only want the folder to be indexed.
How do I make sure that the indexed info is not stored in the main index. I would like an index per server. Or is that a bad idea. I seems easy to remove an index when a server no longer exits.
The logfiles are in IIS format. I read some articles that Splunk can handle this, but I would need to change the props file. Is that the file on the forwarder or on the indexer?
Thanks for any ideas,
Sven
check the tailing status of that directory:
Open a browser to:
https://serverwithuniversalforwarder:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus
Also, when troubleshooting permission issues - ProcMon from Sysinternals is gold.
https://technet.microsoft.com/en-us/library/bb896645.aspx
Filter it to only show "Result"=ACCESS DENIED and "Proccess Name" begins with: splunk
Occasionally Windows files will not update their timestamp, or will not be readable while the writing process still has it open. You can try setting
alwaysOpenFile = true
to the monitor stanza in inputs.conf and see if that helps.
This seems like an excellent tip. I can imagine, you should always set this for windows service logfiles.
It sounds like it might be a permissions issue to me.
As an administrator, you have access to that directory. As the local system account, you might not.
I would try creating an account with local permissions to that directory and running the UniversalForwarder service under that account.
As a quick test, you can have it run under your credentials or under the credentials of an administrator. If it works, all you need to do is add a service account.
Indeed. If I run the UniversalForwarder as an administrator, the files are getting indexed. I will create a service account.
The splunk logs are located in splunk\var\log\splunk\
. The log file you're interested in is the splunkd.log
.
If there is a problem with the indexing or reading of that log file, then you should be able to search the _internal
logs for messages about the file. Example:
index=_internal "*LogFileName*"
For LogFileName just use the name of the smtp logfile, no need to put the whole path.
I would like to add, that when change the UniversalForwarder service to run under .\administrator then the files that i manually add to c:\windows\System32\logfiles are getting indexed. The log files, created by the smtp service however or not indexed. So that rules out any typo.
It seems to be related to ntfs rights. I checked the rights and they look correct. Just to be sure i forced the inheritance again. Still, the manually added files are indexed, the log files created by the smtp service are not indexed.
I'm not that familiar with splunk. Where can i find the errors in the _internal index?
oh , i'm not sure if this is just a typo in your question, but from what I can see the logfiles directory is CamelCase :
System32\LogFiles\ , not System32\logfiles
are any errors showing up in your _internal index?
I installed the Universal Forwarder with domain admin credentials. I can browse to the folder in windows without issues...
Did you install the Universal Forwarder to run as a user with access to the files/directory you need? You don't need to do anything with the props.conf file right now.