Getting Data In

How can i index the log file from a windows smtp service?

svendewindt
Explorer

Hi,

After a lot of searching, trying and bashing my head, i will drop my problem here. I would like to index the logfiles of a windows smtp service. I thought this would be easy, however I can't get it to work.

Splunk is installed on a debian along with some other tools. So to index Windows files, I need to install the universal forwarder. I installed version 6, so the Splunk_TA_windows is installed too. During the installation I cannot browse to c:\windows\System32\logfiles. It's not there. I can browse to the folder in Windows explorer... Ok, no problem, I select a directory to monitor and i will change it in inputs.conf.

I understand that i should edit

C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf. It does not matter what i try, if I add c:\windows\System32\logfiles, the folder is not getting indexed. Any other folder will be indexed.

What is so special about this folder?
I see that Splunk also indexes the eventlog (system, security and application) how can I remove these? During the installation i made sure nothing was selected. I only want the folder to be indexed.
How do I make sure that the indexed info is not stored in the main index. I would like an index per server. Or is that a bad idea. I seems easy to remove an index when a server no longer exits.

The logfiles are in IIS format. I read some articles that Splunk can handle this, but I would need to change the props file. Is that the file on the forwarder or on the indexer?

Thanks for any ideas,
Sven

aalanisr26
Path Finder
0 Karma

bravon
Communicator

Also, when troubleshooting permission issues - ProcMon from Sysinternals is gold.
https://technet.microsoft.com/en-us/library/bb896645.aspx

Filter it to only show "Result"=ACCESS DENIED and "Proccess Name" begins with: splunk

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Occasionally Windows files will not update their timestamp, or will not be readable while the writing process still has it open. You can try setting

alwaysOpenFile = true

to the monitor stanza in inputs.conf and see if that helps.

svendewindt
Explorer

This seems like an excellent tip. I can imagine, you should always set this for windows service logfiles.

0 Karma

thesteve
Path Finder

It sounds like it might be a permissions issue to me.

As an administrator, you have access to that directory. As the local system account, you might not.

I would try creating an account with local permissions to that directory and running the UniversalForwarder service under that account.

As a quick test, you can have it run under your credentials or under the credentials of an administrator. If it works, all you need to do is add a service account.

svendewindt
Explorer

Indeed. If I run the UniversalForwarder as an administrator, the files are getting indexed. I will create a service account.

lukejadamec
Super Champion

The splunk logs are located in splunk\var\log\splunk\. The log file you're interested in is the splunkd.log.
If there is a problem with the indexing or reading of that log file, then you should be able to search the _internal logs for messages about the file. Example:
index=_internal "*LogFileName*"
For LogFileName just use the name of the smtp logfile, no need to put the whole path.

0 Karma

svendewindt
Explorer

I would like to add, that when change the UniversalForwarder service to run under .\administrator then the files that i manually add to c:\windows\System32\logfiles are getting indexed. The log files, created by the smtp service however or not indexed. So that rules out any typo.

It seems to be related to ntfs rights. I checked the rights and they look correct. Just to be sure i forced the inheritance again. Still, the manually added files are indexed, the log files created by the smtp service are not indexed.

0 Karma

svendewindt
Explorer

I'm not that familiar with splunk. Where can i find the errors in the _internal index?

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

oh , i'm not sure if this is just a typo in your question, but from what I can see the logfiles directory is CamelCase :

System32\LogFiles\ , not System32\logfiles

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

are any errors showing up in your _internal index?

0 Karma

svendewindt
Explorer

I installed the Universal Forwarder with domain admin credentials. I can browse to the folder in windows without issues...

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Did you install the Universal Forwarder to run as a user with access to the files/directory you need? You don't need to do anything with the props.conf file right now.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...