Splunk App for Microsoft Exchange: How to configure a filter to only index IIS data for the last 5 days and roll older data to frozen?

[tsvetan]

Hello,

Below you can find our configuration:

Universal Forwarder installed – EXCH – Windows Server 2008 R2
Splunk Enterprise 6.2 installed – Windows Server 2008 R2

What we are trying to do is to filter(or not index) older events from the MSExchange APP - TA-Windows-2008R2-Exchange-IIS. We have logs(C:\inetpub\logs\LogFiles\W3SVC1) from 2010 which we don't need, what we only want is to indexed events from the last 5 days. We already tried by using MAX_DAYS_AGO = 5 in props.conf file - this was set in both Universal Forwarder and on the Web Splunk server(indexer), how ever it didn't worked:

Web Splunk server(indexer)
CProgram FilesSplunketcappssplunk_app_microsoft_exchangelocalprops.conf
[MSWindows:2008R2:IIS]
MAX_DAYS_AGO = 5

For some reason it indexed the events from the last 5 days(2014), but it also indexed events from 2012, don't know why it decide to take events from this particular year.

Could you please help us on this? What we need is to index data from the last 5 days - [MSWindows:2008R2:IIS], all data older than the current 5 days needs to be deleted(frozen/nullQueued). The events from the other sourcetypes from the MSExchange APP are fine to be indexed, only the IIS is the problem one, because it is reaching our license limit. Do we need to create separate index for IIS or do you suggest something else?

Thanks in advanced for your support.

[ekost · Accepted solution]

The easiest way to limit an input monitor "to the last X days" is to set ignoreOlderThan. It must be set in the inputs.conf under the stanza you're using to monitor the IIS logs.

[monitor://C:\inetpublogs\LogFiles\W3SVC1]
ignoreOlderThan = 5d

As Windows isn't consistent about updating file modtime, you may need to use the setting alwaysOpenFile in addition to ignoreOlderThan if the latest IIS logs stop getting sent. Both are set in the inputs.conf of the Splunk instance responsible for watching the logs.

Setting MAX_DAYS_AGO in props.conf sets a limitation on valid dates extracted from events. That setting is for a different use-case.

If you want shorter retention for IIS logs than other logs, write them to a different index. There's no requirement to place IIS logs into a separate index.

There's also no easy way to remove the IIS data that's already indexed. But you're only paying for it in storage space as the license cost of indexing the data has already been incurred. I suggest implementing the setting ignoreOlderThan, and scoping your searches to the last business week. If you're curious, have a look at this post that covers some of the details about deleting indexed data.

[tsvetan]

Hello Ekost,

Thanks for your answer and for the explanations.
I have manage to solved my case by setting the below setups:
1. Create new index - iis
2. Add to the Universal Forwarder
C:Program FilesSplunkUniversalForwarderetcappsTAWindows2008R2ExchangeIISlocal
input.conf
[monitor://C:inetpublogsLogFilesW3SVC1*.log]
sourcetype=MSWindows:2008R2:IIS
queue=parsingQueue
index=iis
ignoreOlderThan = 1d
alwaysOpenFile = 1
disabled=false

So above changes seems to work so far.
I have another case where can't find a solution.
How can I get the EventID/Code from the MSExchange - not the Windows Event IDs from the server. We need that so we can create an alert on when we are switching between exchange servers, if there is an outage or connectivity issues. So is there a way to get these MSExchange EventIDs or if you can provide me with another solution will appreciated.
Thanks for your time.

[ekost]

One other thought. Setting ignoreOlderThan to one day may be too short for your use-case. For example, if the forwarder went down on the weekend and services were not restarted until Monday, ignoreOlderThan= 1d day would skip over Saturday's log file. If the goal is to capture all the IIS logs moving forward, setting it to 3 or 5 days will allow for flexibility, while still preventing the ingestion of all historical data.

[tsvetan]

Hello Ekost,

Thanks for your assistance and support on this matter. I have changed ignoreOlderThan=5d as you advised me and it is working fine. However I will wait for your answer regarding the Exchange events.

[ekost]

Hello tsvetan,
I would create a another answers post tagged with Exchange for your new question. There are a lot of folks that run Exchange servers, but I'm not one of them 🙂

[ekost]

Hello tsvetan,
I would create a another answers post tagged with Exchange for your new question. There are a lot of folks that run Exchange servers, but I'm not one of them 🙂

If you're saying there is another Event log channel for Exchange, not the usual Application, System, or Security, then grab the channel name from the Event Viewer and update the inputs.conf to poll it. Details here.

If the EventID for Exchange represents a fixed field in an event log channel, and the events themselves in Splunk, then it's down to a lookup file that will compare the field value to a list in the lookup and return the match. But I believe the MSExchange app includes the lookup files most people would need.