I’m evaluating the Splunk app for Exchange. I noticed that the logs for the CAS/HubTransport were fairly large or least more than I was expecting. I have two CAS/HubTransport servers and 3 mailbox servers for about 700 employees. The CAS/HubTransport logs were about 15GB a piece with the largest index event being mswindows 2008 iis. I'm running Exchange 2010 and I put the TA's for Hubtransport, IIS, and CAS on the CAS boxes. I was expecting these logs to be a couple of GB at most. I'm I way off or would 30 GB be an expected index size for this?
Your hub transport logs and CAS logs likely span many days. If this is your first import, then the entire logs - including all historical data - was read.
Wait until tomorrow, then look at your ingest rate. You can use the metrics.log file in index=_internal for this purpose.
It certainly seems high for an organization your size, but not unprecedented. It really depends on what your organization is doing. With the Exchange app, you can now find out who is using the mail server. You may find a spammer or two with infected workstations.
It looks like it's indexing the all the logs in the W3SVC1 folder everyday on the CAS servers. Is that normal? For example I could see a log file from Jan of this year getting index. Shouldn't In digest the logs once and then append the next days or is there a setting somewhere that needs to be set to tell it to do that?
I figured it out. The IIS logs on the CAS box were there from the beginning of deployment about 65 GB work per CAS box. What I didn’t realize is the Limits.conf on the UniversalFowarder was set to limit throughput to 256 Kbps. I was expecting Splunk to grab all the logs at once and process them instead of dragging on for days. Once I changed the limits.conf and restarted the fowarder Splunk grabbed the remaining logs and is now process at rate that is acceptable.