topic Re: Source rename for IIS logs on default sites in Getting Data In

Source rename for IIS logs on default sites

fervin — Fri, 26 Aug 2011 15:48:47 GMT

Hello,

I've noticed an issue with IIS logs that are being written from sites with a default site ID such as (...\logfiles\W3SVC1\ex110826.log). Logs from sites with random site IDs work perfectly fine.

The issue is that logs from these sites all get lumped together when the application builds its summaries.

Changing site IDs across the environment would be a less than ideal approach to fixing this behavior.

Would it be possible to rename these sources at index time and substitute a machine specific element into the source name? It would be perfect if I could transform \logfiles\W3SVC1\ to \logfiles\W3SVC#SERVERNAME or something.

If not, are there any other ways around this? Thanks in advance,

-Frank

Re: Source rename for IIS logs on default sites

araitz — Mon, 29 Aug 2011 17:38:24 GMT

That seems like a good enhancement request for the app - essentially, facilitate a mapping of the tuple machine-site id-site name.

Re: Source rename for IIS logs on default sites

sdwilkerson — Thu, 24 Nov 2011 13:17:08 GMT

Fervin,

Great question. I have had the same issue, which is, that different physical servers can have the same siteid and therefore all get lumped in with the same source.

I have also implemented your proposed solution at several sites and it works very well.

I created an app called TA-iss to deal with this (TA=Technology-Addon a splunk term for apps that do these sorts of extractions/manipulation). This does index-time manipulation and therefore it should go on whatever system does your parsing (e.g. the forwarder if you are using anything but a Universal Forwarder, or the Indexer if you send directly to the indexer from a Universal Forwarder).

In the props.conf for IIS logs, I have this line:

TRANSFORMS-force_source_for_iis = force_source_for_iis

And the referenced stanza in transforms.conf looks like this:

[force_source_for_iis]
DEST_KEY = MetaData:Source
REGEX = \d{2}\:\d{2}\s+([^\ ]*)\s+([^\ ]*)
FORMAT = source::$2-$1

This makes each source unique to the host-site_id

Best,
Sean