Getting Data In

How to configure inputs.conf to parse key value pairs in IIS 8.5 logs?

dlofstrom
Path Finder

I'm trying to parse IIS logs in Windows 2012 R2 based on the blog article:
http://blogs.splunk.com/2013/10/18/iis-logs-and-splunk-6/

From what I understand, as long as I set the sourcetype to "iis", the KV pairs should automatically be parsed as indexed extractions, but I'm not seeing them appear in my search results, nor can I specify them at search time (ie: "sourcetype=iis s-port=443" produces 0 results). Our setup is as follows:

Universal Forwarder (6.4.0)
inputs.conf:

[monitor://C:\inetpub\logs\LogFiles\W3SVC*\*.log]
disabled = 0
sourcetype = iis
queue = parsingQueue
ignoreOlderThan = 1d

Our indexing infrastructure uses an indexer cluster with a separate search server. All three systems (2 indexers, 1 search) are at Splunk 6.4.1.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

First guess, remove queue=. You should not have to tell Splunk a specific queue for data at the input-side, like 99.99% of the time. Also, for INDEXED_EXTRACTIONS to work, data needs to go into the structuredParsing queue ... http://wiki.splunk.com/Community:HowIndexingWorks

Also, upgrade that forwarder! There's been plenty of good patches on 6.4 to remove bugs, some of which may impact your situation.

View solution in original post

woodcock
Esteemed Legend

Do no use ignoreOlderThan = 1d for this case. Those logs should already be rotated automatically so there won't be days and weeks and months of logs in it. When using this setting, if the thing that writes to the log EVER stops writing for 24 hours or more, you will have PERMANENTLY blacklisted the log file from ever being forwarded, even when the events start flowing into the log file again. This is a VERY dangerous setting and most people do not realize how it works.

Also, make sure that you are not running your search in fast mode because this prevents search-time field extractions from being executed.

0 Karma

dlofstrom
Path Finder

I had never thought of that! Good to know. I was using it here to reduce indexing volume since I was constantly deleting/cleaning things while I was trying to get this to work, but we certainly use it in other areas really just because we wanted to reduce the index volume at the initial ingestion only. Going forward, it's not really necessary for us.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

First guess, remove queue=. You should not have to tell Splunk a specific queue for data at the input-side, like 99.99% of the time. Also, for INDEXED_EXTRACTIONS to work, data needs to go into the structuredParsing queue ... http://wiki.splunk.com/Community:HowIndexingWorks

Also, upgrade that forwarder! There's been plenty of good patches on 6.4 to remove bugs, some of which may impact your situation.

dlofstrom
Path Finder

Removed the queue=parsingQueue line and it worked like a charm! I copied that from a similar input, and assumed since it was the default anyway, that it wouldn't have made much of a difference. I guess not...

0 Karma

somesoni2
Revered Legend

In what mode you're running your searches? Could you provide a sample raw event? If the log format is standard, they should get extracted.

0 Karma

dlofstrom
Path Finder

Searches are running in Smart Mode. Sample raw event:

2017-02-23 14:06:46 172.17.46.192 GET /userfiles/image/admissions/bg-content.png - 443 - 172.17.46.100 Mozilla/5.0+(iPad;+CPU+OS+9_2_1+like+Mac+OS+X)+AppleWebKit/601.1.46+(KHTML,+like+Gecko)+Mobile/13D15 - 200 0 0 93

This is a standard IIS log.

0 Karma

dlofstrom
Path Finder

Could this have anything to do with it?

curl -k -s https://localhost:8089/services/server/info | grep kvStore: kvStoreStatus reports "failed"

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...