IIS W3C log and ms:iis:auto sourcetype

[sweiland]

Hi there,

Just a quick question as I am not familiar with some basic routines yet..

We use a "ms:iis:auto" to ingest a basic IIS log file (W3C formatted), nothing special except it is a sharepoint website.

After some time, we looked at the data in splunk, and we see this (raw format):

192.168.2.72 GET /_vti_bin/client.svc/web/title - 443 0#.w|domain\login 192.168.52.48 Mozilla/5.0

 After some research, it seems that IIS log file can also contain/encode claims (here it is "0#.w|"):

https://social.technet.microsoft.com/wiki/contents/articles/13921.sharepoint-20102013-claims-encodin...

Is it possible to remove this claim (or even better map it), knowing it is not always present ?

That would allow us to see the user field in the proper format in splunk and not:

"0#w|domain\user"

Have a good day,

[dmarling · Accepted solution]

You could create a somewhat complicated SECMD in your props.conf to map it, but then you are adding to your license usage unnecessarily.  I believe it makes more sense to turn that data from the url your linked into lookup files that return the definition.  I will provide you with queries that will build the lookup files after I copy/pasted the data from that link.

First is the ClaimType definitions:

| makeresults 
| fields - _time
| eval data="    \"#\" for a user logon name
    \".\" for  an anonymous user
    \"5\" for an email address
    \"!\" for an identity provider
    \"+\" for a Group security identifier (SID)
    \"-\" for a role
    \"%\" for a farm ID
    \"?\" for a name identifier
    \"\\\" for a private personal identifier (PPID)
    \"e\" for a user principal name (UPN)
    \"\"\" for a user ID
    \"$\" for a distribution list security identifier (SID)
    \"&\" for a process identity security identifier (SID)
    \"'\" for a process identity logon name
    \"(\" for an authenticated user
    \")\" for a primary security identifier (SID)
    \"*\" for a primary group security identifier (SID)
    \"0\" for an authorization decision
    \"1\" for a country
    \"2\" for a date of birth
    \"3\" for a deny only security identifier (SID)
    \"4\" for DNS
    \"6\" for a gender
    \"7\" for a given name
    \"8\" for a hash
    \"9\" for a home phone
    \"<\" for a locality
    \"=\" for a mobile phone
    \">\" for a name
    \"@\" for other phone
    \"[\" for a postal code
    \"]\" for RSA
    \"^\" for a secure identifier (SID)
    \"_\" for a service principal name (SPN)
    \"`\" for a state or province
    \"a\" for a street address
    \"b\" for a surname
    \"c\" for a system
    \"d\" for a thumbprint
    \"f\" for a uniform resource name (URI)
    \"g\" for a web page" 
| makemv data tokenizer="(?<data>[^\n]+)"
| mvexpand data
| rex field=data "\s+\"(?<ClaimType>.+)\" for\s+((a|an)\s+)?(?<ClaimTypeDefinition>[^\e]+)"
| fields ClaimType ClaimTypeDefinition
| outputlookup IISClaimTypeLookup.csv

Then is the ClaimValueType definitions:

| makeresults 
| fields - _time
| eval data="    \".\" for a string
    \"+\" for an RFC 822-formatted name
    \")\"  for an integer
    \"\"\" for a Boolean
    \"#\" for a date
    \"$\" for a date with time
    \"&\" for a double
    \"!\" for a Base64 formatted binary
    \"0\" for a X.500 formatted name"
| makemv data tokenizer="(?<data>[^\n]+)"
| mvexpand data
| rex field=data "\s+\"(?<ClaimValueType>.+)\"\s+for\s+((a|an)\s+)?(?<ClaimValueTypeDefinition>[^\e]+)"
| fields - data
| outputlookup IISClaimValueTypeLookup.csv

Finally the Auth Mode:

| makeresults 
| fields - _time
| eval data="    \"w\" for Windows claims (no original issuer)
    \"s\" for the local SharePoint security token service (STS) (no original issuer)
    \"t\" for a trusted issuer
    \"m\" for a membership issuer
    \"r\" for a role provider issuer
    \"f\" for forms-based authentication
    \"c\" for a claim provider"
| makemv data tokenizer="(?<data>[^\n]+)"
| mvexpand data
| rex field=data "\s+\"(?<AuthMode>.+)\"\s+for\s+((a|an)\s+)?(?<AuthModeDefinition>[^\e]+)"
| fields - data
| outputlookup IISAuthModeLookup.csv

Now you need to be able to extract each of those fields from your raw data and turn them into fields.  The below extraction will pull those fields as long as there is a space followed by a 0 then the data that matches what is in the url you attached:

| rex "\s0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)"

After you generate those lookup tables in the first three queries you can use them in combination with the above extraction to produce definitions for each of those codes:

| makeresults 
| eval Data="192.168.2.72 GET /_vti_bin/client.svc/web/title - 443 0#.w|domain\login 192.168.52.48 Mozilla/5.0"
| rex field=Data "\s0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)"
| lookup IISClaimTypeLookup.csv ClaimType
| lookup IISClaimValueTypeLookup.csv ClaimValueType
| lookup IISAuthModeLookup.csv AuthMode

[sweiland]

Very interesting and precious answer (and yet complex for me yet)

For instance, the first thing I dont get is that it seems I do not have a small part in the field.

They say in the doc:
"<IdentityClaim>:0<ClaimType><ClaimValueType><AuthMode>|<ClaimValue>"

But apparently I only have in IIS:
"0<ClaimType><ClaimValueType><AuthMode>|<ClaimValue>"

I just double-checked direcly in the IIS log and I see:
- 443 (s-port),
- "0<ClaimType><ClaimValueType><AuthMode>|<ClaimValue>",
- the client IP (c-ip)

So we miss the "<IdentityClaim>:" part for some reason

I will try to understand all this, best would be to update the ms:iis:auto sourcetype so that work whenever this "prefix" field is there or not

[dmarling]

If you modify the iis log configuration so it has the missing IdentityClaim field you will need to adjust your field extractions I wrote a bit to this in search:

| rex "\s((?<IdentityClaim>[ic]):)?0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)"

and to this if you make it an extracted field in props.conf for the ms:iis:auto sourcetype:

\s((?<IdentityClaim>[ic]):)?0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)

[sweiland]

The IIS log is not modified but pretty standard (W3C with some basic fields), so this field we are working on is simply : "cs-username"

Seems sometime this "cs-username" field is adding also claims in some cases, this is why I am trying to make splunk "detect" that sometimes this fields is more complex than only a "domain\username" value

Maybe it could even be easier to simply remove the "0#.w|" string during the ingestion, I guess this should be possible via a transform.conf maybe ? (this is very dirty but so far it is the first time I see this ^^)

[dmarling]

Once you feel comfortable with all of that you can take it a step further by turning that rex to pull out each of those fields in the final search into an extracted field for the ms:iis:auto sourcetype.  You need to adjust it slightly for that though:

\s0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)

Once that is done you can then create lookup definitions for those lookup files you created and use automatic lookups so the fields that are in those lookups already show in searches and can be referenced by other users without the need to do the lookup command manually in your search.