Can you help me with a problem I'm having extracting a field that is coming from a Windows host via a universal forwarder?

[dllb]

I am having some trouble with field extractions coming from a Windows host via a universal forwarder (UF). The log data is being read from a file by the UF. I am hoping someone can offer some insights.

An event that looks like this:

General Information
Additional Information:
SPID: 0000009914
MachineName: WWWWWWW
TimeStamp: 10/17/2018 03:13:32 PM
FullName: log4net Version=1.2.10.0
AppDomainName: /LM/W3SVC/9/ROOT-1-131842870514238769
ThreadIdentity: ABCXYZ\USERID
WindowsIdentity: IIS APPPOOL\VVVtage-Train
Exception Information:
System.Xml.XmlException: Root element is missing.
   at ABCXYZ.Portal.EAI.GetPremises(String SPID, String UID)
   at ABCXYZ.Portal.VVVtage.Main.Refresh()

I can put this event in regex101 and use this regex:

\n([^:]+): ([^\r\n]+)

and it works as desired.

To capture most of the : pairs. I am using the regex in a TRANSFORMS and it works on a *nix host where the source files are manually loaded. However, once I start forwarding the data from the Windows host, no fields are extracted. Since it works on Linux, but not on Windows, I am assuming I am missing something Windows specific.

Here is my props.conf

[sourcetype:xyz]
BREAK_ONLY_BEFORE = General Information
DATETIME_CONFIG = 
MAX_TIMESTAMP_LOOKAHEAD = 128
NO_BINARY_CHECK = true
TIME_PREFIX = TimeStamp:
category = Custom
disabled = false
pulldown_type = true
#REPORT-extractall = extract_new
TRANSFORMS-extractall = extract_new
EXTRACT-Exception_Full = System.Xml.XmlException:\s+(?<Exception_Full>[\S\s]+)
EXTRACT-WebException_Full = System.Net.WebException:\s+(?<Exception_Full>[\S\s]+)[\r\n]Request:
EVAL-Exceptions_Consolidated = coalesce(System_Exception,System_Net_WebException,System_Xml_XmlException)

My transforms.conf

[extract_new]
REGEX=\n([^:]+): ([^\r\n]+)
FORMAT=$1::$2
MV_ADD=true

[FrankVl]

To avoid trouble with different line endings, try: REGEX=[\r\n]+([^:]+): ([^\r\n]+)

But above all: don't use a TRANSFORMS. Search time field extractions should be done as a REPORT.

[dllb]

Frank, Good call on the line endings. I switched back to doing a REPORT but still no luck with either.

[akira_splunk · Accepted solution]

David,

I just tested this on a Windows universal forwarder sending to a Macbook indexer+search head in one. The extractions are working for me.

My suggestions is to remove the props/transforms you have above from the UF and indexers. Place only on the search head. Since you're using a deployment server and having this configuration pushed out to your search head, indexers, and forwarder, you can fix this by creating another serverclass for just the search head. Then place the assign the app where these configs are located to just the search head serverclass, put out the configs, restart all the servers, and you should be good to go.

[dllb]

Ahmed,

I went through and removed any "extra" props.conf and/or transforms.conf from the indexers and the universal forwarders. I also double checked the permissions on the field extractions and transformations. One change - the explicit field extraction that creates Exception_Full has shown up. That is progress. Still not seeing the fields that should have been created by the transforms.