Fields Extraction Not Working

[fabiocaldas]

I'm colleting WinEventLog:Application from windows machine using UniveralForwarder, the collected data is been sent to a HeavyForwarder cluster where the data is analised and if reach my criteria the data is changed using some SEDCMD and transformation, after this the data is sent to a Splunk Indexer.

Doing this, when data arrive to Splunk Indexer the automatic extraction field doesn't work. You can check that my events has well formatted messsages, like:

18/10/2013 01:25:28.7974852 PM AppName="vtex.requestcapture.windowsservice" AppVersion="7.0.1.0" Env="local" AZ="us-east-1e" WorkflowType="events processing" WorkflowInstance="lojaadcos" Level="Warn" LoggerVersion="7.0.0.0" OperationId="1789f3b5-d82a-4aa0-b6de-60ac693f5fb2" ExceptionType="applicationexception" VtexLog

18/10/2013 01:32:07.1257497 PM AppName="vtex.requestcapture.windowsservice" AppVersion="7.0.1.0" Env="local" AZ="us-east-1b" PageLoadTime="30986" Count="5" Browser="Chrome" TimeType="NetworkingTime" Account="EletronicaSantana" VtexMetric

As you can see, my messages only has a Timestamp and a sequence of Key/Values. If I do a simples search using extract auto=t like index=main | extract auto=t all fields are extracted correctly, but honestly I need it done to all data.

Now let me show my configs, to see if someone can help me to have this automatic extraction working again.

HeavyForwarder - props.conf

[WinEventLog:Application]

SEDCMD-remove_eol = s/[\n\r]/ /g

SEDCMD-remove_non_logger_event = s/(.)(VtexLog|VtexMetric|VtexMachineMetric)(.)/\1\2\3\2/g

SEDCMD-remove_primary = s/(.*)(\d\d\/\d\d\/\d\d\d\d\s\d\d:\d\d:\d\d.\d\d\d\d\d\d\d)(.+?$)/\2\3/1

SEDCMD-remove_exception_stacktrace = s/(.+)(ExceptionStack=)(.*)/\1/g

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

TRANSFORMS-set = setNull,setLog,setMetric,setMachineMetric,setLogSource,setMetricSource,setMachineMetricSource,setRequestCaptureIndex

HeavyForwarder - transform.conf

[logger_field_extraction]

DELIMS = " "

[setNull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[setLog]

REGEX = VtexLog

DEST_KEY = queue

FORMAT = indexQueue

[setMetric]

REGEX = VtexMetric

DEST_KEY = queue

FORMAT = indexQueue

[setMachineMetric]

REGEX = VtexMachineMetric

DEST_KEY = queue

FORMAT = indexQueue

[setLogSource]

REGEX = VtexLog

FORMAT = sourcetype::VtexLog

DEST_KEY = MetaData:Sourcetype

[setMetricSource]

REGEX = VtexMetric

FORMAT = sourcetype::VtexMetric

DEST_KEY = MetaData:Sourcetype

[setMachineMetricSource]

REGEX = VtexMachineMetric

FORMAT = sourcetype::VtexMachineMetric

DEST_KEY = MetaData:Sourcetype

[setRequestCaptureIndex]

REGEX = (AppName="vtex.requestcapture.windowsservice"|AppName="vtex.requestcapture.processwebsite")

FORMAT = rc

DEST_KEY = _MetaData:Index

As you guys can see on my HF if data matches what I'm looking for I change the source type and finally and chosse index.

Reading yestarday about how to use DELIMS to extract fields I set the DELIMS on HeavyForwarder, but I also have this on my Indexer:

Indexer - transforms.conf

[logger_field_extraction]

DELIMS = " "

Indexer - props.conf

[WinEventLog:Application]

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

[source::WinEventLog:Application]

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

[VtexLog]

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

[VtexMetric]

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

[VtexMachineMetric]

REPORT-field_extraction = logger_field_extraction

KV_MODE = none

As you guys can see I tried to apply this Delim anyway but it isn't working.

Again, I need to have all my Key/Values avaliable on my Indexer, and I'm losing this because I use a HF.

Any ideias?

[fabiocaldas · Accepted solution]

First I would like to say thanks to alacercogitatus and emiller42, because of your anwsers I knew that I was going on right direction.

So, the reason why my props and transforms conf files weren't working on my indexer/search server was the missing of metadata file exporting props to be used by system.

As soon I added:

[]

export = system

to default.meta.conf file to my app and redeployed it, the extraction started to work.

Thanks God now it's 100% now ...

[emiller42]

With the way your events are formatted, Splunk should have no issue automatically extracting the fields with zero configuration. I think the primary problem with your setup is the

KV_MODE = none

setting. This tells Splunk not to do any automatic key-value pair extraction, which is the opposite of what you want.

Get rid of that setting in all of your props.conf stanzas, and get rid of the [logger_field_extraction] stanza in transforms. I would expect your fields to work as expected after that.

If you insist on keeping KV_MODE = none, then change your transforms stanza as follows:

[logger_field_extraction]
DELIMS = " ", "="

Keep in mind that may get inconsistent where your values can contain spaces. Auto-parsing handles this when the values are quoted, as yours are.

[fabiocaldas]

Hi emiller42, I tried first without KV_MODE and DELIM, since it didn't worked I tried with DELIM and them with KV_MODE. I will be re-doing my tests withou KV_MODE and with DELIM as you told. I will be back telling the results.

[alacercogitatus]

DELIMS won't help you in this case. Make sure your searching with Verbose mode, and not fast or smart. If you have key="value", splunk will pick it up. Additionally, DELIMS is search-time so it needs to be on a search head.

http://docs.splunk.com/Documentation/Splunk/6.0/Search/Changethesearchmode

[fabiocaldas]

alacercogitatus, I had tried it before, and now I tried again but it doesn't work too.

[alacercogitatus]

DELIMS is best used with csv, but can be used with other values. If you want to use DELIMS set it equal to " ", "=" and see if that helps.

[fabiocaldas]

Since I'm using my Indexer server as a search server, why DELIMS aren't working?

[fabiocaldas]

alacercogitatus, even using verbose mode the fields aren't been extracted.