Why are fields not being extracted for some events with no apparent pattern?

[jmwatson]

We are not getting extracted fields for some events and there's no apparent pattern as to why. These are all simple extractions and they usually work. This is very problematic as will result in false statistics.

Extracted:

2014-09-03T10:59:59.316-0400 myAction="CachePut" myActualContext="services.ServiceService" myCacheType="remote" myDurationNanos="2209789" myRecordedTimestamp="2014/09/03 10:59:59.316 EDT" myRequestedContext="services.MemberGetSystemMap" {myUow=c17df9e5-1261-4d2d-907a-12ca954ce11f}

2014-09-03T10:59:59.224-0400 ihAction="CachePut" myActualContext="null" myCacheType="local" myDurationNanos="426969" myRecordedTimestamp="2014/09/03 10:59:59.224 EDT" myRequestedContext="Sxc.getMemberEffectiveDates" {myUow=c17df9e5-1261-4d2d-907a-12ca954ce11f}

Not Extracted:

2014-09-03T10:59:59.264-0400 myAction="CachePut" myActualContext="null" myCacheType="remote" myDurationNanos="2293969" myRecordedTimestamp="2014/09/03 10:59:59.264 EDT" myRequestedContext="Power.getMemberEffectiveDates" {myUow=fadcb445-3722-4289-8821-04c6874942e5}

Is this a known bug and/or is there a way we can get debug why the extraction is not occurring for some events?

[chris · Accepted solution]

I still do not see what is not working with your events. If you do not know if you have any manual field extractions configured you probably don't have any. I have encountered sourcetypes where Splunk did not work in 100% of the cases. What I usually do then is switch to a manually configured extraction.

If you do not know about props & transforms yet -> read this documentation

There is a GUI field extractor that might help -> documented here I don't use it so I can't tell if it will work for your problem

If you have access to the file system of your Splunk server you can either create or add the following stanzas to props.conf & transforms.conf in /etc/system/local

props.conf

[replaceWithYourSourcetype]
KV_MODE=none
REPORT-test=delims

transforms.conf

[delims]
DELIMS = " ", "="

[jmwatson]

I guess this is the best answer I will get for this. I appreciate your help but frankly it leaves me a little cold. I'm pretty new to Splunk. Is it typical to see things not really work right and you are on your own to work around it? I'm not accustomed to this approach with paid software (IBM aside.)

[jmwatson]

Update: Although telling the search to extract (what I think it's always supposed to do anyway) seemed to help, I've found more examples where this doesn't have any effect and no fields are being extracted from events similar to those shown above.

[jmwatson]

So now I'm getting partially extracted data where some fields are always pulled but the durations in nanos are sometimes not getting pulled out (based on a cursory look)

[jmwatson]

Almost forgot: Thanks for your help.

[jmwatson]

1. manual extractions: I don't think so but I need to confirm with a colleague.
2. none or some: when it works, all are extracted, when it doesn't none are
3. same sourcetype: yes
4. any unescaped or extra ": not that I can see, examples are copied from splunk raw value.
5. btool props list display KV_MODE=none: over my head, need to phone a friend
6. adding a "| extract " to the end: Hey, that seems do the trick. Excellent! Does that indicate what the problem is?

[chris]

I do not know the answer to your problem but here are some questions that might help: Did you set up any manual extractions (using props & transforms)? Are none of the fields extracted or only some? All the events have the same sourcetype right? If not does splunk btool props list display KV_MODE=none for one sourcetype? There aren't any unescaped or extra " in there events that do not work,right (I do not see any in your sample)? Does adding a "| extract " to the end of your search change anything?
http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/Extract