topic Generate alert listing broken Field extractions in Alerting

Generate alert listing broken Field extractions

splunk_zen — Tue, 24 Jun 2014 11:53:35 GMT

We get fed quite a wide range of formats.
This makes it hard for us to keep track of log formatting changes.

From time to time we get mission critical search alerts which rely on Field extractions

stopping to work to work because a line format has changed.

Those searches do start generating:

error in 'eval' command: The expression is malformed

so I'll be grateful if someone suggests a search picking on that,

but is it possible to limit it to Field extractions/scheduled alerts ?
We'll inevitably get spammed everytime a user makes a typo/is testing a new search otherwise.

Re: Generate alert listing broken Field extractions

martin_mueller — Tue, 24 Jun 2014 13:10:51 GMT

If your critical logs each have a known regex-testable format you could define alerts that fire if events are indexed that don't match their respective regex.

Re: Generate alert listing broken Field extractions

splunk_zen — Tue, 24 Jun 2014 14:25:25 GMT

Thank you Martin, but wouldn't that double the maintenance overhead?
We'd essentially have to clone the Field extrations expresssions and define search alerts.
Isn't it possible to define a single search alert which would catch all Field extractions erroring out ?

Re: Generate alert listing broken Field extractions

martin_mueller — Tue, 24 Jun 2014 15:19:31 GMT

Field extractions in and of themselves don't produce errors when the events don't match... they just don't match, it's normal behaviour.

If your fields are all required you could create an alert for events that don't have all fields, ie must have a changed format.