Why is signature_id in the Windows TA extracted in such an inefficient manner?!

[pj]

After living with this for a while, I decided that today that I cannot. The signature_id in the Splunk for Windows Add-On (TA) is extracted in a way that massively impacts search performance of Windows security data. This issue may not be noticeable in smaller environments but in environments of 100gb of windows data or more, searching by signature_id is incredibly slow - unnecessarily so. The problem is compounded by the fact that the extracted signature_id field is leveraged in all the eventtypes and tags within the TA too.

The field is really simply based upon EventCode, which is a naturally occurring KV field within Windows events. Searching by EventCode is incredibly fast, but searching by signature_id is incredibly slow. To demonstrate this - search over time for a signature_id in Windows, then run the same search but use EventCode. You will see a massive difference in search performance if you have a decent amount of windows events.

I tracked the issue to the default props.conf entry here:

## Since FIELDALIAS is destructive we need to preserve signature_id for certain SourceName values
EVAL-signature_id = if(SourceName="Microsoft-Windows-WindowsUpdateClient",signature_id,EventCode)

I have verified that evaluating the signature_id like this is the main cause of the issue. For example, if a Field Alias is used, everything (tag searches, eventtype searches, signature_id searches) run a gazillion times faster.

FIELDALIAS-signature_id = EventCode AS signature_id

Now clearly the comment in the props states that the field alias method cannot be used due to the fact that there is one minor case where it might not work in regards to Windows update events. However, I would have thought that an alternative approach could be leveraged that can take into account this windows update scenario, whilst still allowing a field alias (or similar performing) signature_id extraction approach to be leveraged.

In the meantime, if you are not actually indexing the Windows Update event log, then I see no reason why you wouldn't just use the FIELDALIAS approach I outlined above instead. Hopefully this will drive a more permanent fix in a future update of the TA.

[martin_mueller · Accepted solution]

Check out version 4.8.2 released in late February: http://docs.splunk.com/Documentation/WindowsAddOn/4.8.2/User/Releasenotes

2016-02-29  TAG-10607   Event type definitions have been updated to improve search performance. 

[pj]

Thanks. Ok, so looks they like have switched to using EventCode in the eventtypes, rather than signature_id. So yes, this will improve performance in the same manner as described above when searching by tags and eventtypes. However, the actual signature_id extraction, still leverages the inefficient approach. I would view this as somewhat of a band-aid approach and effort should really be put into fixing the inefficient signature_id extraction, which is the correct CIM compliant field, as any search that leverages signature_id will still be dog slow. The fix I have above will still work with the latest version of the TA also (again, assuming you are not indexing the windows update event log).

[pj]

Just to add a comment - you can use something like this in a local props.conf for the Windows TA, which will override the default signature_id extraction.

[source::(MonitorWare|NTSyslog|Snare|WinEventLog|WMI:WinEventLog)...]
EVAL-signature_id =
FIELDALIAS-signature_id = EventCode AS signature_id