ldapfilter - Why can't I use fields from events?

[patpro]

Hello,

I'm trying to use ldapfilter to add some info to events I collect from MS Exchange but as soon as my ldapfilter query is dynamic (makes use of $field$) it does not return anything.

My initial search looks like this:

index=Exchange Mailboxes=* | rex "'?S:Mailboxes=(?<SMailboxes>[^']+)'?;'?S:StoreObjectIds" | makemv SMailboxes delim=";"
| mvexpand SMailboxes| top SMailboxes limit=50
| rex field=SMailboxes "(?<m1>..)(?<m2>..)(?<m3>..)(?<m4>..)-(?<m5>..)(?<m6>..)-(?<m7>..)(?<m8>..)-(?<m9>..)(?<m10>..)-(?<m11>..)(?<m12>..)(?<m13>..)(?<m14>..)(?<m15>..)(?<m16>..)"
| eval conv="\\\\" . m4 . "\\\\" . m3 . "\\\\" . m2 . "\\\\" . m1 . "\\\\" . m6 . "\\\\" . m5 . "\\\\" . m8 . "\\\\" . m7 . "\\\\" . m9 . "\\\\" . m10 . "\\\\" . m11 . "\\\\" . m12 . "\\\\" . m13 . "\\\\" . m14 . "\\\\" . m15 . "\\\\" . m16 
| table SMailboxes,conv

And the result looks like this:

SMailboxes	conv
7409c768-ed1b-45dd-8d5d-d36e65af77c1	\\68\\c7\\09\\74\\1b\\ed\\dd\\45\\8d\\5d\\d3\\6e\\65\\af\\77\\c1

All good.

Things get wrong when I add ldapfilter:

index=Exchange Mailboxes=* | rex "'?S:Mailboxes=(?<SMailboxes>[^']+)'?;'?S:StoreObjectIds" | makemv SMailboxes delim=";"
| mvexpand SMailboxes| top SMailboxes limit=50
| rex field=SMailboxes "(?<m1>..)(?<m2>..)(?<m3>..)(?<m4>..)-(?<m5>..)(?<m6>..)-(?<m7>..)(?<m8>..)-(?<m9>..)(?<m10>..)-(?<m11>..)(?<m12>..)(?<m13>..)(?<m14>..)(?<m15>..)(?<m16>..)"
| eval conv="\\\\" . m4 . "\\\\" . m3 . "\\\\" . m2 . "\\\\" . m1 . "\\\\" . m6 . "\\\\" . m5 . "\\\\" . m8 . "\\\\" . m7 . "\\\\" . m9 . "\\\\" . m10 . "\\\\" . m11 . "\\\\" . m12 . "\\\\" . m13 . "\\\\" . m14 . "\\\\" . m15 . "\\\\" . m16 
| table SMailboxes,conv
| ldapfilter debug=true domain="default" basedn="..." search="(msExchMailboxGuid=$conv$)" attrs="name"

-> the result is empty.

In the ldapfilter, if I replace "$conv$" with "\\68\\c7\\09\\74\\1b\\ed\\dd\\45\\8d\\5d\\d3\\6e\\65\\af\\77\\c1" then the query works and attribute "name" is properly returned and added to the table.

How can I make things work with $conv$ in order to have proper results (and not the same static "name" for every event) ?

I've tried so many combinations: from 1 to 4 \, with or without quotes/simple quotes, making $conv$ the whole "search" value, etc. Nothing works.

[patpro]

The $foo$ format is specific to this command. You can find examples in the documentation: https://docs.splunk.com/Documentation/SA-LdapSearch/3.0.4/User/Theldapfiltercommand

I've alreaady tried "conv" alone, "$$conv$$", and other things without any success so far. Currently I'm digging with tcpdump to try and pinpoint the difference between using $conv$ and using the value of conv directly. Anyway, now I can assure you that "$conv$" is properly replaced dynamically by the value of conv when the ldapfilter command kick's in, unfortunately the behavior of the command / the quality of the LDAP request seems to differ and I have an empty result in Splunk when I use "$conv$" instead of the value of conv.

I've also tried to define "conv" as the whole search filter, without success, but now that I'm tcpdumping everything I might as well test again to see what changes…

[richgalloway]

Yeah, I should have read the docs more closely before replying.

Anything helpful in the Job Inspector?  That may be more useful than tcpdump.

[patpro]

Job inspector is of no help here as it has no access to what's going on inside SA-LdapSearch. So it shows that the ldapfilter command is using "$conv$" but does not show if it's expanded and with what value(s).

tcpdump helps a lot: it tells me exactly what SA-LdapSearch is sending to the LDAP/AD server and what reply it got, will full LDAP protocol details. Anyway it's not enough for me (yet) to understand the different behaviors I have 😞

[richgalloway]

Where is the $conv$ token defined?  I see a field by that name, but no token.  The $foo$ format invokes a token that the dashboard must have set before the query ran (except for the map command, but that doesn't apply here).

Have you tried using an eval to build the complete search string?

| eval conv="(msExchMailboxGuid=\\\\" . m4 . "\\\\" . m3 . "\\\\" . m2 . "\\\\" . m1 . "\\\\" . m6 . "\\\\" . m5 . "\\\\" . m8 . "\\\\" . m7 . "\\\\" . m9 . "\\\\" . m10 . "\\\\" . m11 . "\\\\" . m12 . "\\\\" . m13 . "\\\\" . m14 . "\\\\" . m15 . "\\\\" . m16 . ")"
| table SMailboxes,conv
| ldapfilter debug=true domain="default" basedn="..." search='conv' attrs="name"

[patpro]

As I wrote earlier, defining "conv" as the whole search string yields to nothing better : either no result at all, or an error of invalid filter depending how hard I try to add quotes, escapes, etc.

[patpro]

Ok. The ldapfilter command was unable to work for me. So I've had to rethink the way I work. As my LDAP data would not change much on a daily basis, I've choose to create a lookup table every day (scheduled report, early morning) with the ldapsearch command.

This lookup table contains all the info I need to replace the ldapfilter command and probably yields to better performances at runtime.