Splunk Search

Hi! I would like to create a field extraction of a multi-valued field.

Explorer

This is a typical relevant line from logs:

[28/Jun/2017:07:26:04 -0400] conn=9354 op=7 msgId=8 - SRCH base="o=company" scope=2 filter="(&(|(objectclass=mailgroup)(objectclass=person)(objectclass=alias))(!(objectclass=moderatedgroup))(mailalternateaddress=owner-john.doe@company.com))" attrs="cn uid mailaddress uniquemember" 

The "attrs=" is the required text and the "cn uid mailaddress uniquemember" -attributes would need to be extracted separetly..the space is the delimiter between them. With the wizard I only able to select one of them. But I need to count them one by one .... | chart count by host,attrib_extraction
Thanks!

Tags (2)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi,

So assuming your key value extraction is working fine and there is an attribute named attrs already then you can simply do:

| eval attrib_extraction = split(attrs, " ")

If that's not the case you can extract key value pairs from your raw data using the kv command.
Alternatively a simple regex like the following should do the trick too:

| rex "attrs=\"(?<attrib_extraction_all>[^\"]+)\""
| rex max_match=0 field=attrib_extraction_all "(?<attrib_extraction>\S+)"

If you then need to expand this multivalued field simply do:

| mvexpand attrib_extraction

Hope that helps.

Thanks,
J

View solution in original post

SplunkTrust
SplunkTrust

Hi,

So assuming your key value extraction is working fine and there is an attribute named attrs already then you can simply do:

| eval attrib_extraction = split(attrs, " ")

If that's not the case you can extract key value pairs from your raw data using the kv command.
Alternatively a simple regex like the following should do the trick too:

| rex "attrs=\"(?<attrib_extraction_all>[^\"]+)\""
| rex max_match=0 field=attrib_extraction_all "(?<attrib_extraction>\S+)"

If you then need to expand this multivalued field simply do:

| mvexpand attrib_extraction

Hope that helps.

Thanks,
J

View solution in original post

Explorer

expected output with chart command

 cn uid mailaddress sn  office

host1 564 444 56 77 2222
host2 57 565 11 676 44
host3 569 66 45 565 33

0 Karma

SplunkTrust
SplunkTrust

I have tested your example above in my lab and managed to get the output you are looking for.
Keep in mind it might be easier than the query I'm showing you below, but I don't have all the information about your field extractions, etc.

Hope it helps.

QUERY

| makeresults
| eval raw = "[29/Jun/2017:03:28:37 -0400] conn=16035519 op=32155 msgId=32156 - SRCH base=\"msfwid=1281620,ou=people,o=company\" scope=0 filter=\"(objectclass=)\" attrs=ALL; [29/Jun/2017:03:28:37 -0400] conn=16047221 op=54 msgId=55 - SRCH base=\"msfwid=785323,ou=people,o=company\" scope=2 filter=\"(objectclass=msperson)\" attrs=\"personaltitle givenname msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor\"; [29/Jun/2017:03:28:37 -0400] conn=16035519 op=32154 msgId=32155 - SRCH base=\"msfwid=1487082,ou=people,o=company\" scope=0 filter=\"(objectclass=)\" attrs=ALL; [29/Jun/2017:03:28:37 -0400] conn=16047221 op=53 msgId=54 - SRCH base=\"msfwid=1260584,ou=people,o=company\" scope=2 filter=\"(objectclass=msperson)\" attrs=\"personaltitle givenname msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor\"; [29/Jun/2017:03:28:37 -0400] conn=16047226 op=102 msgId=55603 - SRCH base=\"msfwid=1421236,ou=people,o=company\" scope=2 filter=\"(objectclass=msperson)\" attrs=\"personaltitle msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor\""
| eval raw = split(raw, ";")
| mvexpand raw
| rex field=raw "attrs=\"?(?<attrib_extraction_all>[^\"]+)\"?"
| eval host = if (match(attrib_extraction_all, "ALL"), "myhost1", "myhost2")
| rex max_match=0 field=attrib_extraction_all "(?<attrib_extraction>\S+)"
| chart limit=0 count by host,attrib_extraction 

OUTPUT (see picture below)

alt text

http://i.imgur.com/YUHBsw1.png

0 Karma

Explorer

This works great with that static data you have put after the | eval raw = .... but is it possible to use the data which is filtered out from logs with the / index=log-3155-prod* eventtype=fwdldap-all-prod-hosts sourcetype="ldap-infra:access" laasappId="ldap-infra/prod/ldapFWD_PROD" "SRCH base=" /-command?
Also goes to the hosts..hosts is recognized as interesting filed..so I also need it not as static data given by me. Your answer is really appreciated! Thank you very much in advance!

0 Karma

SplunkTrust
SplunkTrust

You should be able to append the following to your base search and get the same result:

your base search
| rex "attrs=\"?(?<attrib_extraction_all>[^\"]+)\"?"
| rex max_match=0 field=attrib_extraction_all "(?<attrib_extraction>\S+)"
| chart limit=0 count by host, attrib_extraction 

By the way, if you liked the answer please don't forget to mark it as such so that others can benefit from it.

0 Karma

Explorer

Bull's eye! Great! Thanks!

0 Karma

Explorer

..here is another example:
---search command:
index=log-3155-prod* eventtype=fwdldap-all-prod-hosts sourcetype="ldap-infra:access" laasappId="ldap-infra/prod/ldapFWD_PROD" "SRCH base="
---result:
[29/Jun/2017:03:28:37 -0400] conn=16035519 op=32155 msgId=32156 - SRCH base="msfwid=1281620,ou=people,o=company" scope=0 filter="(objectclass=)" attrs=ALL
[29/Jun/2017:03:28:37 -0400] conn=16047221 op=54 msgId=55 - SRCH base="msfwid=785323,ou=people,o=company" scope=2 filter="(objectclass=msperson)" attrs="personaltitle givenname msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor"
[29/Jun/2017:03:28:37 -0400] conn=16035519 op=32154 msgId=32155 - SRCH base="msfwid=1487082,ou=people,o=company" scope=0 filter="(objectclass=
)" attrs=ALL
[29/Jun/2017:03:28:37 -0400] conn=16047221 op=53 msgId=54 - SRCH base="msfwid=1260584,ou=people,o=company" scope=2 filter="(objectclass=msperson)" attrs="personaltitle givenname msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor"
[29/Jun/2017:03:28:37 -0400] conn=16047226 op=102 msgId=55603 - SRCH base="msfwid=1421236,ou=people,o=company" scope=2 filter="(objectclass=msperson)" attrs="personaltitle givenname msmiddleinitial sn cn mail telephonenumber employeenumber uid msfwid businesscategory departmentnumber title building floor"
----field extraction:

 (?=[^a]*(?:attrs=|a.*attrs=))^(?:[^"\n]*"){5}(?P<"attrib_extraction">[^"]+)

"attrib_extraction" -needed to added in this way even if inserted as code example...so please ignore the " " here.

Thanks

0 Karma

Explorer

Thank you! I have tried all the versions but in the end I got the attrs truncated.. 😞
I will try to be more specific..this is the filed extrcation..it works:
(?=[^a](?:attrs=|a.attrs=))^(?:[^"\n]*"){5}(?P[^"]+)
"search commad"..."SRCH base=" | chart count by attrib_extraction ---gives the same without the commands you have posted.:

attrib_extraction count
* 69994
* aci 120
* modifytimestamp 312
1.1 76545
1.1 uid cn objectclass 3
assistant l cn mail maildrop personaltitle secretary title uid workertype 24
attributetypes objectclasses ditcontentrules 90
...so the extracted character line is handled as one..what I would need, is to count every attribute separately. If the user / program does a search for 20 attribute than after the atrrs="
20 attribute will be listed. But it might be a * if they want to have all attribute of the object(s). I hope that was more clear now. Thank you for your help! Best regards! Károly

0 Karma

SplunkTrust
SplunkTrust

Hi, sorry I'm still confused.
Can you post another example but using the code button (the one with 0's and 1's above) so that especial characters are not escaped?
Ideally I would like to see a couple of log lines and then the expected output in Splunk.

Thanks,
J

0 Karma

SplunkTrust
SplunkTrust

For a search-time extraction, try this.

... | rex field=attrs "(?<cn>[^\s]+)\s(?<uid>[^\s]+)\s(?<mailaddress>[^\s]+)\s(?<uniquemember>.+)" | chart count by host cn uid mailaddress uniquemember
---
If this reply helps you, an upvote would be appreciated.
0 Karma

Explorer

Thanks, but the problem is that we have got more then 400 kind of attributes..not only this 4 what you see in the example line. It can be * as well if the LDAP search is for to get back any attribute of the object which mathches as well or it can be anything else like attrs="memberurl uniquemember objectclass uid cn" or attrs="uid cn" etc.

0 Karma