Hello, I’m trying to tune Machine Learning Toolkit in order to detect authentication abuse on a web portal (based upon Lemon LDAP-NG). My logs look like this: (time/host/... header) client=(IP address) user=(login) sessionID=(session-id) mail=(user email address) action=(various statuses: connected / non-existent user / wrong pwd…)   I would like to train the Machine Learning Toolkit so that I can detect anomalies. Those anomalies can be: - that client has made auth attempts for an unusual number of logins - that client has made auth attempts for both non-existing and existing users - …   So far it fails hard.   I’ve trained a model like this on approx. a month of data:   index="webauth" ( TERM(was) TERM(not) TERM(found) TERM(in) TERM(LDAP) ) OR TERM(connected) OR TERM(credentials) linecount=1 | rex "action=(?<act>.*)" | eval action=case(match(act,".* connected"), "connected", match(act,".* was not found in LDAP directory.*"), "unknown", match(act, ".* credentials"),"wrongpassword") | bin span=1h _time | eventstats dc(user) AS dcUsers, count(user) AS countUsers BY client,_time,action|search dcUsers>1|stats values(dcUsers) AS DCU,values(countUsers) AS CU BY client,_time,action| eval HourOfDay=strftime(_time,"%H") | fit DensityFunction CU by "client,DCU" as outlier into app:TEST     Then I’ve tested the model on another time interval where I know there is a big anomaly, by replacing the fit directive by "apply (model-name) threshold=(various values)". No result.   So I guess I’m not on the right track to achieve this. Any help appreciated!   ... View more

Hello, I want to get Rspamd logs into Splunk with every info available. The best I could do with Rspamd config yields to this: 2023-11-03 13:02:24 #56502(rspamd_proxy) <7fcfc8>; lua; [string "return function (t...:4: METATEST {"qid":"8BC8C2F741","user":"unknown","ip":"188.68.A.B","header_from":["foo bar via somelist <somelist@baz.org>"],"header_to":["list <somelist@baz.org>"],"header_subject":["proper subject"],"header_date":["Fri, 3 Nov 2023 08:00:43 -0400 (EDT)"],"scan_time":2457,"rcpt":["me@myself.net"],"size":6412,"score":-5.217652,"subject":"proper subject","action":"no action","message_id":"4SMK7v2HQTzJrP1@spike.bar.org","fuzzy":[],"rspamd_server":"rack.myself.net","from":"somelist-bounces@baz.org","symbols":[{"score":-0.500000,"group":"composite","groups":["composite"],"name":"RCVD_DKIM_ARC_DNSWL_MED"},{"score":0,"group":"headers","groups":["headers"],"name":"FROM_HAS_DN"},{"score":0,"group":"headers","options":["somelist@baz.org","somelist-bounces@baz.org"],"groups":["headers"],"name":"FROM_NEQ_ENVFROM"},{"score":-0.010000,"group":"headers","groups":["headers"],"name":"HAS_LIST_UNSUB"},{"score":0,"group":"headers","options":["somelist@baz.org"],"groups":["headers"],"name":"PREVIOUSLY_DELIVERED"},{"score":-1,"group":"abusix","options":["188.68.A.B:from"],"groups":["abusix","rbl"],"name":"RWL_AMI_LASTHOP"},{"score":-0.100000,"group":"mime_types","options":["text/plain"],"groups":["mime_types"],"name":"MIME_GOOD"},{"score":-0.200000,"group":"headers","options":["mailman"],"groups":["headers"],"name":"MAILLIST"},{"score":1,"group":"headers","groups":["headers"],"name":"SUBJECT_ENDS_QUESTION"},{"score":-0.200000,"group":"policies","options":["+ip4:188.68.A.B"],"groups":["policies","spf"],"name":"R_SPF_ALLOW"},{"score":-1,"group":"policies","options":["list.sys4.de:s=2023032101:i=1"],"groups":["policies","arc"],"name":"ARC_ALLOW"},{"score":0,"group":"ungrouped","options":["asn:19xxxx, ipnet:188.68.A.B/20, country:XY"],"groups":[],"name":"ASN"},{"score":0.100000,"group":"headers","groups":["headers"],"name":"RCVD_NO_TLS_LAST"},{"score":0,"group":"headers","groups":["headers","composite"],"name":"FORGED_RECIPIENTS_MAILLIST"},{"score":0,"group":"policies","options":["baz.org:+","bar.org:-"],"groups":["policies","dkim"],"name":"DKIM_TRACE"},{"score":0,"group":"headers","groups":["headers"],"name":"REPLYTO_DOM_NEQ_FROM_DOM"},{"score":0,"group":"policies","options":["bar.org:s=dktest"],"groups":["policies","dkim"],"name":"R_DKIM_REJECT"},{"score":-2.407652,"group":"statistics","options":["97.28%"],"groups":["statistics"],"name":"BAYES_HAM"},{"score":0,"group":"headers","groups":["headers"],"name":"TO_DN_ALL"},{"score":0,"group":"composite","groups":["composite"],"name":"DKIM_MIXED"},{"score":-0.200000,"group":"policies","options":["baz.org:s=20230217-rsa"],"groups":["policies","dkim"],"name":"R_DKIM_ALLOW"},{"score":0,"group":"headers","options":["3"],"groups":["headers"],"name":"RCVD_COUNT_THREE"},{"score":-0.600000,"group":"rbl","options":["188.68.A.B:from","188.68.A.B:received","168.100.A.B:received"],"groups":["rbl","dnswl"],"name":"RCVD_IN_DNSWL_MED"},{"score":-0.100000,"group":"rbl","options":["188.68.A.B:from"],"groups":["rbl","mailspike"],"name":"RWL_MAILSPIKE_GOOD"},{"score":0,"group":"policies","options":["baz.org"],"groups":["policies","dmarc"],"name":"DMARC_NA"},{"score":0,"group":"headers","options":["1"],"groups":["headers"],"name":"RCPT_COUNT_ONE"},{"score":0,"group":"mime_types","options":["0:+"],"groups":["mime_types"],"name":"MIME_TRACE"},{"score":0,"group":"headers","groups":["headers","composite"],"name":"FORGED_SENDER_MAILLIST"},{"score":0,"group":"headers","groups":["headers"],"name":"TO_EQ_FROM"},{"score":0,"group":"headers","options":["foo@bar.org"],"groups":["headers"],"name":"HAS_REPLYTO"}]}   Currently I’m extracting JSON with a props.conf & a transforms.conf: props.conf [rspamd] KV_MODE = json TRANSFORMS-json_extract_rspamd = json_extract_rspamd transforms.conf [json_extract_rspamd] SOURCE_KEY = _raw DEST_KEY = _raw LOOKAHEAD = 10000 #REGEX = ^([^{]+)({.+})$ REGEX = ^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d) (#\d+)\(([^)]+)\) ([^;]+); lua[^{]+{(.+})$ FORMAT = {"date":"$1","ida":"$2","process":"$3","idb":"$4",$5 CLONE_SOURCETYPE = _json I end up with this in splunk:                                     From here, I have 2 problems. 1st problem: contrary to native JSON (from my Amavis logs for example), Splunk does not extract nor process basic stats about fields unless I’m explicitly extract them… That’s quite a pain. Is there a way / config setting to instruct Splunk to automagically extract every fields? 2nd problem: this JSON is crap. Every object in "symbols[]" looks like this: It’s almost unuseable as it prevent me from linking the name of the symbol to its score and to its options. Is there a parsing option / function I could use to reliably transform this into something I can work with? A good result could be turning { group: abusix groups: [ abusix rbl ] name: RWL_AMI_LASTHOP options: [ A.B.C.D:from ] score: -1 } into RWL_AMI_LASTHOP: [ group: abusix groups: [ abusix rbl ] name: RWL_AMI_LASTHOP options: [ A.B.C.D:from ] score: -1 ]   I’m open to suggestions, I’ve been working for years with the great JSON logs of Amavis (perfect parsing and usability). This problem is new to me… ... View more

Hello, I'm trying to use ldapfilter to add some info to events I collect from MS Exchange but as soon as my ldapfilter query is dynamic (makes use of $field$) it does not return anything. My initial search looks like this:     index=Exchange Mailboxes=* | rex "'?S:Mailboxes=(?<SMailboxes>[^']+)'?;'?S:StoreObjectIds" | makemv SMailboxes delim=";" | mvexpand SMailboxes| top SMailboxes limit=50 | rex field=SMailboxes "(?<m1>..)(?<m2>..)(?<m3>..)(?<m4>..)-(?<m5>..)(?<m6>..)-(?<m7>..)(?<m8>..)-(?<m9>..)(?<m10>..)-(?<m11>..)(?<m12>..)(?<m13>..)(?<m14>..)(?<m15>..)(?<m16>..)" | eval conv="\\\\" . m4 . "\\\\" . m3 . "\\\\" . m2 . "\\\\" . m1 . "\\\\" . m6 . "\\\\" . m5 . "\\\\" . m8 . "\\\\" . m7 . "\\\\" . m9 . "\\\\" . m10 . "\\\\" . m11 . "\\\\" . m12 . "\\\\" . m13 . "\\\\" . m14 . "\\\\" . m15 . "\\\\" . m16 | table SMailboxes,conv     And the result looks like this: SMailboxes conv 7409c768-ed1b-45dd-8d5d-d36e65af77c1 \\68\\c7\\09\\74\\1b\\ed\\dd\\45\\8d\\5d\\d3\\6e\\65\\af\\77\\c1   All good. Things get wrong when I add ldapfilter:     index=Exchange Mailboxes=* | rex "'?S:Mailboxes=(?<SMailboxes>[^']+)'?;'?S:StoreObjectIds" | makemv SMailboxes delim=";" | mvexpand SMailboxes| top SMailboxes limit=50 | rex field=SMailboxes "(?<m1>..)(?<m2>..)(?<m3>..)(?<m4>..)-(?<m5>..)(?<m6>..)-(?<m7>..)(?<m8>..)-(?<m9>..)(?<m10>..)-(?<m11>..)(?<m12>..)(?<m13>..)(?<m14>..)(?<m15>..)(?<m16>..)" | eval conv="\\\\" . m4 . "\\\\" . m3 . "\\\\" . m2 . "\\\\" . m1 . "\\\\" . m6 . "\\\\" . m5 . "\\\\" . m8 . "\\\\" . m7 . "\\\\" . m9 . "\\\\" . m10 . "\\\\" . m11 . "\\\\" . m12 . "\\\\" . m13 . "\\\\" . m14 . "\\\\" . m15 . "\\\\" . m16 | table SMailboxes,conv | ldapfilter debug=true domain="default" basedn="..." search="(msExchMailboxGuid=$conv$)" attrs="name"     -> the result is empty. In the ldapfilter, if I replace "$conv$" with "\\68\\c7\\09\\74\\1b\\ed\\dd\\45\\8d\\5d\\d3\\6e\\65\\af\\77\\c1" then the query works and attribute "name" is properly returned and added to the table. How can I make things work with $conv$ in order to have proper results (and not the same static "name" for every event) ? I've tried so many combinations: from 1 to 4 \, with or without quotes/simple quotes, making $conv$ the whole "search" value, etc. Nothing works. ... View more

Few days ago, a developer has added to John the Ripper the ability to timestamp every line of logs, allowing me to feed them to Splunk in order to derive statistics from this data. JtR's logs are complex, starting with a header of several lines giving details about the current session: 2016-02-23T20:43:57+0100 1 0:00:00:00 Starting a new session 2016-02-23T20:43:57+0100 1 0:00:00:00 Loaded a total of 15151 password hashes with 15151 different salts 2016-02-23T20:43:57+0100 1 0:00:00:00 Sorting salts, for performance 2016-02-23T20:43:57+0100 1 0:00:00:00 Remaining 14755 password hashes with 14755 different salts 2016-02-23T20:43:57+0100 1 0:00:00:00 - Node numbers 1-4 of 4 (fork) 2016-02-23T20:43:57+0100 1 0:00:00:00 Command line: ./john --fork=4 --wordlist=password.lst --rules=JUMBO dump.txt 2016-02-23T20:43:57+0100 1 0:00:00:00 - UTF-8 input encoding enabled 2016-02-23T20:43:57+0100 1 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file ... After this header, each cracked password yields to a new line, and periodically an info line is printed: 2016-02-23T20:50:38+0100 2 0:00:06:41 + Cracked foo 2016-02-23T20:50:50+0100 1 0:00:06:53 + Cracked bar 2016-02-23T20:50:47+0100 3 0:00:06:49 + pot sync removed 2 hashes; Remaining 14583 hashes with 14583 different salts The session can end by exhaustion (no more passwords, or no more candidates), or by interrupt (ctrl-c). Message is either: 2016-02-23T22:17:06+0100 2 0:01:33:09 Session completed or 2016-02-24T11:03:47+0100 4 0:00:38:22 Session aborted I would like to be able to track down and compare efficiency of different sessions. Each session has different parameters that can be read from the header ("Command line:" and other header log lines). Efficiency can be measured as the number of "Cracked" in a given period of time. Difficulty for me is to make a timechart , chart, or stats calculation for each session from a single Splunk request so everything is displayed on one graph/chart. I've started investigating transactions, that's quite easy with a first line containing "Starting a new session" and the last containing "Session *ted". But it seems I can't work inside a transaction to create timechart. Any hint? ... View more