Hi, We receive daily emails with lists of IOC's for malware and phishing alerts, each email may contain multiple ip address, domains and email addresses and we are trying to extract these to run searches against out web and email logs.  I have the regex working for extraction but it will only extract the first match. I've tried multiple ways of achieving this without success, the current config is: Props.conf EXTRACT-IOCURL = (?P<IOCURL>[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9][\[][\.|@][\]][^\s]{2,}|[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9][\[][\.|@][\]][^\s]{2,}|[a-zA-Z0-9]+[\[][\.|@][\]][^\s]{2,}|[a-zA-Z0-9]+[\[][\.|@][\]][^\s]{2,}) EXTRACT-IOCIP = (?P<IOCIP>\d{1,3}\[\.\]\d{1,3}\[\.\]\d{1,3}\[\.\]\d{1,3}+) The indexed email looks like this.... .... Domains comprised[.]site badsite[.]studio malware[.]live IP addresses 192[.]254[.]71[.]78 192[.]71[.]27[.]202 193[.]182[.]144[.]67  ....   but the current config will only extract the first record for each: IOCURL - comprised[.]site and  IOCIP  - 192[.]254[.]71[.]78. Any ideas how to extract all the domains and IP addresses? Thanks  ... View more

We get a weekly ingest of a data set for our vulnerability management. Each line contains a unique value matching a vulnerability with a server I want to be able to report on: a. how many new vulnerabilities are in this weeks report compared to last week and b. how many vulnerabilities have been fixed (so are not reported) in this weeks list compared to last week I'm looking for splunk to tell me whats new and whats missing week by week but also track these over the long term.  Cant seem to get any meaningful results with a 'set diff' search   Any help gratefully received!! ... View more

The event contains a 'before' and 'after' list of permissions and users SIDs, I can get splunk to extract the entire 'before' list and the entire 'after' list but only as single events. but i need to break it down to list  to indivudal Permission and SID   This it the entire event: 2020-12-07 22:45:51.123 91046 SUCCESS Domain\User Archive Permissions Archive 133481FD9531D0347BBCE92FFF45B4FE11110000evaultcol <Archive ArchiveID="133481FD9531D0347vaultcol" ArchiveName="Last, First"><OldManualSD>😧(A;;CCDCLCSWRPWPDT;;;S-1-5-21-299502267-1960408961-839522115-10875)(A;;CCSW;;;S-1-5-21-299502267-1960408961-839522115-2406856)(A;;CCSW;;;S-1-5-21-299502267-1960408961-839522115-2406857)</OldManualSD><NewManualSD>😧(A;;CCDCLCSWRPWPDT;;;S-1-5-21-299502267-1960408961-839522115-10875)(A;;CCSW;;;S-1-5-21-299502267-1960408961-839522115-2406856)(A;;CCSW;;;S-1-5-21-299502267-1960408961-839522115-2406857)(A;;CCDCSWRPDT;;;S-1-5-21-299502267-1960408961-839522115-3949157)</NewManualSD></Archive> ServerName The 'before' list is between the <OldManualSD> and <\OldManualSD> tags, the 'after' list is between the <NewManualSD> and </NewManualSD> tags The Permissions field is between the ;; and ;;; delimiters and is followed by the SID. There is a varying number of permsissons/SIDs in each event   Can get part way there; ex_OldManual_GP and ex_NewManual_GP fields extract from the "Info" field and the contain the before and after, but trying to get a second extraction based off ex_OldManual_GP and ex_NewManual_GP always fails    from the event above, I would like: OldManual = A;;CCDCLCSWRPWPDT;;;S-1-5-21-299502367-1960408961-839522117-10475 OldManual = A;;CCSW;;;S-1-5-21-299502367-1960408961-839522117-2406456 OldManual = A;;CCSW;;;S-1-5-21-299502367-1960408961-839522117-2406457 NewManual = A;;CCDCLCSWRPWPDT;;;S-1-5-21-299502367-1960408961-839522117-10875 NewManual = A;;CCSW;;;S-1-5-21-299502367-1960408961-839522117-2406456 NewManual = A;;CCSW;;;S-1-5-21-299502367-1960408961-839522117-2406457 NewManua l= A;;CCDCSWRPDT;;;S-1-5-21-299502367-1960408961-839522117-3949147 Any ideas?   my transforms.conf file: [ex_fields_extract] FIELDS = "AuditDate","AuditID","Status","UserName","CategoryName","SubCategoryName","ObjectID","Vault","info","MachineName" DELIMS = "\t" [ex_OldManual_GP] SOURCE_KEY = info REGEX=\>(<OldManualSD>D:)((?P<OldManual_GP>.*))(<\/OldManualSD>) [ex_NewManual_GP] SOURCE_KEY = info REGEX=\>(<NewManualSD>D:)((?P<NewManual_GP>.*))(<\/NewManualSD>) [ex_OldManual_MV] SOURCE_KEY = OldManual_GP REGEX=;;(?P<perm>\w+);;;* MV_ADD=true [ex_NewManual_MV] SOURCE_KEY = NewManual_GP REGEX=(?<NewManual>[^,]+),* MV_ADD=true   my props.conf file [exlogs] REPORT-ex_fields = ex_fields_extract REPORT-mvalue = ex_OldManual_MV, ex_NewManual_MV, ex_NewManual_GP, ex_OldManual_GP SHOULD_LINEMERGE = false   ... View more

We have recently turned on journaling within MS Exchange which basically sends a copy of every item to a journaling mail box. We know the email address the process uses and this appears in the message tracking logs as additional emails - in short journaling has doubled our Splunk licence usage!! We would like to exclude the journaling email address from being indexed. Exchange can't turn off message tracking for certain email addresses We are using UF on the Exchange servers with a load balanced intermediate level of Heavy forwarders. We have tried to apply the exclusion based on this answer - Answer 289736 - how to exclude a sourcetype from being indexed, but using a regex that picks up the journaling email address. The address we want to exclude is: journal@ev.local We've tried it on the Intermediate Heavy forwarders and on the index servers with no effect. The config we have applied is: props.conf [MSExchange:2013:MessageTracking] TRANSFORMS-JournalRemoval = JournalRemoval transforms.conf [JournalRemoval] REGEX =.*journal@ev.* DEST_KEY = queue FORMAT = nullqueue Any ideas why this might not be working? Thanks ... View more

We have layer of servers that act as a internet facing, intermediate forwarding layer providing an extra layer of separation to the hosts that may be connecting via the internet. The hosts are configured via a deployment server with these using the built in load balancing but we cannot find a way of identifying which server a received event has connected through. Ideally we would add a meta tag or a new field with the server name to the event as is passes through an intermediate server, but this does not appear to be possible (unless some one can tell me differently) We can get rough stats from metrics.log for the throughput and volume of data that is being forwarded, but is there any way of identifying what intermediate server an individual event ? ... View more

Owing to the way exchange outputs log files, for some reason we get two versions of the cs_username field username eg employeebob or Email address eg Bob.Employee@work.com Both versions exist in the active directory lookup file we have as "sAMAccountName" and "mail" and I want to get an output field of "Email Address". I can get lookup files to work on either version during a search, but not on both at the same time in the same search. Is there a way of running two lookups on the same file in the same search against the same field? Was looking at the "if" and "where" options, but they don't appear to work. Also, I tried to set two lookups in the same search..... index=msexchange sourcetype="MSWindows:2008R2:IIS" WebApplication="Microsoft-Server-ActiveSync" Cmd=Sync | lookup User_Info mail AS cs_username OUTPUT l AS Location, title AS Title, department AS Department, mail AS "Email Address" | lookup User_Info sAMAccountName AS cs_username OUTPUT l AS Location, title AS Title, department AS Department, mail AS "Email Address" ... View more