I think this works great, havent tested fully yet but it looks right to me. Thanks! ... View more

@PickleRick as per usual, your suggestion led me directly to the correct answer! I did not have to change or apply any specific sourcetype, the only change that I made was moving props.conf from /etc/system/local on the HF to the Splunk_TA_Windows/local directory on the HF and restarted Splunk and it worked like a charm.  Thanks very much for all your help, I will accept your response as the solution!  ... View more

Hi @PickleRick, thanks for your response it's always appreciated! To confirm, yes the path of forwarding we are currently using is UF->HF->Indexer (Splunk Cloud) To give some more details, I am currently using the Splunk TA Add On for Windows, with a modified inputs.conf that I stored within the local directory of the app itself. With all the other options disabled, here is the one enabled input for that app: [WinEventLog://Security] disabled = 0 start_from = oldest index=wineventlog current_only = 0 evt_resolve_ad_obj = 1 checkpointInterval = 5 renderXml=false blacklist = 4689,5158 blacklist1 = EventCode="4688" Message="(?:New Process Name:).+(?:\\Google\\Chrome\\Application\\chrome.exe)|.+(?:\\AppData\\Local\\Microsoft\\Teams\\current\\Teams.exe)|.+(?:C:\\Program Files\\SplunkUniversalForwarder\\bin\\)|.+(?:C:\\Program Files\\SplunkUniversalForwarder\\bin\\splunk.exe)"  So I have defined no sourcetype, but I have defined an index. This is exactly the same configuration as in my lab environment. On the HF side of things I have also installed the Splunk Add on for Windows, but am storing the props.conf file within /etc/system/local. Does this need to be stored within the Windows Add on app on the heavy forwarder? Here is the exact syntax for props.conf that I am using: [(?::){0}WinEventLog...] SEDCMD-clean_fluff_from_winsec_events_this_event = s/This event is generated[\S\s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_token_elevation = s/Token Elevation Type indicates[\S\s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_certificate_info = s/Certificate information is only provided if[\S\s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_the_subject_fields = s/The subject fields indicate the account[\S\s\r\n]+$//g Many thanks ... View more

@gcusello thanks again for your quick & thorough response, these will always be rewarded with karma points.  Unfortunately I'm not sure if we're on the same page, I am currently using blacklists on the UF to filter out noisy or redundant events. However my understanding is that Splunk UF is not able to filter out a section of a log, and instead doesn't send that entire event log if something matches within it to what is on the blacklist. Correct me if I'm wrong.  What I'm trying to accomplish is essentially when a user logs in (4624), filter out all of the flavor text at the bottom, which is shown below in an example: Network Information: Workstation Name: EXWKS Source Network Address: 123.45.67.89 Source Port: 0 Detailed Authentication Information: Logon Process: User32 Authentication Package: Negotiate Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The impersonation level field indicates the extent to which a process in the logon session can impersonate. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.  In other words, how do I trim out all of the text after the "detailed login information" section? I know that this is possible using self-hosted or on-prem indexers and search heads, since I've done it. If someone knows how to accomplish this in Splunk cloud please let me know ... View more

Hi @gcusello, thanks for your quick and thorough response.  To give some context, I am actually already using the blacklist utility within the UF to filter out unwanted codes. What I am looking to do is trim out redundant text at the end of important event codes and messages, so essentially decrease the volume and overhead of these messages without blacklisting them entirely.  I have done this successfully in a lab environment by placing the following props.conf file within /etc/system/local on the SH/indexer, but haven't tested it when it's on a HF. [(?::){0}WinEventLog...] SEDCMD-clean_fluff_from_winsec_events_this_event = s/This event is generated[\S\ s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_token_elevation = s/Token Elevation Type i ndicates[\S\s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_certificate_info = s/Certificate informati on is only provided if[\S\s\r\n]+$//g SEDCMD-clean_fluff_from_winsec_events_the_subject_fields = s/The subject fields indicate the account[\S\s\r\n]+$//g  Please advise on whether configs made with props.conf can actually be placed on the HF or if I need to contact Splunk Cloud in order to put it on the cloud-based indexers.  Thanks ... View more

When I run the subsearch it doesn't take painfully long, but your advice is definitely noted and I'll likely proceed forward with the use of CSVs rather than JSON to improve speed/efficiency of the search result.  Thank you again for all your help, I'll be sure to accept your answer as the solution for others who may have a similar issue down the road.  ... View more

@PickleRick thank you for your thorough and detailed response. In terms of ruling out potential issues, I have confirmed that running the following search returns the correct results for undesirable IPv4 addresses from index=threat: index=threat earliest=-24h@h latest=now | rename ipv4{} as dest_ip | mvexpand dest_ip | dedup dest_ip | table dest_ip  When I run the following, I actually get exactly the information I want, which is incredible, and thank you so much for walking me through this process: index=pan earliest=-40m@m latest=now vsys_name=Browser [search index=threat earliest=-24h@h latest=now | rename ipv4{} as dest_ip | mvexpand dest_ip | dedup dest_ip | table dest_ip]  However this search is probably the lengthiest one in terms of time taken to complete (atleast 90 seconds both times I have tried it, which to me is infeasible long term). Would you say the solution to this would be loading the JSON threat list into a lookup table and accessing the information that way? Are CSVs generally preferred for lookup tables, or does JSON work just as well? Thank you again for helping me with this, as this search works absolutely perfectly, and even though I can't really understand why, it's really all about reducing the time it takes to complete now. Karma earned and given! ... View more

@PickleRick  I have since found what I believe is the correct syntax for what you suggested: index=pan [ search index=threat earliest=-24h@h latest=now | rename ipv4{} as dest_ip | mvexpand dest_ip | dedup dest_ip ] | table dest_ip However this search returns no results, when both dest_ip from the pan index, as well as the ipv4{} field from the threat index are populated with data.  Any ideas for where to go from here? Could be that I'm just barely missing something ... View more

@PickleRick thank you for your quick response, and I apologize for my lack of clarity.  Here is what I am trying to accomplish in a bit more detail: index="pan" will have fields like: user, dest_ip, _time, bytes_in, source_ip, etc.  index="threat" has only one field: "ipv4{}" I would like to combine these searches into one, where every 10-15 minutes or so it looks at strictly the dest_ip field in the "pan" index, as well as the "ipv4{}" field from the "threat" index, and finds matches between those two fields, and then returns those matches AS WELL AS the corresponding fields from index="pan" such as user, _time, bytes_in, etc.  For example:  Within index="pan" dest_ip                    user                     _time 1.1.1.1                       santa                  4pm 172.168.1.1            tom                       2pm 8.8.8.8                       daniel                   1pm 10.0.5.1                    peter                      4am Within index="threat" ipv4{} 10.0.5.1 1.1.1.1 Would return a table that looks like this: dest_ip                    user                     _time 1.1.1.1                       santa                  4pm 10.0.5.1                    peter                      4am I think this is much more simple than I am currently making it in my head, as this seems like it would be a common search that many admins would need to implement. Once again, any help or guidance would be very much appreciated, and showered with karma! Thanks in advance ... View more

Thank you both for your responses, however I am still having some trouble with this search.  I not only need to match the fields between the two multifield values, but I need to combine that result with the existing data from the traffic logs so I'm able to find the source IP, user, time, etc of the matched event.  index=pan earliest=-10m@m latest=now vsys_name=Browser | dedup dest_ip | append [ search index=threats earliest=-24h@h latest=now | rename ipv4{} as bad_ip | mvexpand bad_ip | dedup bad_ip ] | eval ipv4List = mvappend(dest_ip,bad_ip) | stats count by ipv4List | search count>1  The search above is what I currently have working that returns IPs found on both of these lists, but I don't see any way to actually return the fields associated with this address within dest_ip without running another subsearch using join to actually get that information, which is obviously less than ideal from a resource utilization standpoint.  I know that I am making this harder on myself than needed, but I am very new to SPL so any syntax that would fully encompass what I am trying to accomplish would be greatly appreciated and rewarded with karma.  Any ideas or advice are not taken for granted, and thank you in advance.  ... View more