I'm very disheartened to hear about this. I run the Idaho Falls Splunk Users Group and will present next month on using Windows Event Logs to find intruders. You are welcome to join our group and attend. I will give many examples you can cut and paste into your Splunk instance. You may need to modify them slightly for your environment, but they will give you an idea of how to build additional use cases. Personally, I never re-invent the wheel. There is a lot of detection out there.  that I look for before building my own. I would start by googling "Splunk," "Threat detection," and then '"Splunk threat detection" github.' There are many people who aren't insecure people, like the people you work with who willingly share their talents. We all got where we are with the help of others. It's sad to hear you are being treated this way. Once you join the user groups, you can contact me directly. I will take time to work with you as time permits. I will also point you to resources that will help you grow in the field and use Splunk for building use cases. I'll include a couple of resources here for you. Another thing to keep in mind is that all threat hunting that finds positive activity should lead to a signature. There are tons of threat-hunting Splunk searches out there, and they can also be used as use cases. You may need to tune them (cast a narrower net) before putting them into production, but they will give you a good idea of how to build out detection.  You will find that industry leaders are always sharing their research and knowledge. Jack Crook was one of my mentors when I was new in this career. Jack and I share the same frustration of attending conferences where many share "theories" but not content. He is big on sharing content and actual Splunk searches and use cases. You can follow his blog. I have included it below. As you grow in this career, remember to not be like the others that have treated you so poorly. Remember, this is a very negative reflection on them, not you! I hope that this helps, and I hope to talk soon!  http://findingbad.blogspot.com  https://www.udemy.com/course/cybersecurity-monitoring-detection-lab/?couponCode=ST9MT22024  https://github.com/splunk/security_content/tree/develop/detections/application https://www.detectionengineering.net https://github.com/west-wind/Threat-Hunting-With-Splunk

So "index=_internal" on your cloud instance doesn't see any data where host=X? Your connections look like they're by IP, but I thought Splunk cloud's little "connect my forwarder up" app did it by DNS entries? Can you confirm one or the other of those things is right? Maybe the forwarder's time is off or TZ is incorrectly specified, have you checked over a longer period like 24 or 48 hours? Also a second point to the TZ issue - if the times are in the future, it can be more difficult to find it in Splunk. Try an *all time* search. I know, it sucks, but one does what one must sometimes.  index=_internal host=*myhost* | stats count by host Try a wildcard ilke I suggested, using "myhost" as any string that should be reasonably unique in the hostname for the host sending in data. You can also confirm what hostname it's sending in as by looking in etc/system/local/server.conf on the UF, there's a "hostname" field. If that's picked something "wrong" then guess what?  Your data will show up as whatever it's picked!

    <layout class="ch.qos.logback.classic.PatternLayout"> <pattern>%msg</pattern> </layout>   in you HEC appender you need to set '%msg' as the pattern, but NOT the one you use for the Console Appender (which is the 'defaultPattern')

If a search "index=_internal" over the last 24 hours is empty, I can think of a couple of reasons. Most likely - your role doesn't have administrative access.  (More specifically, it doesn't have access to the _internal index, which is usually limited to admins).  Either log in as an administrator with access to _internal, or have your Splunk folks add this index to your role. It's also possible that you have DBX installed on a heavy forwarder.  That HF has been told its outputs need to go to your real indexer(s), but it's never been told to *search* the indexer when someone searches for "index=_internal".  The steps you might need are https://docs.splunk.com/Documentation/Splunk/9.2.0/DistSearch/Configuredistributedsearch#Use_Splunk_Web Anyway, if you can confirm the above two things, either one of them is the issue, or you can report back here with what you've found!   -Rich

In my limited testing, SOAR doesn't seem to like handling custom functions within a single code block. It doesn't want to wait for the custom function to actually finish before moving on. For reference, first_code_block just calls a custom function and second_code_block runs phantom.completed() on that function. If you have to call the function from within a code block, you can add a callback. This will make sure the code doesn't move on until the run finishes. I wasn't able to get the callback to work on a second function within the same block. (One note on this: Phantom will call the last two lines of the code block before the custom function finishes) phantom.custom_function(... callback=second_code_block) The easiest method by far is to just put each custom function into their own block, then do whatever processing you need in a custom code block below. By default, SOAR will wait for any simultaneous blocks to finish before running the next step.  

Thank you for an update. I tried suggested SPL and added rename field to see if data exists ------------------- | eval Date = strftime(strptime("policy_applied_at","%FT%T.%6NZ"), "%b-%d-%Y") | eval Time = strftime(strptime("policy_applied_at","%FT%T.%6NZ"), "%H:%M") | rename "policy_applied_at" as "Last Refresh Time" | table "Last Refresh Time", Date, Time ------------- The rename, but not trimmed field has data, other two are empty Last Refresh Time Date Time 2024-02-19T11:16:58.930104Z     2024-02-19T11:16:54.980418Z     2024-02-19T11:18:44.875386Z    

The first one did end up working for me. The second one for whatever reason was throwing Error in 'SearchParser': Mismatched ']'. Not a big deal for me since the first one works, but figured I'd mention it. | rex field=_raw "Key\": \"Owner\", \"ValueString\": \"(?<Owner>[^"])\"}," The second one is what I thought I was doing... capturing everything until it saw "},    Thank you for helping me with this!

So Im understanding here that yeah, syslog itself doesn't lend well to being balanced. Splunk Universal forwarder should not be used to receive Syslog traffic, but a Heavy forwarder "can". And if I wanted to load balance a pair of Splunk UF's then I should setup a real load balancer.. And if I want to LB syslog,  should also setup a real LB and have the syslog and splunk functions on dedicated hosts.  this look accurate?

If policy_refresh_at is a string, you could ty parsing it (to an epoch timestamp) before formatting, something like this: |eval Date = strftime(strptime(policy_refresh_at,"%FT%T.%6NZ"), "%b-%d-%Y") | eval Time = strftime(strptime(policy_refresh_at,"%FT%T.%6NZ"), "%H:%M")