This helped us find a hint of the problem in another environment - thank you for the post! ... View more

If this only happens upon a restart of the server, you will need to ensure that the variable is declared in your startup script. Splunk will be configured to boot-start in a best practice scenario. However, if the boot-start script does not declare that variable, it will not exist when Splunk is started by init or systemd. Your shell RC file does not execute upon boot or init/systemd starting services; only upon interactive login. Keep it there. It is important for when Splunk is restarted by a human running the shell. Or automation for that matter. Solution: declare the variable. For init, you can simply declare it just like you already do in your shell RC in the start) f. For systemd, you will declare it like this in the [Settings] stanza:     Environment="REQUESTS_CA_BUNDLE=/full/path/to/your/internal/ca/pem/file/with/no/variables"       ... View more

I know this is five years later... but people are bound to run across this post. If the saved search is shared to the app, then the user context of your API call should be: nobody If you follow the instructions as written in the solution, sending a POST this way, even when an identically-named saved search exists in the App or Global context, because the POST is in the user's context, it POSTs to the user's private savedsearches.conf within the same app context, not the app's savedsearches.conf. If you want to update a saved search which is shared into an app, you must change the user context to nobody when you POST. Keep in mind, you're still in servicesNS as this point. The only thing that changes is the user context. ... View more

OK, progress... how is your regex?  😉 renderXML won't work on an output stanza.  Somehow we need to pipe the feed into a (hopefully native) XML transformation. Or possibly do it the other way around... rendering XML at the UFWs and reconstructing to a "normal" event on the HFW side. I'm not sure which is better or worse TBH.  I don't have a Windows box to play with either. Now that I think of it... Your appetite may vary... what about doing native Windows Event Forwarding to the intermediate forwarder you just created... take your feed from there with LogRhythm and leave Splunk out of the equation.  Your UFW config would go back to the way it was, sending data straight to Splunk Cloud, and you use the WEF feed to do what needs to be done for the corporate/parent company requirement. ... View more

Take a look at this documentation.  That's the beginning of what you will need to do. You will need a props stanza (I recommend basing it on the sourcetype) to call two transforms.  One targets Splunk Cloud directly, the other targets LogRhythm.  And the two tcpouts to match up with the transforms. ... View more

Same format (export from certmgr.msc as Base-64 encoded X.509 (.CER)).  Then change the extension. The only difference is that the server PEM requires the full certificate chain in the file:  Subject, Intermediate(s), Root. You'll also need to specify the issuing CA and its chain all the way to the root in a separate PEM file which is referenced in server.conf. Relevant Documentation for Forwarding Relevant Documentation for Splunk-to-Splunk All about PEM files and third-party certificates Finally, a word of caution.  Keep good track of where you install ALL of your certificates.  You do not want them to expire.  This happened to me, and it wasn't pretty.  If you have a two-year validity, just renew them all annually. ... View more

Yeah, I would give that a try with a couple servers to see if it behaves how you want it to.  I haven't tried this myself before.  It should be possible. ... View more

Hi there!  I assume you are using the Palo Alto TA.  It has a few layers to it where it recognizes patterns in the logs to classify it beyond the default pan:firewall sourcetype. First Change Look at the default transforms.conf.  You're going to need to change the REGEX so it matches your changed format.  Remember to put this stanza in the local folder of the TA.     [pan_traffic] DEST_KEY = MetaData:Sourcetype REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC, FORMAT = sourcetype::pan:traffic     Becomes...   [pan_traffic] DEST_KEY = MetaData:Sourcetype REGEX = ^[^,]+,TRAFFIC, FORMAT = sourcetype::pan:traffic     Second Change Next, also in transforms.conf, you'll also need to tweak this stanza to match your new format:     [extract_traffic] DELIMS = "," FIELDS = "future_use1","receive_time","serial_number","type","log_subtype","version","generated_time","src_ip","dest_ip","src_translated_ip","dest_translated_ip","rule","src_user","dest_user","app","vsys","src_zone","dest_zone","src_interface","dest_interface","log_forwarding_profile","future_use3","session_id","repeat_count","src_port","dest_port","src_translated_port","dest_translated_port","session_flags","transport","action","bytes","bytes_out","bytes_in","packets","start_time","duration","http_category","future_use4","sequence_number","action_flags","src_location","dest_location","future_use5","packets_out","packets_in","session_end_reason","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name","action_source","src_vm","dest_vm","tunnel_id","tunnel_monitor_tag","tunnel_session_id","tunnel_start_time","tunnel_type"     Becomes... [extract_traffic] DELIMS = "," FIELDS = "$receive_time","$type","$subtype","$time_generated","$src","$dst","$natsrc","$natdst","$rule","$srcuser","$dstuser","$app","$to","$from","$inbound_if","$outbound_if","$repeatcnt","$sport","$dport","$natsport","$natdport","$flags","$proto","$action","$bytes","$bytes_sent","$bytes_received","$packets","$category","$seqno","$srcloc","$dstloc","$pkts_sent","$pkts_received","$session_end_reason","$device_name","$action_source" ... View more

Have you talked to your parent company about switching to Splunk?  😉 Just kidding.  First, are you sending directly from the UFs to Splunk Cloud?  Or are you sending to an intermediate forwarder first?  If you have an intermediate forwarder with that data feed already, you are in luck and have some flexibility. If you are sending straight to Splunk Cloud, the only option I see is still an intermediate forwarder running Splunk Enterprise.  Keep in mind, if you do this, you will need a license for that intermediate forwarder if you don't already have one.  If you route both feeds through the intermediate forwarder, you won't get hit twice on the license ingest meter.  This also introduces a single point of failure.  If that matters, which I'm sure it does, you'll want two or more of these. Finally, could just stick an agent provided by LogRhythm on the boxes.  But that's advice for another forum. ... View more

You might be getting blocked by AWS security groups.  Try creating a security group allowing outbound to that address and port.  Don't forget to assign it to the instance where it is running. Once that is in place, try making an outbound connection from the instance running docker (but not inside docker itself). openssl s_client -connect smtp.gmail.com:587 If successful, you will see a connection and a certificate sent back.  Then, at least, you'll know you can connect from the instance itself. ... View more