All,  Running into some trouble here with Syslog > SC4S > Splunk Edge. I think I have tuned every knob there is to be tuned but I am finding events are still being dropped.  SC4S writing to a file, does NOT drop events. But sending to the Edge HEC interface did. Sending Syslog direct to Edge , no dropped events. Not seeing queues anywhere back up. Plenty of resources. Not really sure where to start.  I followed the SC4S troubleshooting guide and we checked all the boxes there.  Any secrets docs or troubleshooting steps I should be looking into?    Below is an AI summary of what has been done so far:    ``` SC4S → Edge Processor over HEC: a tiny, stubborn ~0.008% loss we could never tune to zero (and why we gave up on the design) **Short version:** We ran SC4S (axosyslog) forwarding to a co-located Splunk Edge Processor via HEC on `http://localhost:8088`. Under UDP syslog load it dropped a small, maddening fraction of events — around **0.008%–0.2%** — that we could never drive to a repeatable zero, no matter what we tuned. We eventually proved the loss is inherent to pushing UDP (no backpressure) through syslog-ng's `http()` destination, and abandoned the SC4S→EP-over-HEC design. Posting the full knob list, the measurement method, and the lab-vs-prod result in case it saves someone else the two nights we spent. --- ## The setup ```text syslog senders (UDP :514) │ ▼ SC4S (axosyslog 4.x, container3, podman, --network host) │ classifies vendor, stamps index + sourcetype │ forwards over HEC to a co-located Edge Processor ▼ Edge Processor (same box, HEC receiver on :8088, plaintext) │ ▼ S2S :9997 → indexer cluster ``` - Both SC4S and EP on one 8-core box, ~72% idle at our volume. Not resource-starved. - The HEC hop is over **loopback** — SC4S `http()` → EP HEC on `localhost:8088`. - We must support **UDP** ingest (device standard). This turns out to be central. --- ## How we measured loss (this is the part that matters) We stopped trusting counters early — they lie in both directions (indexing lag makes good delivery look lossy; all-time counters hide per-run drops). The only number we believed: 1. **Sequenced load.** A generator sends exactly **N = 100,000** synthetic events at a fixed rate. Every event carries a unique run tag plus a zero-padded sequence number `seq00000000 … seq00099999`. 2. **Full settle.** Wait 90 seconds so all indexer peers have received and indexed everything. (Skipping this is how you manufacture fake loss — we did it a couple of times early and chased ghosts.) 3. **Count distinct sequence numbers that landed:** ```spl index=* "seq-<runtag>" | rex "seq(?<n>\d{8})" | stats dc(n) as distinct count as raw ``` `100000 - distinct` = true loss. `raw - distinct` = duplicates (mattered — see below). 4. **Forensics.** Bin the sequence space into 100 buckets of 1,000 and count missing per bucket, to see *where* loss happens (burst vs steady, contiguous vs scattered). > Gotcha: `... | stats count by n` silently caps at 50,000 rows, which makes everything above seq 50000 look "missing." Use binned `dc(n)`, never per-seq `stats` over 50k events. --- ## The single most useful test: does SC4S even have the events? SC4S has a built-in archive/file destination (`SC4S_ARCHIVE_GLOBAL=yes`) that writes every received event to disk in parallel with HEC — no HTTP, no flow-control window. We turned it on and ran the sequenced 100k: ```text sent: 100,000 archive file (disk): 100,000 distinct seq ← zero missing, zero dups HEC (into indexes): 99,905 distinct seq ← ~95 missing ``` **SC4S receives, parses, and retains 100% of events.** The loss is **entirely** the SC4S `http()` → EP HEC delivery hop. That one test ruled out UDP-receive loss (already zeroed with kernel tuning), SC4S internals, and classification. --- ## Every knob we turned, and what each did Baseline loss on the sequenced 100k: **~0.008%–0.2%** (noisy, never a clean repeatable zero). Measured by distinct-seq-landed. | Knob | Change | Result | |---|---|---| | `net.core.rmem_max` (kernel UDP recv buffer) | 208 KB → 512 MB (+ SO_RCVBUFF, IW) | **Fixed the UDP *receive* loss → 0** (was ~2.3%, 49.5M rcvbuf errors). Separate problem, worth doing. | | `net.core.wmem_max` (kernel TCP send buffer) | 208 KB → 512 MB (+ netdev_max_backlog, tcp_wmem) | Helped a bit — produced our only true 0.000% run — but not repeatably (next run 0.14%). Kept it; costs only memory. | | SC4S HEC `workers()` | 1 → 20 → 40 | workers=1 was ~11× worse (0.89%). workers=40 best (~0.008%). But the workers sit ~idle (0.4% CPU, queued=0) — not the bottleneck. | | `batch-lines` | 100 → 5000 | **Worse** (0.61%). Bigger batch = a bigger chunk exposed at the flow-control window when a flush stalls during a UDP burst. | | `batch-lines` | 100 → 50 | **Worse** (0.20%). SC4S's small-batch default is the shallow minimum — both directions hurt. | | Reliable disk-buffer (mem + disk) | enabled, 256 MB / 10 GB | No effect on this loss. `queued=0, memory_usage=0` when it drops — the buffer sits behind the queue; the drop is *before* the queue. | | HEC URL | `localhost` → external NIC IP | No change (~0.04%). Rules out a loopback-stack artifact. | | `useACK` (HEC indexer acknowledgement) | considered | Dead end — syslog-ng's `http()` dest doesn't do HEC ACK, and SC4S docs warn against it. | | `response-action(400 => drop)` → `retry` + `retries(10)` | tried | **Worse** (0.9% → 2.2%). Retrying rejected batches consumed the workers and starved *fresh* traffic. The `400 => drop` default exists for a reason. Reverted. | The two kernel-buffer changes are the only ones we kept. Everything on the SC4S send side was neutral or harmful. --- ## The fingerprint that told us it wasn't tunable Sequence-gap forensics on a lossy run: ~60% of a 100k run was **literally zero loss**, and all the drops fell into a handful of **2–5 second windows** — contiguous bursts, not a uniform statistical rate. SC4S and EP logs were both **silent** in those windows (no error, no reconnect). `dst.http dropped` incremented while `queued=0` and `memory_usage=0`. That is the signature of a **flow-control-window discard at the log path *before* the destination queue** — a sub-logging-threshold drain stall (a GC pause / scheduling hiccup / momentary EP HEC pause) that the flow-control window can't fully absorb in that instant. Because the source is UDP, syslog-ng **can't** apply backpressure — there's no sender to slow down — so it discards at the window instead of blocking. Buffers, workers, CPU, and retries all live *downstream* of where the drop happens, which is why none of them reach zero. --- ## Lab and prod: same result We built a clean lab clone (fresh podman SC4S, idle 16-core box, no real traffic, no other tenants) and ran the identical sequenced load: it dropped **0.15%** — same order of magnitude as prod. So the residual is **not** a prod-contention or noisy-neighbor artifact. It reproduces on an idle lab box. It's inherent to the transport, not the environment. --- ## Conclusion — why we abandoned the SC4S→EP-over-HEC design After the two kernel fixes and ~nine send-side levers, the honest conclusion is: **you cannot tune the SC4S `http()` → EP HEC hop to a consistent 0% under UDP burst.** The residual is the UDP-has-no-backpressure property surfacing as a flow-control-window discard. Every lever either did nothing, made it worse, or was a band-aid over a statistical edge race. What actually gets to zero (neither is a knob): - **Archive/disk-buffer as a safety net** — proven to hold 100% on disk. It makes "we never *lose* an event" true even if HEC sheds, but it's a replay net, not a fix for the hop. - **Get off HEC entirely** — end-to-end **TCP** (backpressure exists → the sender waits instead of anything dropping). In our case we ended up dropping SC4S and sending syslog **directly to the Edge Processor's own syslog receiver**, which sidesteps the SC4S `http()` destination completely. --- ## Questions for anyone who's run this 1. Has anyone driven the SC4S (axosyslog `http()`) → Edge Processor HEC hop to a *repeatable* 0% under sustained UDP burst? If so, what did it take? 2. Is there an `http()` setting that makes the destination **block/park into the reliable disk-buffer** instead of discarding at the flow-control window when the receiver stalls — without the retry-storm side effect we hit? 3. Anyone tuned the **Edge Processor HEC receiver** side (accept concurrency / receive window / max in-flight) rather than the SC4S send side? The EP docs we found only mention the sink send-queue. 4. Is SC4S → EP over **syslog/S2S** (bypassing HEC) a supported pattern, and if so how do you preserve SC4S's index/sourcetype routing on that transport? Happy to share the sequenced-load generator and the exact SPL. Measuring by distinct sequence IDs (and checking SC4S's own on-disk archive as ground truth) is what finally made this honest — and what convinced us the design wasn't tunable, only replaceable. Thanks in advance.   ``` ... View more

All,  Our SentinelOne EDR started detecting Alert Manager Enterprise's vsw.exe as Malware https://www.virustotal.com/gui/file/1cb09276e415c198137a87ba17fd05d0425d0c6f1f8c5afef81bac4fede84f6a/detection. Anyone else run into this before I start digging into this? Is there a proper course of action Splunkbase would like if this ends up being positive?  thanks -Daniel ... View more

All,  I am currently working with Splunk Add-on for Microsoft Office 365 4.5.1 on Linux. All inputs enabled and collecting. I am trying to see who approved a Privileged Identity Management event. I can't find the relevant events in Splunk but I do find them in Entra ID and Microsoft Purview dashboards?  1. Is there a TA I am missing?  2. If indeed this TA is not correctly scripting this data in, do I open a support case? Or is there another custom way to his that endpoint and snag that data.  thanks, -Daniel    ... View more

Having trouble with my roles/groups mapping with SAML. Setting up Azure AD+SAML on a test host here and my claim for group is coming back like so "d5366c24-8188-xxxx-xxxx-65e599a64ed9" rather than the human readable "SplunkSSO" group name which I expect. Funny enough this works [roleMap_SAML] power = d5366c24-8188-xxxx-xxxx-65e599a64ed9 But I kinda expecting to have human readable groups to roles. I assume there is an error in Attributes and Claims in the Splunk Azure App. Not seeing it though. Any ideas where I might look? ... View more

All,  Just noticed when Splunk UF installs it creates a user "splunk" with a login shell /bin/bash in /etc/passwd.  e.g. splunk:1007:/bin/bash Is that needed? Can I switch it to a nologin? Anyone familiar with the impact of doing that?  ... View more

All,  I am seeing the following error on my Splunk server in the logs and I have not been able to figure out what it means?  "11-17-2020 17:40:55.204 -0800 WARN SearchAssistant - recurseSyntax: Stanza entry not found for data-type 'time-specifier'" Any idea what it's referring to? What config?  ... View more

All,  Thought I posted this before, but can't find it in my history.  I am seeing alerts in my Splunk logs statin that the I am getting data from the future on my sourcetype  script:installedapps. It's default and unmodified from the Splunk_TA_Window standard. From there I did notice that _indextime and _time were off a bit.  When I look at props.conf provided by Splunk_TA_Windows it has no time stamp recognition. Is there a reason for this? Should I go ahead and add it or is there a trick for this I am missing?    thanks -Daniel  ... View more

All,  I have about 200 machines with UF installed. I want to monitor bash_history and a few other Linux /home items. The challenge is on about half the machines the home directory is an NFS mount and the other half are local file system.  Monitoring the NFS every every end point is IO prohibitive and double indexes the same data.  Is there a way in Splunk to programmatically handle this? That is, I only need to gather the files/logs from one host if it's mounted from NFS but if it's local file system I need to run the input on each machine.  Any recommendations? I was thinking of writing a script input in a Splunk app that creates and manages an app in the UF app folder. But seems very clunky.  ... View more

All,  I have an index (index=config) where all I store are the sourcetype=config_file. I currently use the stock config from Splunk_TA_nix.  What I am thinking is happening is that when we provision new server the config_files are often dated in years old so Splunk is creating new bucked dated years old. Is that reasonable assumption?  Looking at btool I am thinking all I need to do is set MAX_DAYS_AGO = 1 from 2000.  Here is the error message I am seeing The percentage of small buckets (100%) created over the last hour is high and exceeded the red thresholds (50%) for index=configs, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=4, small buckets=4 Any other input or best practices around indexing config_files?  ... View more

All,    Just trying to get a swag of cost by sourcetype. I wrote this search, but seems to me there is a more cost effective way to do this.    index=* sourcetype=* NOT sourcetype=stash NOT index=stash | fields _raw | eval bytes=len(_raw)/1024/1024/1024 | eval splunk_cost = 685 | eval gigcost = bytes * splunk_cost | stats sum(gigcost) as CostPerYear by sourcetype | sort - CostPerYear | addcoltotals ... View more

All,  I am in a transition state moving from one instance of Splunk to another. The old instance needs to stay up for a while, but I'd like to start shipping a certain subset of data (one sourcetype) to the new stack as well.    Is there a way to get a universal forwarder to send all data to two separate indexers?  ... View more

All, Setting up a Splunk instance and in the past I used a load balancer that handled certs for me. But this instance is going to terminate directly on the Splunk instance. Getting letsencrypt working on Splunk web was trivial about 20 minutes of time. But for the life of me I can't get it working with the HTTP Event Collector (HEC). Any walk through or tutorials on cert management on HEC, ideally with letsencrypt. Thought I could just point at the same files as Splunk web but doesn't seem to work that way. thanks -Daniel ... View more

All, I enabled in powershell input in Splunk_TA_nix for windows update logs on Win2016 and all I get it this. 1600/12/31 16:00:00.0000000 1128 7576 Unknown( 28): GUID=a09e99c9-a6c4-3261-6004-3aafb80214f7 (No Format Information found). Any idea what's going on here? ... View more

All, I am breaking my index=windows up into index=oswin and index=oswinsec. Any tricks or tools to search for searches, reports, dashboard, macros, tags, eventtypes, data model etc that might be tied to that index? ... View more

All, Trying to make a basic scripted python lookup. The examples and tutorials were just way over my head. So trying to do something simpler. I coped the example file and tried to simplify the problem a little. What I am aiming to do here is pass a field called 'mystring' and get back a field called 'myoutput'. I am passing hello as the field value for mystring and expecting world as the value in the new field myoutput. When I execute this I get the following index=* | head 1 | eval mystring = "hello" | lookup mylookup mystring I get" Script execution failed for external search command '/opt/splunk/etc/apps/TA-myapp/bin/mylookup.py'." here is my python. #!/usr/bin/env python import csv import sys def main(): if len(sys.argv) != 3: print("Usage: python mylookup.py [mystring] [myoutput]") sys.exit(1) # always passing hello as a eval mystring = "hello" mystring = sys.argv[1] myoutput = sys.argv[2] infile = sys.stdin outfile = sys.stdout r = csv.DictReader(infile) header = r.fieldnames w = csv.DictWriter(outfile, fieldnames=r.fieldnames) w.writeheader() for result in r: result[mystring] = "hello" result[myoutput] = "world" w.writerow(result) main() ... View more

All, I am attempting to read in a pfSense, /tmp/config.cache. Which carries the active running config. I can see some structure to it. Looking to get this loaded into Splunk. Anyone familiar with this file format? It has some sort of structure, but Splunk isn't detecting and I can't say I can detect it either. a:27:{s:7:"version";s:4:"19.1";s:10:"lastchange";s:0:"";s:6:"system";a:23:{s:12:"optimization";s:6:"normal";s:8:"hostname";s:7:"pfSense";s:6:"domain";s:11:"localdomain";s:9:"dnsserver";a:2:{i:0;s:7:"8.8.8.8";i:1;s:7:"4.2.2.2";}s:16:"dnsallowoverride";s:2:"on";s:5:"group";a:2:{i:0;a:5:{s:4:"name";s:3:"all";s:11:"description";s:9:"All Users";s:5:"scope";s:6:"system";s:3:"gid";s:4:"1998";s:6:"member";a:1:{i:0;s:1:"0";}}i:1;a:6:{s:4:"name";s:6:"admins";s:11:"description";s:21:"System Administrators";s:5:"scope";s:6:"system";s:3:"gid";s:4:"1999";s:6:"member";a:1:{i:0;s:1:"0";}s:4:"priv";a:1:{i:0;s:8:"page-all";}}}s:4:"user";a:1:{i:0;a:7:{s:4:"name";s:5:"admin";s:5:"descr";s:20:"System Administrator";s:5:"scope";s:6:"system";s:9:"groupname";s:6:"admins";s:11:"bcrypt-hash";s:60:"$2y$10$QDCfvt17W67gtAjpEfPgzO0rwz78bkHrEi5BIsDvnMKi3mNNZ7ysq";s:3:"uid";s:1:"0";s:4:"priv";a:1:{i:0;s:17:"user-shell-access";}}}s:7:"nextuid";s:4:"2000";s:7:"nextgid";s:4:"2000";s:11:"timeservers";s:22:"0.pfsense.pool.ntp.org";s:6:"webgui";a:5:{s:8:"protocol";s:5:"https";s:17:"loginautocomplete";s:0:"";s:11:"ssl-certref";s:13:"5e79fb1489ce6";s:16:"dashboardcolumns";s:1:"2";s:12:"althostnames";s:0:"";}s:20:"disablenatreflection";s:3:"yes";s:29:"disablesegmentationoffloading";s:0:"";s:29:"disablelargereceiveoffloading";s:0:"";s:9:"ipv6allow";s:0:"";s:19:"maximumtableentries";s:6:"400000";s:14:"powerd_ac_mode";s:4:"hadp";s:19:"powerd_battery_mode";s:4:"hadp";s:18:"powerd_normal_mode";s:4:"hadp";s:6:"bogons";a:1:{s:8:"interval";s:7:"monthly";}s:26:"already_run_config_upgrade";s:0:"";s:3:"ssh";a:1:{s:6:"enable";s:7:"enabled";}s:8:"timezone";s:7:"Etc/UTC";}s:10:"interfaces";a:1:{s:3:"wan";a:10:{s:6:"enable";s:0:"";s:2:"if";s:3:"em0";s:6:"ipaddr";s:4:"dhcp";s:8:"ipaddrv6";s:5:"dhcp6";s:7:"gateway";s:0:"";s:11:"blockbogons";s:2:"on";s:5:"media";s:0:"";s:8:"mediaopt";s:0:"";s:10:"dhcp6-duid";s:0:"";s:15:"dhcp6-ia-pd-len";s:1:"0";}}s:12:"staticroutes";s:0:"";s:5:"dhcpd";s:0:"";s:7:"dhcpdv6";s:0:"";s:5:"snmpd";a:3:{s:11:"syslocation";s:0:"";s:10:"syscontact";s:0:"";s:11:"rocommunity";s:6:"public";}s:4:"diag";a:1:{s:7:"ipv6nat";a:1:{s:6:"ipaddr";s:0:"";}}s:6:"syslog";a:9:{s:18:"filterdescriptions";s:1:"1";s:8:"nentries";s:2:"50";s:12:"remoteserver";s:17:"192.168.1.16:9514";s:13:"remoteserver2";s:0:"";s:13:"remoteserver3";s:0:"";s:8:"sourceip";s:0:"";s:7:"ipproto";s:4:"ipv4";s:6:"logall";s:0:"";s:6:"enable";s:0:"";}s:6:"filter";a:1:{s:4:"rule";a:3:{i:0;a:7:{s:4:"type";s:4:"pass";s:10:"ipprotocol";s:4:"inet";s:5:"descr";s:29:"Default allow LAN to any rule";s:9:"interface";s:3:"lan";s:7:"tracker";s:10:"0100000101";s:6:"source";a:1:{s:7:"network";s:3:"lan";}s:11:"destination";a:1:{s:3:"any";s:0:"";}}i:1;a:7:{s:4:"type";s:4:"pass";s:10:"ipprotocol";s:5:"inet6";s:5:"descr";s:34:"Default allow LAN IPv6 to any rule";s:9:"interface";s:3:"lan";s:7:"tracker";s:10:"0100000102";s:6:"source";a:1:{s:7:"network";s:3:"lan";}s:11:"destination";a:1:{s:3:"any";s:0:"";}}i:2;a:8:{s:6:"source";a:1:{s:3:"any";s:0:"";}s:9:"interface";s:3:"wan";s:8:"protocol";s:3:"tcp";s:11:"destination";a:2:{s:7:"address";s:7:"4.3.2.1";s:4:"port";s:9:"1512-1712";}s:5:"descr";s:10:"NAT wefewf";s:18:"associated-rule-id";s:27:"nat_5e7a6639ad2df8.55902217";s:7:"tracker";s:10:"1585079865";s:7:"created";a:2:{s:4:"time";s:10:"1585079865";s:8:"username";s:16:"NAT Port Forward";}}}}s:5:"ipsec";s:0:"";s:7:"aliases";s:0:"";s:8:"proxyarp";s:0:"";s:4:"cron";a:1:{s:4:"item";a:6:{i:0;a:7:{s:6:"minute";s:4:"1,31";s:4:"hour";s:3:"0-5";s:4:"mday";s:1:"*";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:31:"/usr/bin/nice -n20 adjkerntz -a";}i:1;a:7:{s:6:"minute";s:1:"1";s:4:"hour";s:1:"3";s:4:"mday";s:1:"1";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:43:"/usr/bin/nice -n20 /etc/rc.update_bogons.sh";}i:2;a:7:{s:6:"minute";s:1:"1";s:4:"hour";s:1:"1";s:4:"mday";s:1:"*";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:40:"/usr/bin/nice -n20 /etc/rc.dyndns.update";}i:3;a:7:{s:6:"minute";s:4:"*/60";s:4:"hour";s:1:"*";s:4:"mday";s:1:"*";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:67:"/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot";}i:4;a:7:{s:6:"minute";s:2:"30";s:4:"hour";s:2:"12";s:4:"mday";s:1:"*";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:43:"/usr/bin/nice -n20 /etc/rc.update_urltables";}i:5;a:7:{s:6:"minute";s:1:"1";s:4:"hour";s:1:"0";s:4:"mday";s:1:"*";s:5:"month";s:1:"*";s:4:"wday";s:1:"*";s:3:"who";s:4:"root";s:7:"command";s:46:"/usr/bin/nice -n20 /etc/rc.update_pkg_metadata";}}}s:3:"wol";s:0:"";s:3:"rrd";a:1:{s:6:"enable";s:0:"";}s:13:"load_balancer";a:1:{s:12:"monitor_type";a:5:{i:0;a:4:{s:4:"name";s:4:"ICMP";s:4:"type";s:4:"icmp";s:5:"descr";s:4:"ICMP";s:7:"options";s:0:"";}i:1;a:4:{s:4:"name";s:3:"TCP";s:4:"type";s:3:"tcp";s:5:"descr";s:11:"Generic TCP";s:7:"options";s:0:"";}i:2;a:4:{s:4:"name";s:4:"HTTP";s:4:"type";s:4:"http";s:5:"descr";s:12:"Generic HTTP";s:7:"options";a:3:{s:4:"path";s:1:"/";s:4:"host";s:0:"";s:4:"code";s:3:"200";}}i:3;a:4:{s:4:"name";s:5:"HTTPS";s:4:"type";s:5:"https";s:5:"descr";s:13:"Generic HTTPS";s:7:"options";a:3:{s:4:"path";s:1:"/";s:4:"host";s:0:"";s:4:"code";s:3:"200";}}i:4;a:4:{s:4:"name";s:4:"SMTP";s:4:"type";s:4:"send";s:5:"descr";s:12:"Generic SMTP";s:7:"options";a:2:{s:4:"send";s:0:"";s:6:"expect";s:5:"220 *";}}}}s:7:"widgets";a:2:{s:8:"sequence";s:88:"system_information:col1:show,netgate_services_and_support:col2:show,interfaces:col2:show";s:6:"period";s:2:"10";}s:7:"openvpn";s:0:"";s:8:"dnshaper";s:0:"";s:7:"unbound";a:8:{s:6:"enable";s:0:"";s:6:"dnssec";s:0:"";s:16:"active_interface";s:0:"";s:18:"outgoing_interface";s:0:"";s:14:"custom_options";s:0:"";s:12:"hideidentity";s:0:"";s:11:"hideversion";s:0:"";s:14:"dnssecstripped";s:0:"";}s:8:"revision";a:3:{s:4:"time";s:10:"1585081388";s:11:"description";s:100:"[email protected] (Local Database): Firewall: NAT: Port Forward - saved/edited a port forward rule.";s:8:"username";s:35:"[email protected] (Local Database)";}s:6:"shaper";s:0:"";s:4:"cert";a:1:{i:0;a:5:{s:5:"refid";s:13:"5e79fb1489ce6";s:5:"descr";s:39:"webConfigurator default (5e79fb1489ce6)";s:4:"type";s:6:"server";s:3:"crt";s:2152:"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVlakNDQTJLZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREJhTVRnd05nWURWUVFLRXk5d1psTmwKYm5ObElIZGxZa052Ym1acFozVnlZWFJ2Y2lCVFpXeG1MVk5wWjI1bFpDQkRaWEowYVdacFkyRjBaVEVlTUJ3RwpBMVVFQXhNVmNHWlRaVzV6WlMwMVpUYzVabUl4TkRnNVkyVTJNQjRYRFRJd01ETXlOREV5TWpBek5sb1hEVEkxCk1Ea3hOREV5TWpBek5sb3dXakU0TURZR0ExVUVDaE12Y0daVFpXNXpaU0IzWldKRGIyNW1hV2QxY21GMGIzSWcKVTJWc1ppMVRhV2R1WldRZ1EyVnlkR2xtYVdOaGRHVXhIakFjQmdOVkJBTVRGWEJtVTJWdWMyVXROV1UzT1daaQpNVFE0T1dObE5qQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUs4M21sc0FUNjhyCk9XRThhUnYydFhudjBVZkh3Q3JLb1oxYjBkd25RSXRJS0RiUmZLK0FtcDlqck1RQUd6TWllME5uLzZMZVNNODEKVzdXRlVtYUxFTnFTSGpuK1h0ajkxbnJVekF6cno4ZVAxR3RvcHEzNzdPdUxzSUlBcXdQaURxQTh5K3dOVkFiSQpzY2dTYXA0WkYxTXRGKzhxOUROMllFTkVVaUdxdzRTU0o1L3U2dkp2blo3MXh2L1FaVFowTkl3TGs5STB4bWhpCnpYOWxwSDVkWG5aVjlLR083eG10azhVdXVyeGx6ZGxRRmdLU214QU9KQzZzTUdXRSt3UGlHZzg4QzdhYmtNMlkKN3R2TnVwa2VzaXMwOThaL1pNa0F1ZTM0RVhzV3hkS3RJRnRKS3o0YVBIdEdpSjRWVCtLS2RlUVk0UjRkQjlLSwplWGhmeDN4YTdDa0NBd0VBQWFPQ0FVa3dnZ0ZGTUFrR0ExVWRFd1FDTUFBd0VRWUpZSVpJQVliNFFnRUJCQVFECkFnWkFNQXNHQTFVZER3UUVBd0lGb0RBekJnbGdoa2dCaHZoQ0FRMEVKaFlrVDNCbGJsTlRUQ0JIWlc1bGNtRjAKWldRZ1UyVnlkbVZ5SUVObGNuUnBabWxqWVhSbE1CMEdBMVVkRGdRV0JCUWdoaGY2dGsyVmI1SzRDcFdtK1JmUQpFRzFXbVRDQmdnWURWUjBqQkhzd2VZQVVJSVlYK3JaTmxXK1N1QXFWcHZrWDBCQnRWcG1oWHFSY01Gb3hPREEyCkJnTlZCQW9UTDNCbVUyVnVjMlVnZDJWaVEyOXVabWxuZFhKaGRHOXlJRk5sYkdZdFUybG5ibVZrSUVObGNuUnAKWm1sallYUmxNUjR3SEFZRFZRUURFeFZ3WmxObGJuTmxMVFZsTnpsbVlqRTBPRGxqWlRhQ0FRQXdIUVlEVlIwbApCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQ0FJQ01DQUdBMVVkRVFRWk1CZUNGWEJtVTJWdWMyVXROV1UzCk9XWmlNVFE0T1dObE5qQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFRYU1iY2xZS3pDRDlKOFJuTnAzMmF1MDkKckxtdnFFckVrSXhUTWkyWG9mZFdUV29KMzVZR2k0aGVKK1k3MEhNN3pOQ054d0lOVHVDY2loSGgzMmNhTXphQgpkczBpVFpoS21JRmdrUkxNMjB6YVFhUzNTTVFDUVJlQWNsUWRSeUg2V3B5WlBYTmNzV292TkRwUGRIbGFSc1djCnVtenlKLzBBWmV2V3IybHRXSktGSmZKUU8wa1l4NnRFcHRnUnlSdWh2cVpPYXBuVDMzaEJhTDdDYnRYazhWczQKZXZyYStpQUYvMkhnSkE5Sy9STGt6YkFjQ2xFMEV1M1Y5Tk9nUDNpOWxHRTRQcWlQSGpZcDJrcDhzZHVHMEoyMQpML2xRcDhFMC9yYlQzK2FXMGRhTEZ2QjNYT2JYREMwTjVyU0JMbmlubllDODdWZWF5aXc4UmUyaG9pOVhwQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K";s:3:"prv";s:2280:"LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2d0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktrd2dnU2xBZ0VBQW9JQkFRQ3ZONXBiQUUrdkt6bGgKUEdrYjlyVjU3OUZIeDhBcXlxR2RXOUhjSjBDTFNDZzIwWHl2Z0pxZlk2ekVBQnN6SW50RFovK2kza2pQTlZ1MQpoVkptaXhEYWtoNDUvbDdZL2RaNjFNd002OC9IajlScmFLYXQrK3pyaTdDQ0FLc0Q0ZzZnUE12c0RWUUd5TEhJCkVtcWVHUmRUTFJmdkt2UXpkbUJEUkZJaHFzT0VraWVmN3VyeWI1MmU5Y2IvMEdVMmREU01DNVBTTk1ab1lzMS8KWmFSK1hWNTJWZlNoanU4WnJaUEZMcnE4WmMzWlVCWUNrcHNRRGlRdXJEQmxoUHNENGhvUFBBdTJtNURObU83Ygp6YnFaSHJJck5QZkdmMlRKQUxudCtCRjdGc1hTclNCYlNTcytHang3Um9pZUZVL2lpblhrR09FZUhRZlNpbmw0Clg4ZDhXdXdwQWdNQkFBRUNnZ0VBQzFDRDN5eDkrTW5Kd3NXcjQrcGlmYVZHMW1QSHZQdW94QWlSM0syTU5YSkwKWm43UWxtU3ZsMnRRVkxmTkNkaElMV29oejlxYXlRYWhEVysyaW5pZ2RmekpodVV1S3NUNWZLVVJLQ1J5SG1qagpScXhUVnhqVmk4QlJmWk9kZDNxNWh3OWwrN0JBcE0rQTYzS0ZBQUNPeVFnNGEzRlNvNkFaUno2Nkx3SmY3Y2VHCjdxNjZJMEpnN3ZhVlJFMU8xMm5nN05xemtEUThoMEhjYnhlZW5LZlRabDVtQlZZbUtGZFRmNmcya2VNeExXZXMKd1Jua2lIQmJNZDFiK3VRdUlpL0t1cVZWc2c2YUcyZ1d1MTZER2RCbjZ5dzlCUmhuU3ZpcmlRREJDRW41MkQ4NQo0VUV4bHhCamJIUEJFRVFDa1c5UFpmMm9GV3U5b1BSL0JQVUFNdEs0SVFLQmdRRGgyN251Q0VpekFXMUpNc3Q1ClpvRVNOSUpxZ0pBMnhsRSttKzlLa3dnYkRYaW55cVVBUEMzZG0xOERtR2taQkxlc21wZGQ2Y2FhamhRSEFIQkoKM3VLaTJQaXA3cU95d0dKK1B4WWVublU5a3RndGZSaFB1czJFRHh4dHV3SG1rYkt4aTl6Zy9EKzBQZ3B2ZXhQOQpQNUdubGhXQXo0Tk9BUTFmUUxzY3hjdWdnd0tCZ1FER21jTTIzY2ZQNnYzVXUrMEVzTzgrQUxGTmNDU0VTcE1IClZIVUp6OWZyN1ZwZmZLWjBoWk5TRFFqdE5zRC9VZUhXdWxUb3plaDNsbVgya1J2amtuMzFYdWpkdEZuYWl1YnkKR01WQWZ3d1NuOXFaMEJkenM3VCttZUZTM3Q4R3pVRDFVSEpvdXJlSjRkSkVjb3dLTTQzT2Q4ODZYOHBySXlVSApRWkFIbzRLSTR3S0JnUURJanlsbjZndEVpYnZXQ0RrUE1LcmswNlFMbHVaNC9Wb2YwckNHOUZGNlZGZ1VCNnJGCnJxcTc0c0JZblBxV3NNMjVnLzF0ODYzY2lOWFg4ZGZFZ1J1WHFEd0lDbFZxNGRPVWI4amduNjFVWkJWN0wxNXIKVG1JNUpvSUVIcy90UXV2L0pVZWFzZVNQMVpmR3J2QnRMZ25WV3p6MUNWQjc4QXREem1OWmhYcndxUUtCZ1FDMwo0cDAvQ3dDOGdoKyt2clpKOXEyK0loUUkySUhuUDhsOUt2VW5QWnYyWmhHY2dpVDVsTWlBVzNOZGVLb2dmYWQzCkU1WVU3THFISittRzhIcjdMcU9UOHVuNGhjb0FzVVgrK1hLQ01tQnlTakswNGxra2wwdEp4aDg4aFFIS0lYZzQKNitEVEdiZGhZb2MzT3p4eElhVDJmRGFUSFNpbUpLZGZYWlJIamwwSjh3S0JnUUM1TDhFLzZUNnZhSkVPLzROYwo2dVJBMm1RYjg4cE9DYXFpUTBWdUtYYWQ0QUxCTlFtdGREcVZFVGVObWszNU55SUJvUG5UUHNpY0FmY003b3kyCjRTMUQ2alc2aUl0bnluUHJLbVBXRkZVYWNsWXF1a0hINTRFWFhOSGdQbUtJbGwraWRHbmxieWhGR05MVzFSVWQKZXBDUW8waXdWWlhxelZON2VZK2pKZXZ2TVE9PQotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg==";}}s:4:"ppps";s:0:"";s:3:"nat";a:2:{s:9:"separator";s:0:"";s:4:"rule";a:1:{i:0;a:10:{s:6:"source";a:1:{s:3:"any";s:0:"";}s:11:"destination";a:2:{s:7:"network";s:5:"wanip";s:4:"port";s:6:"22-222";}s:8:"protocol";s:3:"tcp";s:6:"target";s:7:"4.3.2.1";s:10:"local-port";s:4:"1512";s:9:"interface";s:3:"wan";s:5:"descr";s:6:"wefewf";s:18:"associated-rule-id";s:27:"nat_5e7a6639ad2df8.55902217";s:7:"created";a:2:{s:4:"time";s:10:"1585079865";s:8:"username";s:35:"[email protected] (Local Database)";}s:7:"updated";a:2:{s:4:"time";s:10:"1585081388";s:8:"username";s:35:"[email protected] (Local Database)";}}}}} ... View more

All, Any service you recommend for doing domain classification and lookups against my Squid proxy logs? Just general classification and of course alerts for malware. ... View more

All, The default hostname should be fine for my use cases with /var/log/messages brought in with the pretrained sourcetype of linux_messages_syslog. How ever there is a host overwrite in the default install of Splunk. Is there a formal way to disable this? This stanza is in /opt/splunk/etc/system/default. [syslog-host] DEST_KEY = MetaData:Host REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s FORMAT = host::$1 I was just going to create a local transforms.conf that uses a different variable, [syslog-host] DEST_KEY = MetaData:Extracted_Host REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s FORMAT = host::$1 but figure I can't be the first person to run into this. So probably a better way to do it. ... View more

Morning everyone, Came in to work today and seeing this error. Anyone familiar with it? What's the impact and fix? Stock install of Splunk with no custom certs. 03-16-2020 03:01:55.285 -0700 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py" HTTPSConnectionPool(host='e1345286.api.splkmobile.com', port=443): Max retries exceeded with url: /1.0/e1345286/81416994-c2ef-5c6f-a3de-68fb09953b0d/100/0?hash=none (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:742)'),)) ... View more

All, Member of our management team is concerned about a Splunk Forwarder with a number of processes and threads. Curious what's normal ? What might create more threads? Less? Most my servers have anywhere from 40-49. # ps auwxH|grep splunk|wc -l 43 ps auwwxH | grep -i splunk Tue Mar 3 11:24:51 PST 2020 root 4980 0.0 0.2 362396 166344 ? Sl Feb27 0:24 splunkd -p 8089 restart root 4980 0.0 0.2 362396 166344 ? Sl Feb27 0:00 splunkd -p 8089 restart [...........] root 4980 0.0 0.2 362396 166344 ? Sl Feb27 0:12 splunkd -p 8089 restart root 4980 0.0 0.2 362396 166344 ? Sl Feb27 4:59 splunkd -p 8089 restart root 4980 0.0 0.2 362396 166344 ? Sl Feb27 0:00 splunkd -p 8089 restart root 4986 0.0 0.0 86200 1056 ? Ss Feb27 0:00 [splunkd pid=4980] splunkd -p 8089 restart [process-runner] root 7619 0.0 0.0 103328 916 pts/32 S+ 11:24 0:00 grep -i splunk My knee jerk is that maybe we have we have thread per file, another one reaching back to deployment server and maybe another per scripted input from Splunk_TA_nix? ... View more

All, We're reselecting our endpoint protection for Windows Servers and Workstation. I'd like to start with solutions that speak splunk or have good Splunk apps. That is to say, good logs, CIM, tags, great apps that can go into Splunk relatively easy. Obviously Splunk isn't the only measure of success here, but it's one element we're looking at. Any recommendations? ... View more