Hi @livehybrid, api_lt and api_et should correspond to the UI time range or the earliest_time and latest_time search API paramters as you noted, although I don't know if this is publicly documented. Similarly, api_index_et and api_index_lt should correspond to the index_earliest and index_latest search API parameters. search_lt and search_et should correspond to the computed epoch second values from the earliest, latest, and other time modifiers if they're provided as part of the base search: index=main foo earliest=-24h@h latest=now index=main foo starttime=06/29/2025:20:50:00 The audit log doesn't appear to capture the values passed to _index_earliest and _index_latest or translate them to api_index_et and api_index_lt, unfortunately, but they should be present in the search text.   ... View more

Hi @mristic, While no specific guidance is available for Splunk Universal Forwarder, Splunk did publish RHEL 7/8-compatible SELinux policies as recently as Splunk Enterprise 9.2.2. You may be able to adapt them to your needs. See https://docs.splunk.com/Documentation/Splunk/9.2.2/CommonCriteria/InstallSELinux. https://download.splunk.com/products/security/splunk-selinux-0-0.9.0.el7.noarch.tgz https://download.splunk.com/products/security/splunk-selinux-0-0.9.0.el8.noarch.tgz ... View more

Hi @kn450, For a basic setup with either a standalone Splunk/Stream instance or separate Splunk and Stream instances, the steps at https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/UseStreamtoingestNetflowandIPFIXdata result in a working configuration. In my test environment using a standalone instance on RHEL, I made only the following changes to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local/streamfwd.conf to enable both capture and NetFlow/IPFIX: [streamfwd] streamfwdcapture.0.interfaceRegex = ens.+ netflowReceiver.0.port = 9996 netflowReceiver.0.decoder = netflow I then enabled the netflow metadata stream in the Splunk Stream app. Using SolarWinds NetFlow Generator <https://www.solarwinds.com/free-tools/flow-tool-bundle> (not an endorsement, but it's free), I sent sample IPFIX data to the standalone instance, which Stream successfully decoded: {"endtime":"2025-06-29T23:20:12Z","timestamp":"2025-06-29T23:20:12Z","bytes_in":0,"dest_ip":"192.168.1.25","dest_port":443,"dest_sysnum":0,"event_name":"netFlowData","exporter_ip":"192.168.1.158","exporter_time":"2025-Jun-29 23:20:12","flow_end_rel":0,"flow_start_rel":0,"input_snmpidx":8,"netflow_version":10,"nexthop_addr":"1.1.1.2","observation_domain_id":0,"output_snmpidx":5,"packets_in":0,"protoid":6,"seqnumber":23000,"src_ip":"192.168.1.132","src_port":15449,"src_sysnum":0,"tcp_flags":0,"tos":0} Custom NetFlow parsing is described at https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/AutoinputNetflow. Can you confirm the default configuration works? If it does, we can dig into any customizations you need. If it doesn't, confirm your Stream instance is receiving correctly formatted IPFIX packets using tcpdump or another local capture tool. ... View more

Hi @chrisboy68, There are lots of options presented, but combining @yuanliu's response with a conversion from bill_date to year and month gives the output closest to "ID Cost by month": | makeresults format=csv data="bill_date,ID,Cost,_time 6/1/25,1,1.24,2025-06-16T12:42:41.282-04:00 6/1/25,1,1.4,2025-06-16T12:00:41.282-04:00 5/1/25,1,2.5,2025-06-15T12:42:41.282-04:00 5/1/25,1,2.2,2025-06-14T12:00:41.282-04:00 5/1/25,2,3.2,2025-06-14T12:42:41.282-04:00 5/1/25,2,3.3,2025-06-14T12:00:41.282-04:00 3/1/25,1,4.4,2025-06-13T12:42:41.282-04:00 3/1/25,1,5,2025-06-13T12:00:41.282-04:00 3/1/25,2,6,2025-06-13T12:42:41.282-04:00 3/1/25,2,6.3,2025-06-13T12:00:41.282-04:00" | eval _time=strptime(_time, "%FT%T.%N%z") ``` end test data ``` ``` assuming month/day/year for bill_date ``` | eval Month=strftime(strptime(bill_date, "%m/%e/%y"), "%Y-%m") | stats latest(Cost) as Cost by Month ID Month ID Cost ----- -- ---- 2025-03 1 4.4 2025-03 2 6 2025-05 1 2.5 2025-05 2 3.2 2025-06 1 1.24 You can alternatively use chart, xyseries, etc. to pivot the results: | chart latest(Cost) over ID by Month ID 2025-03 2025-05 2025-06 -- ------- ------- ------- 1 4.4 2.5 1.24 2 6 3.2 ... View more

Hi @Namo, Make sure $SPLUNK_HOME/etc/auth/cacert.pem contains all certificates in the trust chain. If you're using a self-signed certificate, add this certificate to cacert.pem. If you've changed the name or location of the file, update the new file. If you're also attempting a KV store upgrade, check the prerequisites at https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version#ariaid-title2 as others have recommended. Also note that your private key must be encrypted with the correct sslPassword value in server.conf for a KV store upgrade to succeed. When using a blank/empty password, you'll see a message similar to the following in splunkd.log: 06-21-2025 00:00:00.000 -0000 WARN KVStoreUpgradeToolTLS [133719 KVStoreConfigurationThread] - Incomplete TLS settings detected, skipping creation of KVStore TLS credentials file!   ... View more

Hi @kn450, With respect to your prior comments: "... it's important to note that the queries used are not written in Splunk’s native SPL language; instead, they rely on Elasticsearch queries. This limits the integration with some of Splunk’s core functionalities and does not provide the desired level of efficiency in terms of performance and deep analysis." I use custom generating commands to run Elasticsearch searches, and I treat the results as if they came from a similar base SPL command. I agree the ideal would be a virtual index or federated search that compiles a search command into equivalent Elasticsearch Query DSL, for example, but that isn't presently feasible. What Splunk functionality would you like to use with custom search commands, including those from apps on Splunkbase, that you cannot use? Do you have specific use cases in mind? ... View more

Hi @luminousplumz, For index-time field extractions, you want something like this (note the order of the transforms in the TRANSFORMS-mqtt setting): # fields.conf [sourcetype::mqtttojson_ubnpfc_all::Topic] INDEXED = true # props.conf [mqtttojson_ubnpfc_all] TRANSFORMS-mqtt = mqtttopic,mqtttojson # transforms.conf [mqtttojson] CLEAN_KEYS = 0 DEST_KEY = _raw FORMAT = $1 REGEX = msg=(.+) [mqtttopic] CLEAN_KEYS = 0 FORMAT = Topic::$1 REGEX = topic=(?:[^/]*/){3}([^/]+) WRITE_META = true  For search-time field extractions, you want something like this: [mqtttojson_ubnpfc_all] EXTRACT-Topic = topic=(?:[^/]*/){3}(?<Topic>[^/]+) EVAL-_raw = replace(_raw, ".*? msg=", "")  However, in the search-time configuration, you'll need to extract the JSON fields in a search as automatic key-value field extraction happens before calculated fields (EVAL-*): sourcetype=mqtttojson_ubnpfc_all | spath You'll note that the original name, event_id, topic, and msg (value possibly truncated) fields are automatically extracted before the full value of msg is assigned to _raw. ... View more

Hi @molla, The geo_countries lookup shipped with Splunk provides boundaries for countries. The tutorial at https://docs.splunk.com/Documentation/Splunk/latest/Viz/GenerateMap provides an example for counties, but you can replace the county references with country references: | makeresults format=csv data="x,country 3,United States 5,United States 4,Canada 1,Canada 1,Mexico 2,Mexico" | stats sum(x) by country | geom geo_countries featureIdField=country The output of geom can be used with choropleth maps in both classic (Simple XML) dashboards and Dashboard Studio. You can use the inputlookup command to see the list of supported countries: | inputlookup geo_countries | table featureId ... View more

If the Good, Resetting, etc. fields are counts, @shrija may have been looking for this: | fields lat lon Good Resetting Starting Unknown Faulty | eval Count=0 | foreach Good Resetting Starting Unknown Faulty [| eval Count=Count+coalesce('<<FIELD>>', 0) ] | geostats globallimit=0 latfield=lat longfield=lon sum(Good) as Good sum(Resetting) as Resetting sum(Starting) as Starting sum(Unknown) as Unknown sum(Faulty) as Faulty sum(Count) as Count However, the cluster map visualization generates a pie chart with one half of the pie representing the total count and the other half of the pie representing the individual sums: ... View more

... and the forum injected an unintended emoji. I really wish it wouldn't do that. 🙂 ... View more

Hi @Vignesh, The alerts/suppressions endpoint is hard-coded to use 'nobody' as the owner, which the internal saved/eventtypes/_new endpoint interprets as the current user context. You can change the owner and sharing scope of the event type after it's created using the saved/eventtypes/{name}/acl endpoint (see https://docs.splunk.com/Documentation/Splunk/latest/RESTUM/RESTusing#Access_Control_List😞 curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/nobody/SA-ThreatIntelligence/saved/eventtypes/notable_suppression-foo/acl \ --data-urlencode owner=jsmith \ --data-urlencode sharing=global You can create the event type directly using the saved/eventtypes endpoint and an alternate owner; however, you'll need to call the saved/eventtypes/{name}/acl endpoint separately to change sharing from private to global. The owner argument is required by the endpoint, so it's effectively the same number of steps as creating the suppression using the alerts/suppressions endpoint: curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/jsmith/SA-ThreatIntelligence/saved/eventtypes \ --data-urlencode name=notable_suppression-foo \ --data-urlencode description=bar \ --data-urlencode search='`get_notable_index` _time>1737349200 _time<1737522000' \ --data-urlencode disabled=false curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/jsmith/SA-ThreatIntelligence/saved/eventtypes/notable_suppression-foo/acl \ --data-urlencode owner=jsmith \ --data-urlencode sharing=global     ... View more

Hi @shrija, You can create choropleth (shaded outline) maps in both Classic (Simple XML) and Dashboard Studio map visualizations. In Simple XML, you can also create categorical choropleth maps and pie chart bubbles. In Dashboard Studio, you can create pie charts bubbles and categorical markers. Neither supports color bars. To map events to geographic boundaries, you can use the bundled United States geo lookups or you can upload custom KML files. Combined with a custom tile server, the KML files can represent anything with features and coordinates: topographical maps, nautical charts, office layouts, theme parks, rail/subway systems, etc. Are you working with a specific geographic region? ... View more

Hi @jonxilinx, The aws:cloudwatch:guardduty source type was intended to be used with a CloudWatch Logs input after a transform from the aws:cloudwatchlogs source type. To use an SQS input, you can transform the data on your heavy forwarder. The configuration below works on the following event schema: { "BodyJson": { "version": "0", "id": "cd2d702e-ab31-411b-9344-793ce56b1bc7", "detail-type": "GuardDuty Finding", "source": "aws.guardduty", "account": "111122223333", "time": "1970-01-01T00:00:00Z", "region": "us-east-1", "resources": [], "detail": { ... } } } You may need to adjust the configuration to match your specific input and event format. # local/inputs.conf [my_sqs_input] aws_account = xxx aws_region = xxx sqs_queues = xxx index = xxx sourcetype = aws:sqs interval = xxx # local/props.conf [aws:sqs] TRANSFORMS-aws_sqs_guardduty = aws_sqs_guardduty_remove_bodyjson, aws_sqs_guardduty_to_cloudwatchlogs_sourcetype # local/transforms.conf [aws_sqs_guardduty_remove_bodyjson] REGEX = "source"\s*\:\s*"aws\.guardduty" INGEST_EVAL = _raw:=json_extract(_raw, "BodyJson") [aws_sqs_guardduty_to_cloudwatchlogs_sourcetype] REGEX = "source"\s*\:\s*"aws\.guardduty" DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::aws:cloudwatchlogs:guardduty   ... View more

Hi @rfdickerson, The Python source code for Splunk's implementation of StateSpaceForecast is collectively in: $SPLUNK_HOMEetc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py $SPLUNK_HOMEetc/apps/Splunk_ML_Toolkit/bin/algos_support/statespace/* The StateSpaceForecast algorithm is similar to the Splunk predict command. If you're not managing your own Splunk instance, you can download the MLTK archive from Splunkbase and inspect the files directly. The holdback and forecast_k parameters function as described. You may want to look at the partial_fit parameter for more control over the window of data used to update your model dynamically before using apply and (eventually) calculating TPR and FPR. ... View more

Hi @Rakzskull, Splunk support can assist with migrations from DDAA (Splunk-provided S3) to DDSS (customer-provided S3). ... View more

I included this: | search PROJECTNAME="*" INVOCATIONID="*" RUNMAJORSTATUS="*" RUNMINORSTATUS="*" as a placeholder for filtering using Simple XML inputs. The most likely cause of the difference in the number of results is one of the fields above not being present after spath extracts fields. In your second search, the events missing from the first search would have Status=="Unknown". Have you compared the results at the event level to look for differences other than simple truncation? ... View more

Hi @arunssd, If 1) your KV store collection uses array fields, 2) all field values have a 1:1:1:1 relationship, and 3) there are no empty/missing/null values within a field, i.e. all array values "line up": asn country maliciousbehavior riskscore 103.152.101.251 => PK => 3 => 9 103.96.75.159 => HK => 3 => 11 104.234.115.155 => CA => 4 => 9 you can transform the data with the transpose, mvexpand, and chart commands: | inputlookup arunssd_kv | transpose 0 | mvexpand "row 1" | chart values("row 1") over _mkv_child by column | fields - _mkv_child | outputlookup arunssd_lookup.csv However, your results may be truncated by mvexpand if the total size of the in-memory result is greater than the limits.conf max_mem_usage_mb setting (default: 500 MB). See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bmvexpand.5D. If this doesn't work for you, please share your collections.conf (KV store) and transforms.conf (lookup) settings. I used the following settings to test: # collections.conf [arunssd_kv] field.asn = array field.country = array field.maliciousbehavior = array field.riskscore = array # transforms.conf [arunssd_kv] collection = arunssd_kv external_type = kvstore fields_list = asn,country,maliciousbehavior,riskscore If your KV store fields are strings, the search can be adapted with the foreach and eval commands to coerce the fields values into a multi-valued type. You can also transform the results from a shell using curl and jq or your scripting tools of choice. ... View more

You could leave it that way, but you're maintaining 200 connections to the downstream receivers. If you have, for example, 16 cores on your intermediate forwarder and want to leave 2 cores free for other activity (so much overhead!), you can do the same thing with larger queues and fewer pipelines by increasing maxSize values by the same relative factor. If your forwarder doesn't have enough memory to hold all queues, keep an eye on memory, paging, and disk queue metrics. ... View more

Hi @anissabnk, Can you describe what's limited? @PickleRick showed a value length example. The spath command is limited to the first 5,000 bytes of the event by default. What is your maximum event length from | stats max(eval(len(_raw))) as max_len? If you meant the number of results, and the xyseries command returns no more than 50,000 results, you may be hitting a limit in an early search command, although I don't see a limited command in your original example. ... View more

I'm also assuming that you've already set maxKBps = 0 in limits.conf: # $SPLUNK_HOME/etc/system/local/limits.conf [thruput] maxKBps = 0   ... View more

Hi @MichaelM1, Increasing parallelIngestionPipelines to a value larger than 1 is similar to running multiple instances of splunkd with splunktcp inputs on different ports. As a starting point, however, I would leave parallelIngestionPipelines unset or at the default value of 1. splunkd uses a series of queues in a pipeline to process events. Of note: parsingQueue aggQueue typingQueue rulesetQueue indexQueue There are other queues, but these are the most well-documented. See https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774/highlight/true#M103484. I have copies of the printer and high-DPI display friendly PDFs if you need them. On a typical universal forwarder acting as an intermediate forwarder, parsingQueue, which performs minimal event parsing, and indexQueue, which sends events to outputs, are the likely bottlenecks. Your metrics.log event provides a hint: <date time> Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=1217, largest_size=1217,smallest_size=0 Note that metrics.log logs queue names in lower case, but queue names are case-sensitive in configuration files. parsingQueue is blocked because 1217KB is greater than 512KB. The inputs.conf splunktcp stopAcceptorAfterQBlock setting controls what happens to the listener port when a queue is blocked, but you don't need to modify this setting. In your case, I would start by leaving parallelIngestionPipelines at the default value of 1 as noted above and increasing indexQueue to the next highest factor of 128 bytes larger than twice the largest_size value observed for parsingQueue. In %SPLUNK_HOME\etc\systeml\local\server.conf on the intermediate forwarder: [queue=indexQueue] # 2 * 1217KB <= 20 * 128B = 2560KB maxSize = 2560KB (x86-64, ARM64, and SPARC architectures have 64 byte cache lines, but on the off chance you encounter AIX on PowerPC with 128 byte caches lines, for example, you'll avoid buffer alignment performance penalties, closed-source splunkd memory allocation overhead notwithstanding.) Observe metrics.log following the change and keep increasing maxSize until you no longer see instances of blocked=true. If you run out of memory, add more memory to your intermediate forwarder host or consider scaling your intermediate forwarders horizontally with additional hosts. As an alternative, you can start by increasing maxSize for parsingQueue and only increase maxSize for indexQueue if you see blocked=true messages in metrics.log: [queue=parsingQueue] maxSize = 2560KB You can usually find the optimal values through trail and error without resorting to a queue-theoretic analysis. If you find that your system becomes CPU-bound at some maxSize limit, you can increase parallelIngestionPipelines, for example, to N-2, where N is the number of cores available. Following that change, modify maxSize from default values by observing metrics.log. Note that each pipeline consumes as much memory as a single-pipeline splunkd process with the same memory settings. ... View more

Hi @MichaelM1, Does your test script fail at ~1000 connections when sending a handshake directly to the intermediate forwarder input port and not your server script port? Completing a handshake and sending no data while holding the connection open should work. The splunktcp input will not reset the connection for at least (by default) 10 minutes (see the inputs.conf splunktcp s2sHeartbeatTimeout setting). It still seems as though there may be a limit at the firewall specific to your splunktcp port, but the firewall would be logging corresponding drops or resets. The connection(s) from the intermediate forwarder to the downstream receiver(s) shouldn't directly impact new connections from forwarders to the intermediate forwarder, although blocked queues may prevent new connections or close existing ones. Have you checked metrics.log on the intermediate forwarder for blocked=true events? A large number of streams moving through a single pipeline on an intermediate forwarder will likely require increasing queue sizes or adding pipelines. ... View more

Hi @LinkLoop, You can verify Splunk is connected to outputs with the list forward-server command: & "C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" list forward-server -auth admin:password Active forwards: splunk.example.com:9997 (ssl) Configured but inactive forwards: None The command requires authentication, so you'll need to know the local Splunk admin username and password defined at install time. If the local management port is disabled, the command will not be available. You can otherwise search local logs for forwarding activity: Select-String -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" -Pattern "Connected to idx=" Select-String -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log" -Pattern "group=per_index_thruput, series=`"_internal`""   ... View more