Hi @kn450, With respect to your prior comments: "... it's important to note that the queries used are not written in Splunk’s native SPL language; instead, they rely on Elasticsearch queries. This limits the integration with some of Splunk’s core functionalities and does not provide the desired level of efficiency in terms of performance and deep analysis." I use custom generating commands to run Elasticsearch searches, and I treat the results as if they came from a similar base SPL command. I agree the ideal would be a virtual index or federated search that compiles a search command into equivalent Elasticsearch Query DSL, for example, but that isn't presently feasible. What Splunk functionality would you like to use with custom search commands, including those from apps on Splunkbase, that you cannot use? Do you have specific use cases in mind? ... View more

Hi @luminousplumz, For index-time field extractions, you want something like this (note the order of the transforms in the TRANSFORMS-mqtt setting): # fields.conf [sourcetype::mqtttojson_ubnpfc_all::Topic] INDEXED = true # props.conf [mqtttojson_ubnpfc_all] TRANSFORMS-mqtt = mqtttopic,mqtttojson # transforms.conf [mqtttojson] CLEAN_KEYS = 0 DEST_KEY = _raw FORMAT = $1 REGEX = msg=(.+) [mqtttopic] CLEAN_KEYS = 0 FORMAT = Topic::$1 REGEX = topic=(?:[^/]*/){3}([^/]+) WRITE_META = true  For search-time field extractions, you want something like this: [mqtttojson_ubnpfc_all] EXTRACT-Topic = topic=(?:[^/]*/){3}(?<Topic>[^/]+) EVAL-_raw = replace(_raw, ".*? msg=", "")  However, in the search-time configuration, you'll need to extract the JSON fields in a search as automatic key-value field extraction happens before calculated fields (EVAL-*): sourcetype=mqtttojson_ubnpfc_all | spath You'll note that the original name, event_id, topic, and msg (value possibly truncated) fields are automatically extracted before the full value of msg is assigned to _raw. ... View more

Hi @molla, The geo_countries lookup shipped with Splunk provides boundaries for countries. The tutorial at https://docs.splunk.com/Documentation/Splunk/latest/Viz/GenerateMap provides an example for counties, but you can replace the county references with country references: | makeresults format=csv data="x,country 3,United States 5,United States 4,Canada 1,Canada 1,Mexico 2,Mexico" | stats sum(x) by country | geom geo_countries featureIdField=country The output of geom can be used with choropleth maps in both classic (Simple XML) dashboards and Dashboard Studio. You can use the inputlookup command to see the list of supported countries: | inputlookup geo_countries | table featureId ... View more

If the Good, Resetting, etc. fields are counts, @shrija may have been looking for this: | fields lat lon Good Resetting Starting Unknown Faulty | eval Count=0 | foreach Good Resetting Starting Unknown Faulty [| eval Count=Count+coalesce('<<FIELD>>', 0) ] | geostats globallimit=0 latfield=lat longfield=lon sum(Good) as Good sum(Resetting) as Resetting sum(Starting) as Starting sum(Unknown) as Unknown sum(Faulty) as Faulty sum(Count) as Count However, the cluster map visualization generates a pie chart with one half of the pie representing the total count and the other half of the pie representing the individual sums: ... View more

... and the forum injected an unintended emoji. I really wish it wouldn't do that. 🙂 ... View more

Hi @Vignesh, The alerts/suppressions endpoint is hard-coded to use 'nobody' as the owner, which the internal saved/eventtypes/_new endpoint interprets as the current user context. You can change the owner and sharing scope of the event type after it's created using the saved/eventtypes/{name}/acl endpoint (see https://docs.splunk.com/Documentation/Splunk/latest/RESTUM/RESTusing#Access_Control_List😞 curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/nobody/SA-ThreatIntelligence/saved/eventtypes/notable_suppression-foo/acl \ --data-urlencode owner=jsmith \ --data-urlencode sharing=global You can create the event type directly using the saved/eventtypes endpoint and an alternate owner; however, you'll need to call the saved/eventtypes/{name}/acl endpoint separately to change sharing from private to global. The owner argument is required by the endpoint, so it's effectively the same number of steps as creating the suppression using the alerts/suppressions endpoint: curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/jsmith/SA-ThreatIntelligence/saved/eventtypes \ --data-urlencode name=notable_suppression-foo \ --data-urlencode description=bar \ --data-urlencode search='`get_notable_index` _time>1737349200 _time<1737522000' \ --data-urlencode disabled=false curl -k -u admin:pass -X POST https://splunk:8089/servicesNS/jsmith/SA-ThreatIntelligence/saved/eventtypes/notable_suppression-foo/acl \ --data-urlencode owner=jsmith \ --data-urlencode sharing=global     ... View more

Hi @shrija, You can create choropleth (shaded outline) maps in both Classic (Simple XML) and Dashboard Studio map visualizations. In Simple XML, you can also create categorical choropleth maps and pie chart bubbles. In Dashboard Studio, you can create pie charts bubbles and categorical markers. Neither supports color bars. To map events to geographic boundaries, you can use the bundled United States geo lookups or you can upload custom KML files. Combined with a custom tile server, the KML files can represent anything with features and coordinates: topographical maps, nautical charts, office layouts, theme parks, rail/subway systems, etc. Are you working with a specific geographic region? ... View more

Hi @jonxilinx, The aws:cloudwatch:guardduty source type was intended to be used with a CloudWatch Logs input after a transform from the aws:cloudwatchlogs source type. To use an SQS input, you can transform the data on your heavy forwarder. The configuration below works on the following event schema: { "BodyJson": { "version": "0", "id": "cd2d702e-ab31-411b-9344-793ce56b1bc7", "detail-type": "GuardDuty Finding", "source": "aws.guardduty", "account": "111122223333", "time": "1970-01-01T00:00:00Z", "region": "us-east-1", "resources": [], "detail": { ... } } } You may need to adjust the configuration to match your specific input and event format. # local/inputs.conf [my_sqs_input] aws_account = xxx aws_region = xxx sqs_queues = xxx index = xxx sourcetype = aws:sqs interval = xxx # local/props.conf [aws:sqs] TRANSFORMS-aws_sqs_guardduty = aws_sqs_guardduty_remove_bodyjson, aws_sqs_guardduty_to_cloudwatchlogs_sourcetype # local/transforms.conf [aws_sqs_guardduty_remove_bodyjson] REGEX = "source"\s*\:\s*"aws\.guardduty" INGEST_EVAL = _raw:=json_extract(_raw, "BodyJson") [aws_sqs_guardduty_to_cloudwatchlogs_sourcetype] REGEX = "source"\s*\:\s*"aws\.guardduty" DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::aws:cloudwatchlogs:guardduty   ... View more

Hi @rfdickerson, The Python source code for Splunk's implementation of StateSpaceForecast is collectively in: $SPLUNK_HOMEetc/apps/Splunk_ML_Toolkit/bin/algos/StateSpaceForecast.py $SPLUNK_HOMEetc/apps/Splunk_ML_Toolkit/bin/algos_support/statespace/* The StateSpaceForecast algorithm is similar to the Splunk predict command. If you're not managing your own Splunk instance, you can download the MLTK archive from Splunkbase and inspect the files directly. The holdback and forecast_k parameters function as described. You may want to look at the partial_fit parameter for more control over the window of data used to update your model dynamically before using apply and (eventually) calculating TPR and FPR. ... View more

Hi @Rakzskull, Splunk support can assist with migrations from DDAA (Splunk-provided S3) to DDSS (customer-provided S3). ... View more

Welcome, AppDynamics practitioners! You'll find Splunkers here, of course, but many of us have experience with AppDynamics, too! ... View more

I included this: | search PROJECTNAME="*" INVOCATIONID="*" RUNMAJORSTATUS="*" RUNMINORSTATUS="*" as a placeholder for filtering using Simple XML inputs. The most likely cause of the difference in the number of results is one of the fields above not being present after spath extracts fields. In your second search, the events missing from the first search would have Status=="Unknown". Have you compared the results at the event level to look for differences other than simple truncation? ... View more

Hi @arunssd, If 1) your KV store collection uses array fields, 2) all field values have a 1:1:1:1 relationship, and 3) there are no empty/missing/null values within a field, i.e. all array values "line up": asn country maliciousbehavior riskscore 103.152.101.251 => PK => 3 => 9 103.96.75.159 => HK => 3 => 11 104.234.115.155 => CA => 4 => 9 you can transform the data with the transpose, mvexpand, and chart commands: | inputlookup arunssd_kv | transpose 0 | mvexpand "row 1" | chart values("row 1") over _mkv_child by column | fields - _mkv_child | outputlookup arunssd_lookup.csv However, your results may be truncated by mvexpand if the total size of the in-memory result is greater than the limits.conf max_mem_usage_mb setting (default: 500 MB). See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bmvexpand.5D. If this doesn't work for you, please share your collections.conf (KV store) and transforms.conf (lookup) settings. I used the following settings to test: # collections.conf [arunssd_kv] field.asn = array field.country = array field.maliciousbehavior = array field.riskscore = array # transforms.conf [arunssd_kv] collection = arunssd_kv external_type = kvstore fields_list = asn,country,maliciousbehavior,riskscore If your KV store fields are strings, the search can be adapted with the foreach and eval commands to coerce the fields values into a multi-valued type. You can also transform the results from a shell using curl and jq or your scripting tools of choice. ... View more

You could leave it that way, but you're maintaining 200 connections to the downstream receivers. If you have, for example, 16 cores on your intermediate forwarder and want to leave 2 cores free for other activity (so much overhead!), you can do the same thing with larger queues and fewer pipelines by increasing maxSize values by the same relative factor. If your forwarder doesn't have enough memory to hold all queues, keep an eye on memory, paging, and disk queue metrics. ... View more

Hi @anissabnk, Can you describe what's limited? @PickleRick showed a value length example. The spath command is limited to the first 5,000 bytes of the event by default. What is your maximum event length from | stats max(eval(len(_raw))) as max_len? If you meant the number of results, and the xyseries command returns no more than 50,000 results, you may be hitting a limit in an early search command, although I don't see a limited command in your original example. ... View more

I'm also assuming that you've already set maxKBps = 0 in limits.conf: # $SPLUNK_HOME/etc/system/local/limits.conf [thruput] maxKBps = 0   ... View more

Hi @MichaelM1, Increasing parallelIngestionPipelines to a value larger than 1 is similar to running multiple instances of splunkd with splunktcp inputs on different ports. As a starting point, however, I would leave parallelIngestionPipelines unset or at the default value of 1. splunkd uses a series of queues in a pipeline to process events. Of note: parsingQueue aggQueue typingQueue rulesetQueue indexQueue There are other queues, but these are the most well-documented. See https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774/highlight/true#M103484. I have copies of the printer and high-DPI display friendly PDFs if you need them. On a typical universal forwarder acting as an intermediate forwarder, parsingQueue, which performs minimal event parsing, and indexQueue, which sends events to outputs, are the likely bottlenecks. Your metrics.log event provides a hint: <date time> Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=1217, largest_size=1217,smallest_size=0 Note that metrics.log logs queue names in lower case, but queue names are case-sensitive in configuration files. parsingQueue is blocked because 1217KB is greater than 512KB. The inputs.conf splunktcp stopAcceptorAfterQBlock setting controls what happens to the listener port when a queue is blocked, but you don't need to modify this setting. In your case, I would start by leaving parallelIngestionPipelines at the default value of 1 as noted above and increasing indexQueue to the next highest factor of 128 bytes larger than twice the largest_size value observed for parsingQueue. In %SPLUNK_HOME\etc\systeml\local\server.conf on the intermediate forwarder: [queue=indexQueue] # 2 * 1217KB <= 20 * 128B = 2560KB maxSize = 2560KB (x86-64, ARM64, and SPARC architectures have 64 byte cache lines, but on the off chance you encounter AIX on PowerPC with 128 byte caches lines, for example, you'll avoid buffer alignment performance penalties, closed-source splunkd memory allocation overhead notwithstanding.) Observe metrics.log following the change and keep increasing maxSize until you no longer see instances of blocked=true. If you run out of memory, add more memory to your intermediate forwarder host or consider scaling your intermediate forwarders horizontally with additional hosts. As an alternative, you can start by increasing maxSize for parsingQueue and only increase maxSize for indexQueue if you see blocked=true messages in metrics.log: [queue=parsingQueue] maxSize = 2560KB You can usually find the optimal values through trail and error without resorting to a queue-theoretic analysis. If you find that your system becomes CPU-bound at some maxSize limit, you can increase parallelIngestionPipelines, for example, to N-2, where N is the number of cores available. Following that change, modify maxSize from default values by observing metrics.log. Note that each pipeline consumes as much memory as a single-pipeline splunkd process with the same memory settings. ... View more

Hi @MichaelM1, Does your test script fail at ~1000 connections when sending a handshake directly to the intermediate forwarder input port and not your server script port? Completing a handshake and sending no data while holding the connection open should work. The splunktcp input will not reset the connection for at least (by default) 10 minutes (see the inputs.conf splunktcp s2sHeartbeatTimeout setting). It still seems as though there may be a limit at the firewall specific to your splunktcp port, but the firewall would be logging corresponding drops or resets. The connection(s) from the intermediate forwarder to the downstream receiver(s) shouldn't directly impact new connections from forwarders to the intermediate forwarder, although blocked queues may prevent new connections or close existing ones. Have you checked metrics.log on the intermediate forwarder for blocked=true events? A large number of streams moving through a single pipeline on an intermediate forwarder will likely require increasing queue sizes or adding pipelines. ... View more

Hi @LinkLoop, You can verify Splunk is connected to outputs with the list forward-server command: & "C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" list forward-server -auth admin:password Active forwards: splunk.example.com:9997 (ssl) Configured but inactive forwards: None The command requires authentication, so you'll need to know the local Splunk admin username and password defined at install time. If the local management port is disabled, the command will not be available. You can otherwise search local logs for forwarding activity: Select-String -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" -Pattern "Connected to idx=" Select-String -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log" -Pattern "group=per_index_thruput, series=`"_internal`""   ... View more

Hi @antoniolamonica, Data model root search datasets start with a base search. Endpoint.Processes is: (`cim_Endpoint_indexes`) tag=process tag=report | eval process_integrity_level=lower(process_integrity_level) This search is expanded by Splunk to include the contents of the cim_EndPoint_indexes macros and all event types that match tag=process and tag=report. To compare like for like searches, start with the same base search: (`cim_Endpoint_indexes`) tag=process tag=report earliest=-4h@h latest=-2h@h | eval process_integrity_level=lower(process_integrity_level) | stats count values(process_id) as process_id by dest and construct a similar tstats search: | tstats summariesonly=f count values(Processes.process_id) as process_id from datamodel=Endpoint.Processes where earliest=-4h@h latest=-2h@h by Processes.dest The underlying searches should be similar. Optimization may vary. You can verify the SPL in the job inspector: Job > Inspect Job > search.log and the UnifiedSearch component's log output. When summariesonly=f, the searches have similar behavior. When summariesonly=t, the data model search only looks at indexed field values. This is similar to using field::value for indexed fields and TERM() for indexed terms in a normal search. ... View more

Hi @madhav_dholakia, Can you provide more context for the defaults object and a small sample dashboard that doesn't save correctly? Is defaults defined at the top level of the dashboard? I.e.: { "visualizations": { ... }, "dataSources": { ... }, "defaults": { ... }, ... }   ... View more

Hi @splunk_user_99, Which version of MLTK do you have installed? The underlying API uses a simple payload: {"name":"my_model","app":"search"} where name is the value entered in New Main Model Title and app is derived from the value selected in Destination App. The app list is loaded when Operationalize is clicked and sorted alphabetically by display name. On submit, the request payload is checked to verify that it contains only the 'app' and 'name' keys. Do you have the same issue in a sandboxed (private, incognito, etc.) browser session with extensions disabled? ... View more