Splunk Assist generating DNS queries to beam.scs.splunk.com every 15 seconds Environment: Splunk Enterprise 10.2.3 Component: Splunk Assist (splunk_assist) Observed domain: beam.scs.splunk.com I noticed that my Splunk server was generating DNS queries for beam.scs.splunk.com approximately every 15 seconds. I wanted to determine exactly which Splunk component was responsible and whether the frequency could be reduced without disabling Splunk Assist or restarting Splunk. Initial observation DNS telemetry showed at least one lookup for beam.scs.splunk.com in essentially every 15-second interval. Most queries were AAAA lookups, with occasional A lookups. The Assist configuration contains several references to Beam, including UI assets and supervisor updates: [ui] assets_url=https://beam.scs.splunk.com/com.splunk.assist.ui/latest/com.splunk.assist.ui-assets.tgz sig_url=https://beam.scs.splunk.com/com.splunk.assist.ui/latest/com.splunk.assist.ui-assets.tgz.asc [updates] global_url=https://beam.scs.splunk.com/supervisors/package.json The default Assist modular-input configuration was: [supervisor_modular_input://default] disabled = 0 interval = 15 [instance_id_modular_input://default] disabled = 0 interval = 15 [uiassets_modular_input://default] disabled = 0 interval = 15 [selfupdate_modular_input://default] disabled = 0 interval = 300 Identifying the process responsible Using bpftrace, I correlated execution of the Assist Python modular inputs with DNS connections. For example: sudo bpftrace -e ' tracepoint:syscalls:sys_enter_execve /str(args->filename) == "/opt/splunk/bin/python3.9"/ { printf("%s EXEC pid=%d script=%s\n", strftime("%H:%M:%S", nsecs), pid, str(args->argv[1])); } tracepoint:syscalls:sys_enter_connect { $sa = (struct sockaddr *)args->uservaddr; if ($sa->sa_family == 2) { $sin = (struct sockaddr_in *)args->uservaddr; $port = (($sin->sin_port & 0xff) << 😎 | (($sin->sin_port >> 😎 & 0xff); if ($port == 53) { printf("%s DNS pid=%d comm=%s dst=%s\n", strftime("%H:%M:%S", nsecs), pid, comm, ntop($sin->sin_addr.s_addr)); } } }' This showed that the DNS lookup was being made by the same PID running uiassets_modular_input.py. instance_id_modular_input.py was also executing every 15 seconds, but it was not the process making the observed DNS request. I separately confirmed that selfupdate_modular_input.py accesses: https://beam.scs.splunk.com/supervisors/package.json on its 300-second schedule. So there were two distinct Beam-related activities: uiassets_modular_input.py every 15 seconds selfupdate_modular_input.py every 300 seconds The continuous DNS traffic was caused by the UI-assets modular input. Reducing the UI-assets polling interval Rather than modify the files under default, I created: /opt/splunk/etc/apps/splunk_assist/local/inputs.conf with: [uiassets_modular_input://default] interval = 300 btool confirmed the override: sudo /opt/splunk/bin/splunk btool inputs list \ uiassets_modular_input://default \ --debug Result: /opt/splunk/etc/apps/splunk_assist/local/inputs.conf [uiassets_modular_input://default] /opt/splunk/etc/apps/splunk_assist/default/inputs.conf disabled = 0 host = $decideOnStartup index = default /opt/splunk/etc/apps/splunk_assist/local/inputs.conf interval = 300 /opt/splunk/etc/apps/splunk_assist/default/inputs.conf python.version = python3 The REST API also reported the new effective value: "interval":"300" However, uiassets_modular_input.py initially continued executing every 15 seconds. In other words, changing inputs.conf and seeing the new value in btool/REST did not by itself update the already-running modular-input schedule. Reloading without restarting Splunk The REST endpoint exposed a _reload action for the modular input, so I used: curl -ks -u <username> -X POST \ 'https://127.0.0.1:8089/servicesNS/nobody/splunk_assist/data/inputs/uiassets_modular_input/_reload' This was the key step. Immediately afterward, bpftrace no longer showed uiassets_modular_input.py starting every 15 seconds. instance_id_modular_input.py continued on its normal 15-second schedule. DNS telemetry confirmed the same change. Before the reload, Beam queries appeared every 15 seconds; afterward, that continuous pattern disappeared. Pasted markdown Subsequent Beam queries were much more widely spaced, consistent with the remaining periodic Assist activities. Pasted markdown Result The effective configuration is now: [uiassets_modular_input://default] interval = 300 This reduced the uiassets_modular_input.py execution interval from 15 seconds to 300 seconds, eliminating the continuous 15-second beam.scs.splunk.com DNS lookup pattern. Most importantly, this was accomplished without restarting Splunk: Add the local/inputs.conf override. Verify interval = 300 with btool. Verify the REST endpoint also reports interval=300. POST to the modular input's _reload endpoint. Verify the new execution/DNS behavior. One additional observation: beam.scs.splunk.com returned IPv4 addresses but no AAAA answer in my environment. The AAAA requests were therefore returning NOERROR/NODATA rather than NXDOMAIN. This was not a DNS failure; it simply made the frequent Assist lookups particularly visible in DNS telemetry. Bottom line: In Splunk Enterprise 10.2.3, the Splunk Assist uiassets_modular_input was responsible for the 15-second beam.scs.splunk.com DNS activity I observed. Overriding its interval to 300 seconds and explicitly reloading that modular input substantially reduced the DNS traffic without requiring a Splunk restart. ... View more

Greetings community experts Search results for JSON data received via curl and Rest API from AWS are five times the actual events. Seeking help understanding why events are counted more than once. Indexed using sourcetype _JSON Looking at the data Splunk reports 4 deviceConnectivityUpdate events and 1 deviceStateEvent which agrees with the data. However when I run stats count by("hits{}._source.logType") by "hits{}._source.userName I get 5x count of events.  Same is true with this search using dedup | eval DCU=mvfilter(match('hits{}._source.logType',"deviceConnectivityUpdate")) | eval DSE=mvfilter(match('hits{}._source.logType',"deviceStateEvent")) | dedup DCU DSE | stats count(DCU) count(DSE) by hits{}._source.userName The data _raw {"total":{"value":5,"relation":"eq"},"max_score":1,"hits":[{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"zbIdu4gBwP_vIV4KexH0","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390007","userName":"gary.whitlocks22","cloudTimestampUTC":"2023-06-14T18:14:11Z","isDeviceOffline":false}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"z7Ieu4gBwP_vIV4KARGG","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390007","userName":"gary.whitlocks22","cloudTimestampUTC":"2023-06-14T18:14:45Z","isDeviceOffline":true}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"0LIeu4gBwP_vIV4KHxHn","_score":1,"_source":{"version":1,"logType":"deviceStateEvent","deviceSerialNumber":"4931490086","userName":"NSSS","cloudTimestampUTC":"2023-06-14T18:14:53Z","deviceTimestampUTC":"2023-06-14T18:14:55Z","batteryPercent":49,"isCheckIn":false,"isAntiSurveillanceViolation":false,"isLowBatteryViolation":false,"isCellularViolation":false,"isDseDelayed":false,"bleMacAddress":"7d:8e:1a:be:92:5a","cellIpv4Address":"0.0.0.0","cellIpv6Address":"::"}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"zrIdu4gBwP_vIV4KsxFQ","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390006","userName":"PennyAndroid","cloudTimestampUTC":"2023-06-14T18:14:25Z","isDeviceOffline":true}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"0bIeu4gBwP_vIV4KhBGr","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390006","userName":"PennyAndroid","cloudTimestampUTC":"2023-06-14T18:15:19Z","isDeviceOffline":false}}]} JSON format { "total": { "value": 5, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "zbIdu4gBwP_vIV4KexH0", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390007", "userName": "gary.whitlocks22", "cloudTimestampUTC": "2023-06-14T18:14:11Z", "isDeviceOffline": false } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "z7Ieu4gBwP_vIV4KARGG", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390007", "userName": "gary.whitlocks22", "cloudTimestampUTC": "2023-06-14T18:14:45Z", "isDeviceOffline": true } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "0LIeu4gBwP_vIV4KHxHn", "_score": 1, "_source": { "version": 1, "logType": "deviceStateEvent", "deviceSerialNumber": "4931490086", "userName": "NSSS", "cloudTimestampUTC": "2023-06-14T18:14:53Z", "deviceTimestampUTC": "2023-06-14T18:14:55Z", "batteryPercent": 49, "isCheckIn": false, "isAntiSurveillanceViolation": false, "isLowBatteryViolation": false, "isCellularViolation": false, "isDseDelayed": false, "bleMacAddress": "7d:8e:1a:be:92:5a", "cellIpv4Address": "0.0.0.0", "cellIpv6Address": "::" } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "zrIdu4gBwP_vIV4KsxFQ", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390006", "userName": "PennyAndroid", "cloudTimestampUTC": "2023-06-14T18:14:25Z", "isDeviceOffline": true } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "0bIeu4gBwP_vIV4KhBGr", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390006", "userName": "PennyAndroid", "cloudTimestampUTC": "2023-06-14T18:15:19Z", "isDeviceOffline": false } } ] } ... View more

Greetings experts Big picture: using Bash script and curl to download Rest API/JSON  from an AWS instance. The beginning of each download is unstructured followed by the structured JSON. Four different "logType", two logTypes (deviceConnectivityUpdate and deviceStateEvent) shown in a five event example below _raw example {"total":{"value":5,"relation":"eq"},"max_score":1,"hits":[{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"zbIdu4gBwP_vIV4KexH0","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390007","userName":"gary.whitlocks22","cloudTimestampUTC":"2023-06-14T18:14:11Z","isDeviceOffline":false}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"z7Ieu4gBwP_vIV4KARGG","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390007","userName":"gary.whitlocks22","cloudTimestampUTC":"2023-06-14T18:14:45Z","isDeviceOffline":true}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"0LIeu4gBwP_vIV4KHxHn","_score":1,"_source":{"version":1,"logType":"deviceStateEvent","deviceSerialNumber":"4931490086","userName":"NSSS","cloudTimestampUTC":"2023-06-14T18:14:53Z","deviceTimestampUTC":"2023-06-14T18:14:55Z","batteryPercent":49,"isCheckIn":false,"isAntiSurveillanceViolation":false,"isLowBatteryViolation":false,"isCellularViolation":false,"isDseDelayed":false,"bleMacAddress":"7d:8e:1a:be:92:5a","cellIpv4Address":"0.0.0.0","cellIpv6Address":"::"}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"zrIdu4gBwP_vIV4KsxFQ","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390006","userName":"PennyAndroid","cloudTimestampUTC":"2023-06-14T18:14:25Z","isDeviceOffline":true}},{"_index":"index_44444444-4444-4444-4444-444444444444","_id":"0bIeu4gBwP_vIV4KhBGr","_score":1,"_source":{"version":1,"logType":"deviceConnectivityUpdate","deviceSerialNumber":"4931390006","userName":"PennyAndroid","cloudTimestampUTC":"2023-06-14T18:15:19Z","isDeviceOffline":false}}]} JSON { "total": { "value": 5, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "zbIdu4gBwP_vIV4KexH0", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390007", "userName": "gary.whitlocks22", "cloudTimestampUTC": "2023-06-14T18:14:11Z", "isDeviceOffline": false } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "z7Ieu4gBwP_vIV4KARGG", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390007", "userName": "gary.whitlocks22", "cloudTimestampUTC": "2023-06-14T18:14:45Z", "isDeviceOffline": true } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "0LIeu4gBwP_vIV4KHxHn", "_score": 1, "_source": { "version": 1, "logType": "deviceStateEvent", "deviceSerialNumber": "4931490086", "userName": "NSSS", "cloudTimestampUTC": "2023-06-14T18:14:53Z", "deviceTimestampUTC": "2023-06-14T18:14:55Z", "batteryPercent": 49, "isCheckIn": false, "isAntiSurveillanceViolation": false, "isLowBatteryViolation": false, "isCellularViolation": false, "isDseDelayed": false, "bleMacAddress": "7d:8e:1a:be:92:5a", "cellIpv4Address": "0.0.0.0", "cellIpv6Address": "::" } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "zrIdu4gBwP_vIV4KsxFQ", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390006", "userName": "PennyAndroid", "cloudTimestampUTC": "2023-06-14T18:14:25Z", "isDeviceOffline": true } }, { "_index": "index_44444444-4444-4444-4444-444444444444", "_id": "0bIeu4gBwP_vIV4KhBGr", "_score": 1, "_source": { "version": 1, "logType": "deviceConnectivityUpdate", "deviceSerialNumber": "4931390006", "userName": "PennyAndroid", "cloudTimestampUTC": "2023-06-14T18:15:19Z", "isDeviceOffline": false } } ] } CSV version logType deviceSerialNumber userName cloudTimestampUTC isDeviceOffline deviceTimestampUTC batteryPercent isCheckIn isAntiSurveillanceViolation isLowBatteryViolation isCellularViolation isDseDelayed bleMacAddress cellIpv4Address cellIpv6Address 1 deviceConnectivityUpdate 4931390007 gary.whitlocks22 6/14/2023 18:14 FALSE                     1 deviceConnectivityUpdate 4931390007 gary.whitlocks22 6/14/2023 18:14 TRUE                     1 deviceStateEvent 4931490086 NSSS 6/14/2023 18:14   6/14/2023 18:14 49 FALSE FALSE FALSE FALSE FALSE 7d:8e:1a:be:92:5a 0.0.0.0 :: 1 deviceConnectivityUpdate 4931390006 PennyAndroid 6/14/2023 18:14 TRUE                     1 deviceConnectivityUpdate 4931390006 PennyAndroid 6/14/2023 18:15 FALSE                         Indexed using sourcetype _JSON Looking at hits{}._source.logType Splunk reports 4 deviceConnectivityUpdate events and 1 deviceStateEvent which agrees with the data. However when I run stats count by("hits{}._source.logType") by "hits{}._source.userName I get 5x count of events (see attachment) Help please, what am I missing or doing wrong?    ... View more