I'm trying to create a calculated field (eval) that will coalesce a bunch of username fields, then perform match() and replace() functions within a case statement. Here's a scenario: Possible user fields: UserName, username, User_ID User values need domain removed (e.g., "user@domain.com" or "ad\user" needs to be "user"). Here is how it can be done in two evals (I newlined and indented each case for readability):   | eval user_coalesced = coalesce(UserName, username, User_ID) | eval user = case( match(user_coalesced, ".*@.*"), replace(user_coalesced, "@.*", ""), match(user_coalesced, "^ad\\\\"), replace(user_coalesced, "^ad\\\\", ""), true(), user )    Any ideas on how I can get this down to one? I thought about putting the coalesce() into each case, but that seems inefficient. ... View more

I have a tstats search that isn't returning a count consistently. In the where clause, I have a subsearch for determining the time modifiers. Here's the search:   | tstats count from datamodel=Vulnerabilities.Vulnerabilities where index=qualys_i [| search earliest=-4d@d index=_internal host="its-splunk7-hf.ucsd.edu" sourcetype="ta_QualysCloudPlatform*" host_detection ("Done loading detections" OR "Running now") | stats `stime(_time)` `slist(_raw)` count by PID | eval duration = last_seen - first_seen ,earliest = strftime(first_seen - 300, "%m/%d/%Y:%H:%M:%S") ,latest = strftime(last_seen + 300, "%m/%d/%Y:%H:%M:%S") | where count > 1 AND duration < 82800 | sort -last_seen | head 1 | return earliest latest ]   If I run the subsearch on its own...   earliest=-4d@d index=_internal host="its-splunk7-hf.ucsd.edu" sourcetype="ta_QualysCloudPlatform*" host_detection ("Done loading detections" OR "Running now") | stats `stime(_time)` `slist(_raw)` count by PID | eval duration = last_seen - first_seen ,earliest = strftime(first_seen - 300, "%m/%d/%Y:%H:%M:%S") ,latest = strftime(last_seen + 300, "%m/%d/%Y:%H:%M:%S") ``` Exclude results that ran over 23 hours or didn't finish ``` | where count > 1 AND duration < 82800 | sort -last_seen | head 1 | return earliest latest   I get the time modifiers accurately (e.g., earliest="11/05/2021:06:25:51" latest="11/05/2021:11:31:12"). When I inspect the job (of the first search), it is able to derive the same time modifiers (in phase0, phase1, and remoteSearch). The issue - when I run the first search, my count is double. In other words, it's double counting each record. If I explicitly put the time modifiers in place of the subsearch, the count is accurate (not double). Anyone run into this? ... View more

I am ingesting Qualys data via the Qualys Technology Add-on for Splunk (v1.8.7). To reduce daily volume, I have chosen to ingest incrementally (i.e., do not perform a full backup/pull every day). Fortunately, the TA tags events appropriate to leverage the Vulnerabilities data model. Unfortunately, since I'm only receiving incremental updates every day, I need to perform a dedup after querying the accelerated data model in order to get the current/latest status for each vulnerability. For example, most of my searches begin like this: | tstats count from datamodel=Vulnerabilities where index=qualys by _time, Vulnerabilities.dest, Vulnerabilities.dest_id, Vulnerabilities.dest_host, Vulnerabilities.severity_id, Vulnerabilities.severity, Vulnerabilities.signature_id, Vulnerabilities.status, Vulnerabilities.type span=1s | fields - count | dedup Vulnerabilities.signature_id Vulnerabilities.dest_id sortby -_time The tstats command targets the accelerated data model, and separates each event by vulnerability and host. The fields command remove the count, since it's not used. The dedup command removes all events, except for the most recent, on signature_id (QID) and dest_id (HOST_ID). This works well when filtering individual hosts or QIDs in the tstats where clause, but if I wanted to gather a count of all active, confirmed vulnerabilities by severity and host, The search is impossibly long because it first puts all results into memory before attempting a dedup. Ideally, part of running the data model acceleration would be able to update the summary with only the latest status, so all that is needed is running tstats against the data model without a dedup. Unfortunately, this doesn't appear to be possible. Anyone have any ideas on how to deal with this issue? ... View more

In the Splunk Add-on Builder, I configured a modular input using a REST API to pull data from FortiOS/FortiGate. I am trying to pass global account values (setup parameters), but the recommended variables from the following link do not work when I test the connection: https://docs.splunk.com/Documentation/AddonBuilder/3.0.1/UserGuide/ConfigureDataCollection#Pass_values_from_setup_parameters Here is the error message: [ERROR] - [test] HTTPError reason=HTTP Error Unable to find the server at %7b%7bglobal_account.username%7d%7d when sending request to url=https://{{global_account.username}}/api/v2/cmdb/firewall/address?with_meta=true&vdom=*&access_token={{global_account.password}} method=GET I'm guessing the variable syntax is incorrect, or am I wrong to try and pass the global account values to the URL? ... View more

In the Splunk Add-on Builder, I configured a modular input using a REST API to pull data from FortiOS/FortiGate. I tested my REST URL and received the following error message: [ERROR] - [test] HTTPError reason=HTTP Error Content purported to be compressed with deflate but failed to decompress. when sending request to url=https://x.x.x.x/api/v2/cmdb/firewall/address?with_meta=true&vdom=*&access_token=xxxxxxxxxxxxx method=GET When I tested the URL in a browser, the JSON comes back fine. Authentication takes place in the URL via the access_token parameter. Any ideas on how to get around this compression error? Here is the full log: 2020-03-19 19:42:56,347 - test_fortigate_configuration_firewall_addresses - [ERROR] - [test] HTTPError reason=HTTP Error Content purported to be compressed with deflate but failed to decompress. when sending request to url=https://x.x.x.x/api/v2/cmdb/firewall/address?with_meta=true&vdom=*&access_token=xxxxxxxxxxxxx method=GET Traceback (most recent call last): File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/httplib2/__init__.py", line 455, in _decompressContent content = zlib.decompress(content, -zlib.MAX_WBITS) zlib.error: Error -3 while decompressing data: invalid stored block lengths During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/cloudconnectlib/core/http.py", line 224, in _retry_send_request_if_needed uri=uri, body=body, method=method, headers=headers File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/cloudconnectlib/core/http.py", line 195, in _send_internal uri, body=body, method=method, headers=headers File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/httplib2/__init__.py", line 1957, in request cachekey, File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/httplib2/__init__.py", line 1622, in _request conn, request_uri, method, body, headers File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/httplib2/__init__.py", line 1592, in _conn_request content = _decompressContent(response, content) File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/httplib2/__init__.py", line 466, in _decompressContent content, httplib2.FailedToDecompressContent: Content purported to be compressed with deflate but failed to decompress. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/cloudconnectlib/core/engine.py", line 291, in _send_request response = self._client.send(request) File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/cloudconnectlib/core/http.py", line 275, in send url, request.method, request.headers, request.body File "/opt/splunk/etc/apps/sherlock_fortigate-config_ta/bin/sherlock_fortigate_config_ta/aob_py3/cloudconnectlib/core/http.py", line 229, in _retry_send_request_if_needed raise HTTPError('HTTP Error %s' % str(err)) cloudconnectlib.core.exceptions.HTTPError: HTTP Error Content purported to be compressed with deflate but failed to decompress. 2020-03-19 19:42:56,347 - test_fortigate_configuration_firewall_addresses - [INFO] - [test] This job need to be terminated. 2020-03-19 19:42:56,347 - test_fortigate_configuration_firewall_addresses - [INFO] - [test] Job processing finished 2020-03-19 19:42:56,347 - test_fortigate_configuration_firewall_addresses - [INFO] - [test] 1 job(s) process finished 2020-03-19 19:42:56,347 - test_fortigate_configuration_firewall_addresses - [INFO] - [test] Engine executing finished ... View more

I'm trying to configure the Duo Splunk Connector on a Splunk heavy forwarder to leverage the web proxy configuration I have in Splunk's server.conf. This configuration works for all Splunk web communication, but doesn't seem to apply for the Duo inputs. ... View more