Hello everyone, I'm facing an inconsistent parsing issue while ingesting JSON events through HEC and I'd appreciate any guidance. Goal I'm trying to: Change the incoming sourcetype (currently arriving as json"). Dynamically assign the source value based on the value of evt.name inside each event. The sourcetype assignment works perfectly, but the dynamic source assignment only works for approximately 80-90% of the events. The remaining 10-20% keep the original source instead of being updated. Current configuration transforms.conf [set_source_from_evt] REGEX = .+\"evt\"\s*:\s*\{[\s\S]*?\"name\"\s*:\s*\"([^\"]+)\".+ FORMAT = source::$1 DEST_KEY = MetaData:Source [set_datadog_sourcetype] REGEX = . DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::datadog:audit props.conf [json"] TRANSFORMS-fix_sourcetype = set_datadog_sourcetype, set_source_from_evt I also tried applying the transform after changing the sourcetype: [datadog:audit] TRANSFORMS-set_source = set_source_from_evt but the behavior remained exactly the same. Troubleshooting already performed The regular expression has been tested with multiple online regex tools and matches 100% of the events, including those that fail during indexing. I also tested the same regex in Splunk Search (using rex) against indexed events, and it successfully extracts the value from every event. The issue is not tied to a specific value of evt.name; it appears to happen randomly. Sometimes an event is parsed correctly, while another event with the same structure is not. For example, the following fragment should produce: source = Synthetics Monitoring because it contains: "tags":["public_id:ec8-knz-4bv","version:4"],"evt":{"name":"Synthetics Monitoring","actor":{"type":"USER"}}},"message": However, some events containing this structure still keep the original source. Question Has anyone experienced inconsistent behavior when setting MetaData:Source using a transform during HEC ingestion? Is there any known limitation regarding: MetaData:Source transforms on HEC data? The use of [\s\S]*? in index-time regex? Regex evaluation at index time versus search time? Event parsing/truncation/order of processing that could explain why identical events are sometimes matched and sometimes not? Any ideas or suggestions would be greatly appreciated. Thanks! ... View more

Hi everyone, I'm currently experiencing an intermittent issue with the CrowdStrike Falcon Event Streams Technical Add-On in Splunk Enterprise, and I’d like to know if anyone else has faced something similar or has insights into a possible solution. Environment: Splunk Enterprise (on-prem) CrowdStrike Falcon Event Streams Technical Add-On (latest version, issue also occurred in previous versions) Indexers and Search Head in cluster Issue description: Approximately every 10–15 days, the CrowdStrike input stops ingesting events. The only workaround so far has been to restart the input, after which ingestion resumes normally. Relevant logs (_internal): File "/opt/splunk/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "/opt/splunk/lib/python3.9/site-packages/urllib3/connectionpool.py", line 407, in _make_request self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) File "/opt/splunk/lib/python3.9/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPoo1(host='api.crowdstrike.com' ,port=443): Read timed out. (read timeout=10) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/OAuth2.py", line 35, in get_token response = helper.send_http_request(url=tokenURL, method="POST", timeout=10, payload-payload, headers=headers, use_proxy-proxy) File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/../lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 496, in send_http_requ est return self.rest_helper.send_http_request( File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/../lib/splunktaucclib/splunk_aoblib/rest_helper.py" , line 68, in send_http_request return self.http_session. request(method, url, **requests_args) File "/opt/splunk/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/opt/splunk/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/opt/splunk/lib/python3.9/site-packages/requests/adapters.py", line 713, in send raise ReadTimeout(e, request=request) requests. exceptions. ReadTimeout: HTTPSConnectionPoo1(host='api.crowdstrike.com', port=443): Read timed out. (read timeout=10) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/../lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 141, in stream_events self. collect_events(ew) File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/crowdstrike_event_streams.py", line 485, in collect_events crowdstrike_client() File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/crowdstrike_event_streams.py", line 354, in crowdstrike_client token_result, token_message, token_url= Stream() get_token(clientid, secret, api_endpoint, proxy, stanza_name, helper, user_agent, event_streams_titl e) File "/opt/splunk/etc/apps/TA-crowdstrike-falcon-event-streams/bin/OAuth2.py", line 67, in get_token result_code = str(response. status_code) UnboundLocalError: local variable 'response' referenced before assignment Analysis performed: The issue occurs during the OAuth2 token retrieval process from the CrowdStrike API. A timeout (10 seconds) happens in the HTTPS request. Due to improper exception handling, the response variable is never assigned, leading to an UnboundLocalError. After this failure, the input appears to become “stuck” and stops ingesting new events. Additionally, the offset handling becomes inconsistent: the input attempts to retrieve older events and does not properly resume real-time ingestion. Restarting the input restores normal behavior. What has been ruled out: No general connectivity issues detected No KV Store-related problems Add-on is up to date Questions: Has anyone encountered similar behavior with this add-on or other modular inputs using OAuth2? Any recommendations or solution to prevent the input from getting stuck after a failed token request? Any guidance or shared experiences would be greatly appreciated. Thanks in advance! ... View more

Hi, I’ve noticed an inconsistency with the Last Login Time field in Settings > Users. For two specific users, the timestamps shown in the UI don’t match what I see in the _audit index. For one user, for example, the UI shows the last login as May 19, but _audit shows a successful login on August 18. For another user, they have logged in at least two times after the date shown in the UI, but the Last Login Time in the UI has not updated. It looks like the UI field is not being refreshed correctly. Has anyone else experienced this issue? Is there a known way to force the Last Login Time field to update? Does anyone know the root cause of this problem? The Splunk version installed is 9.4.1 enterprise Thanks in advance for your help! ... View more