We've been collecting data with the inputs add-on (Input Add On for SentinelOne App For Splunk) for several years now.  The applications channel has always been a bit problematic with the collection process running for several days but now we haven't seen any data since Monday February 19th around 5:00 PM. It's February 22nd and we generally see applications data every day. We started seeing errors on February 16th error_message="cannot unpack non-iterable NoneType object" error_type="&lt;class 'TypeError'&gt;" error_arguments="cannot unpack non-iterable NoneType object" error_filename="s1_client.py" error_line_number="500" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" error_message="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_type="&lt;class 'management.mgmtsdk_v2.exceptions.InternalServerErrorException'&gt;" error_arguments="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_filename="s1_client.py" error_line_number="223" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" And have seen a few errors since then error_message="cannot unpack non-iterable NoneType object" error_type="&lt;class 'TypeError'&gt;" error_arguments="cannot unpack non-iterable NoneType object" error_filename="s1_client.py" error_line_number="500" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" error_message="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_type="&lt;class 'management.mgmtsdk_v2.exceptions.InternalServerErrorException'&gt;" error_arguments="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_filename="s1_client.py" error_line_number="188" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" error_message="cannot unpack non-iterable NoneType object" error_type="&lt;class 'TypeError'&gt;" error_arguments="cannot unpack non-iterable NoneType object" error_filename="s1_client.py" error_line_number="500" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" error_message="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_type="&lt;class 'management.mgmtsdk_v2.exceptions.InternalServerErrorException'&gt;" error_arguments="[{'code': 5000010, 'detail': 'Server could not process the request.', 'title': 'Internal server error'}]" error_filename="s1_client.py" error_line_number="188" input_guid="8bb303-be5-6fe3-1b6-63a0c52b60c" input_name="Applications" After noting the following in the release notes Improvements ... -- Applications input uses a new S1 API endpoint to reduce load on ingest. we upgraded the add-on from version 5.19 to version 5.20. Now we're seeing the following messages in the sentinelone-modularinput.log 2024-02-22 13:40:02,171 log_level=WARNING pid=41568 tid=MainThread file="sentinelone.py" function="get_channel" line_number="630" version="IA-sentinelone_app_for_splunk.5.2.0b87" action=saving_checkpoint msg='not saving checkpoint in case there was a communication error' start=1708026001000 items_found=0 channel=applications 2024-02-22 13:40:01,526 log_level=WARNING pid=41568 tid=MainThread file="sentinelone.py" function="get_channel" line_number="599" version="IA-sentinelone_app_for_splunk.5.2.0b87" action=calling_applications_channel status=start start=1708026001000 start_length=13 start_type=<class 'str'> end=1708630801000 end_length=13 end_type=<class 'str'> checkpoint=1708026001.525169 channel=applications 2024-02-22 13:40:01,526 log_level=WARNING pid=41568 tid=MainThread file="sentinelone.py" function="get_channel" line_number="580" version="IA-sentinelone_app_for_splunk.5.2.0b87" action=got_checkpoint checkpoint={'last_execution': 1708026001.525169} channel=applications last_execution=1708026001.525169 2024-02-22 13:40:01,525 log_level=WARNING pid=41568 tid=MainThread file="sentinelone.py" function="get_channel" line_number="565" version="IA-sentinelone_app_for_splunk.5.2.0b87" action=got_checkpoint checkpoint={'last_execution': 1708026001.525169} channel=applications type=<class 'dict'> It appears that the input is running but we're not seeing any events.  We also noted the following in the documentation for version 5.2.0. sourcetype SentinelOne API Description ...     sentinelone:channel:applications web/api/v2.1/installed-applications Deprecated sentinelone:channel:applications web/api/v2.1/installed-applications Deprecated Does this mean that the input has been deprecated? If so, what does the statement "Applications input uses a new S1 API endpoint to reduce load on ingest." in the release notes mean?  And why is the Applications channel still an option when creating inputs through the Splunk IU? Any information you can provide on the application channel would be greatly appreciated. __PRESENT ... View more

I am observing intermittent issues parsing IIS data.  Splunk is configured for index time parsing of IIS events on the universal forwarders (INDEXED_EXCTRACTIONS).  The extraction works fine for most events, but a small percentage (less than 1%) fail parsing. I am detecting the events that fail parsing with the following SPL index=[IIS INDEXES] sourcetype=iis NOT c_ip=* I have noticed an error in the splunkd.log on the universal forwarders that accounts for some of these issues. 04-06-2022 20:08:42.602 -0500 WARN CsvLineBreaker - Parser warning: Encountered unescaped quotation mark in field while parsing. This may cause inaccurate field extractions or corrupt/merged events. - data_source="e:\iis-logs\W3SVC1\u_ex220407.log", data_host="XXXXX", data_sourcetype="iis" In these cases, it appears that not only does index time field parsing fail but event breaking fails resulting many events getting lumped into a single event.  This may not be avoidable and we’re at least able to point to a cause for these issues but many more are unexplained. For most of the events that fail parsing the result is a single line event which appears to be formatted correctly but has no indexed fields.  I was originally having an issue with these events reporting in the future as well but adding a time zone to props.conf seems to have at least resolved that issue. I have upgraded through several versions (8.1.2, 8.2.3, 8.2.7.1) on the Universal forwarders and have seen this issue across all these versions. If you have and ideas on what might be causing failures in index time parsing issues for IIS data I would love to hear them. ... View more

The sentinelone.py process for the applications channel input is running under a single PID for several days.  It does not appear to be respecting the checkpoint.  @aplura_llc_supp  any assistance on whether this is expected behavior and if not how the issue might be resolved would be greatly appreciated. We have three Sentinel One input channels enabled (Agents, Threats, Applications).  The modular input is configured with an interval of 300 for the threats channel seems to run fairly quickly (less than 2 minutes) and does not seem to be duplicating ingest.   The modular input for the agents channel is configured with an interval of 86400 and seems to run in about 45 minutes to 1 hour but does seem to be duplicating ingest based on the following search.     index=sentinelone_index sourcetype="sentinelone:channel:agents" earliest=1 latest=now | rex mode=sed field=_raw "s/, \"modular_input_consumption_time\": \"\w{3}, \d{2} \w{3} \d{4} \d{2}:\d{2}:\d{2} (\+|\-)\d{4}\"/, \"modular_input_consumption_time\": \"\"/g" | stats count as dup_count by _raw | stats count by dup_count     The modular input for the applications channel is configured with an interval of 3600. It runs for multiple days with the same PID and seems to be duplicating ingest based on a similar search.  It seems that it may not be respecting the checkpoint. The checkpoints for all three input channels appear to be getting set correctly on the following path $SPLUNK_HOME/var/lib/splunk/modinputs/sentinelone/. /opt/splunk/var/lib/splunk/modinputs/sentinelone/usea1-014.sentinelone.net_sentinelone-input-efd4172-40fe-b76-811f-c8cdf72132e-channel-applications.json   {"next_page": "", "last_execution": "1637470721"}   The input also appears to get the checkpoint successfully   2021-11-29 07:40:21,521 log_level=INFO pid=31196 tid=MainThread file="s1_client.py" function="get_channel" line_number="373" version="sentinelone_app_for_splunk.v5.1.2.b35" action=calling_applications_channel status=start start=1637470721000 start_length=13 start_type=<class 'str'> end=1638193221000 end_length=13 end_type=<class 'str'> checkpoint=1637470721 channel=applications 2021-11-29 07:40:21,521 log_level=WARNING pid=31196 tid=MainThread file="s1_client.py" function="get_channel" line_number="365" version="sentinelone_app_for_splunk.v5.1.2.b35" action=got_checkpoint checkpoint={'next_page': '', 'last_execution': '1637470721'} channel=applications   The Input Add On for SentinelOne App For Splunk (IA-sentinelone_app_for_splunk) is installed on the heavy forwarder.  The Input Add On for SentinelOne App For Splunk (TA-sentinelone_app_for_splunk) is installed on a search head cluster and a stand alone Enterprise Security search head.  The SentinelOne App For Splunk is not currently installed. All input channels are producing events. Any ideas on how to troubleshoot and or resolve this issue would be appreciated. ... View more