These were the recent error messages. Also, I am loading LLM in bin folder. Is that an issue? Following is the output I can see when I ran the .py file in the terminal   splunkd.log 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroup 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroup 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroups 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroups 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUser 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUser 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUsers 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUsers 07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: reading standard input 07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 [begin] SIGUSR1 handler 07-30-2024 18:23:30.002 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 Supervisor logs printed at : /opt/splunk/var/log/splunk 07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts Not supported, since no remote queue is configured in inputs.conf 07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts message="Unable to load smartbus conf" 07-30-2024 18:23:30.110 -0500 ERROR loader [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor 07-30-2024 18:23:30.110 -0500 ERROR pipeline [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor 07-30-2024 18:23:30.110 -0500 ERROR PipelineComponent [3255548 MainThread] - The pipeline datalakeinput threw an exception during initialize 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroup 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroup 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroups 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroups 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUser 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUser 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUsers 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUsers ... View more

I figured out that is not causing the issue, now my disk space is 2GB and MinFreeSpace 500mb.  Also, I am loading huggingface models in splunk/apps/myapp/bin/user-queries.py file.  When I hardcoded output without loading model, it worked. also, I am able to see response from my model for the query on the terminal.   The recent errors in splunkd.log are: 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroup 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroup 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroups 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroups 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUser 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUser 07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUsers 07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUsers 07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: reading standard input 07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 [begin] SIGUSR1 handler 07-30-2024 18:23:30.002 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 Supervisor logs printed at : /opt/splunk/var/log/splunk 07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts Not supported, since no remote queue is configured in inputs.conf 07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts message="Unable to load smartbus conf" 07-30-2024 18:23:30.110 -0500 ERROR loader [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor 07-30-2024 18:23:30.110 -0500 ERROR pipeline [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor 07-30-2024 18:23:30.110 -0500 ERROR PipelineComponent [3255548 MainThread] - The pipeline datalakeinput threw an exception during initialize 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroup 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroup 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroups 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroups 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUser 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUser 07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUsers 07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUsers ... View more

I checked splunkd.log and and found the error::: The diskspace remaining=0 is less than 1 x minFreeSpace on /opt/splunk/var/lib/splunk/_metrics/db 07-29-2024 11:16:16.785 -0500 WARN DiskMon [2047349 indexerPipe] - MinFreeSpace=5000. The diskspace remaining=0 is less than 1 x minFreeSpace on /opt/splunk/var/lib/splunk/_metrics/db 07-29-2024 11:16:16.969 -0500 ERROR Logger [2047618 HttpDedicatedIoThread-3] - Error writing to "/opt/splunk/var/log/splunk/splunkd_access.log": No space left on device help me in resolving this ... View more

 I checked, python.log is not throwing any errors and something like this : <{"name": "/en-US/static/@8FAC62B5BA1D0F7D9B1061086E2CF5B3713CA4C1EA33ED54B8F3EAA6F4E20FF2/fonts/splunkicons-regular-webfont.woff", "hqp": false, "redirect": 0, "dns": 0, "tcp": 0, "blocked": 9, "request": 23, "response": 1, "processing": null, "load": null, "ts": 13362}, {"name": "long-animation-frame", "hqp": false, "redirect": null, "dns": null, "tcp": null, "blocked": null, "request": null, "response": null, "processing": null, "load": null, "ts": null}, {"name": "long-animation-frame", "hqp": false, "redirect": null, "dns": null, "tcp": null, "blocked": null, "request": null, "response": null, "processing": null, "load": null, "ts": null}> search.log has <ERROR SearchMessages - orig_component="script" app="search" sid="1722375466.3" message_key="EXTERN:SCRIPT_NONZERO_RETURN" message=External search command 'test' returned error code 1.> The model I am using is LLM, Can Splunk able to load the large language models?  ... View more

I'm hoping you've found a solution. I'm working on a similar project where I created an app in splunk/etc/apps/my-app with a .py file in the bin folder and a .conf file in the default folder. Initially, when I ran the command <| mycommand "hello"> in Splunk, it outputted a response that I had hardcoded in my .py file. However, after updating the script to generate responses via a large language model, I started encountering the following error. Error in 'mycommand' command: External search command exited unexpectedly with non-zero error code 1. The search job has failed due to an error. You may be able view the job in the Job Inspector. Please help me with this.  Thanks in advance #splunk #LLM's ... View more