UserQuery Option for Splunk Dashboard

[syaganti]

Hey, 

I am doing Predictive Maintenance using LLM's and I want to use Splunk to build dashboard. There I am going to include Question/Answering mechanism where my model going to answer the question user has. 

For that, I have created an app "customApp" in src/etc/apps and added user-queries.py file in apps/customApp/bin folder and command.conf file in apps/myapp/default folder. 

I am getting the following error on Splunk dashboard when calling this command. Btw, it worked when I hardcoded instead of using my model to generate response.

<Error in 'userquery' command: External search command exited unexpectedly with non-zero error code 1.>

Can you someone please save me from this, Thanks in advance!

#CustomApp / #userQuery / #Dashboard / #LLM's / #models

[syaganti]

I figured out that is not causing the issue, now my disk space is 2GB and MinFreeSpace 500mb.  Also, I am loading huggingface models in splunk/apps/myapp/bin/user-queries.py file. 
When I hardcoded output without loading model, it worked.

also, I am able to see response from my model for the query on the terminal.

The recent errors in splunkd.log are:

07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroup
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroup
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroups
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroups
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUser
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUser
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUsers
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUsers
07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: reading standard input
07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 [begin] SIGUSR1 handler
07-30-2024 18:23:30.002 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 Supervisor logs printed at : /opt/splunk/var/log/splunk
07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts Not supported, since no remote queue is configured in inputs.conf
07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts message="Unable to load smartbus conf"
07-30-2024 18:23:30.110 -0500 ERROR loader [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor
07-30-2024 18:23:30.110 -0500 ERROR pipeline [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor
07-30-2024 18:23:30.110 -0500 ERROR PipelineComponent [3255548 MainThread] - The pipeline datalakeinput threw an exception during initialize
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroup
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroup
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroups
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroups
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUser
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUser
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUsers
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUsers

[richgalloway]

Check search.log and python.log for any messages that might explain why the script returned an error code.

[syaganti]

 I checked, python.log is not throwing any errors and something like this :

<{"name": "/en-US/static/@8FAC62B5BA1D0F7D9B1061086E2CF5B3713CA4C1EA33ED54B8F3EAA6F4E20FF2/fonts/splunkicons-regular-webfont.woff", "hqp": false, "redirect": 0, "dns": 0, "tcp": 0, "blocked": 9, "request": 23, "response": 1, "processing": null, "load": null, "ts": 13362}, {"name": "long-animation-frame", "hqp": false, "redirect": null, "dns": null, "tcp": null, "blocked": null, "request": null, "response": null, "processing": null, "load": null, "ts": null}, {"name": "long-animation-frame", "hqp": false, "redirect": null, "dns": null, "tcp": null, "blocked": null, "request": null, "response": null, "processing": null, "load": null, "ts": null}>

search.log has

<ERROR SearchMessages - orig_component="script" app="search" sid="1722375466.3" message_key="EXTERN:SCRIPT_NONZERO_RETURN" message=External search command 'test' returned error code 1.>


The model I am using is LLM, Can Splunk able to load the large language models? 

[syaganti]

I checked splunkd.log and and found the error:::

The diskspace remaining=0 is less than 1 x minFreeSpace on /opt/splunk/var/lib/splunk/_metrics/db
07-29-2024 11:16:16.785 -0500 WARN DiskMon [2047349 indexerPipe] - MinFreeSpace=5000.

The diskspace remaining=0 is less than 1 x minFreeSpace on /opt/splunk/var/lib/splunk/_metrics/db
07-29-2024 11:16:16.969 -0500 ERROR Logger [2047618 HttpDedicatedIoThread-3] - Error writing to "/opt/splunk/var/log/splunk/splunkd_access.log": No space left on device


help me in resolving this

[richgalloway]

The disk is full.  The resolution is to free up space or add more storage.

Indexes should not be allowed to grow large enough to fill the disk.  Indexes should be part of a volume that is smaller than the total disk space available (to allow for metadata).  That MinFreeSpace is 5GB and disk space is 0 tells me that Splunk is sharing storage with other applications and/or the OS.  That is a Bad Practice.

The operating system, $SPLUNK_HOME, and $SPLUNK_DB should be on separate and independent mount points.

[syaganti]

These were the recent error messages. Also, I am loading LLM in bin folder. Is that an issue?

Following is the output I can see when I ran the .py file in the terminal

splunkd.log
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroup
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroup
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimGroups
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimGroups
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUser
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUser
07-30-2024 18:23:29.862 -0500 ERROR RegisterPackageHandler [3255548 MainThread] - Failed to get destination for endpoint: tlPackage-scimUsers
07-30-2024 18:23:29.862 -0500 ERROR LocalProxyRestHandler [3255548 MainThread] - destination not found for tlPackage-scimUsers
07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: reading standard input
07-30-2024 18:23:29.998 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 [begin] SIGUSR1 handler
07-30-2024 18:23:30.002 -0500 ERROR SidecarThread [3255548 MainThread] - <stderr> Sidecar: 2024/07/30 18:23:30 Supervisor logs printed at : /opt/splunk/var/log/splunk
07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts Not supported, since no remote queue is configured in inputs.conf
07-30-2024 18:23:30.098 -0500 ERROR NoahHeartbeat [3255712 SplunkdSpecificInitThread] - event=deleteReceipts message="Unable to load smartbus conf"
07-30-2024 18:23:30.110 -0500 ERROR loader [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor
07-30-2024 18:23:30.110 -0500 ERROR pipeline [3255846 datalakeinput] - Couldn't find library for: datalakeinputprocessor
07-30-2024 18:23:30.110 -0500 ERROR PipelineComponent [3255548 MainThread] - The pipeline datalakeinput threw an exception during initialize
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroup
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroup
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimGroups
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimGroups
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUser
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUser
07-30-2024 18:23:35.350 -0500 ERROR RegisterPackageHandler [3255554 HTTPDispatch] - Failed to get destination for endpoint: tlPackage-scimUsers
07-30-2024 18:23:35.350 -0500 ERROR LocalProxyRestHandler [3255554 HTTPDispatch] - destination not found for tlPackage-scimUsers