Hi @bgill0123, You should run your search using a time range of 15 days or more to be able to compare and use "delta" command like below; (assuming you 5 days average hit count field is "five_days_avg_count") | delta five_days_avg_count as diff | eval perc_diff=abs(diff*100/five_days_avg_count) | search perc_diff > 10  

Hi @Skeer-Jamf, This is out of Splunk context but you can check Linux Keepalived service for redundancy. Keepalived supports active/passive failover mode and load balancing setup is possible. It creates and manages virtual IP address that forwards the incoming traffic to healthy backend servers.  

Thank you @gcusello   When I look at the add-on you posted, it seems to be a tool that helps to use OpenAI, within Splunk, to request additional context/perspective on specific questions or monitors (e.g. "Is this malicious?") My goal is actually just to get the content into Splunk first. So, for example, a web app that has a chatbot built in. The chatbot might receive input in text (or image/video) from users and then it generates a response (e.g. "Here is the documentation/help file relevant to that particular request"). I want to store all the activity between the application/user and OpenAI for evaluating security or compliance concerns using Splunk searches. Any thoughts on that? I could ask the App Engineer to log the input and output via HEC I guess. But I'm wondering if others have started logging OpenAI calls/responses (or any LLM such as Anthropic or Cohere or Gemini etc.) into Splunk yet? Thanks!

The only reason I can think of to try to do such thing would be to set up a small lab for learning Splunk in a bit more "distributed" setup than just an all-in-one server. But in such case, I'd go for spinning up separate VMs and installing each component on a separate VM. Also be prepared for a very very low performance.

Main question is whether your data is getting truncated when you're displaying it (which is kinda unlikely) or has it been truncated on ingestion. Check your data with this: index=gbi_* (AppName=*) | eval strlen=len(_raw) | stats max(strlen)  

OK. So do search sourcetype="my_source" "failed request, request id=" | rex “failed request, request id=(?<request_id>[\w-]+)" | fields request_id | rename request_id as search | format And substitute manually your subsearch with the results from this one.

Hi @PickleRick , sorry for that typo search sourcetype="my_source" "failed request, request id=" | rex “failed request, request id=(?<request_id>[\w-]+)" | fields request_id | rename request_id as search above is the correct one 

From a strictly theoretical perspective, you could store your data on any storage your OS can access. After all Splunk uses system calls to access its files so as long as it can open those files, you're "good". But the problem is that not every storage performs equally well hence the rule of thumb about using local storage only. The "slow" storage which can be used for cold storage which is typically less often used means usually still relatively quick HDDs versus SDD recommended for hot/warm storage. Remember that latency in accessing slow storage would have noticeable impact on overall Splunk's performance, not just those searches that access cold data. That's one thing. Another thing is that if you want to reach over the network for data, Splunk process must be able to access the share the data is stored on so you will definitely _not_ be able to do so running Splunk with either LOCAL_SYSTEM user or the default Splunk user. But still, the most important thing is that you should not use NAS or NFS for Splunk storage - there is too much overhead and the latency is too high for reasonable performance.

OK. this thread is soooo long The search you just posted has definitely at least one error - there is no command called "request_id". It's your field's name. So either you copy-pasted it wrong or it's not gonna work at all.

Hi @PickleRick,   Yes I have that.  If you scroll on above conversation, I have pasted the result.  Do you want me to post the result of below query? search sourcetype="my_source" "failed request, request id=" | rex “failed request, request id=(?<request_id>[\w-]+)" | request_id | fields request_id | rename request_id as search

Splunk has a limit on how long a single event can be. If an event is longer, it is truncated on ingestion which means that only first $LIMIT (by default it's 10000) characters are stored within Splunk's index. The rest of the event is irrevocably lost. So you can't display what isn't there - it's simply not saved in your Splunk. That's why @gcusello said you have to talk with your Splunk team about raising the limit for this particular sourcetype if this is an issue for your data.

Again my 3 cents - do you have the field called request_ids in your data? Because that's what your subsearch will generate a condition for. And you don't need an explicit format command if you're not overriding any default options for it.