My splunk architecture is as below:
UF - > HF -> IX - SH
Here, UF and HF are in same network where as IX and SH are in Splunk cloud. I need to run certain script on UF only on user request. I can not schedule it on time basis. However, considering my SH is in AWS cloud and out of the network boundry, how can I run the script based on user demand. Is there any solution/workaround for this?
What determines when the script runs?
Are you trying to make a search (on the SH) fire a script action on the UF?
What does the script on the UF have to do?
I want to trigger the script when user clicks the Submit button on a dashboard. Does the purpose of the script matters how it can be done? Anyways, the script needs to read the server local time and few other details.
You will need to do quite a bit of development outside of Splunk to make this work.
You would need:
c.) An application server to which you fire the event from the Splunk UI, and that server will need to orchestrate the remote data collection
d.) A client installed on your remote UF to communicate with your application server - or - the application server needs some mechanism to be able to remotely invoke a powershell or some other process.
e.) The results of the remote script need to be sent to Splunk for indexing
All of the above is totally possible, but you would need to develop and build this yourself.
You might be able to get some of the way using Splunk workflows and/or Phantom, but you are straying quite far from "conventional" Splunk usage.
Is there any reason you cant use a scripted input and schedule the script to run every 5 minutes? That would get you very close to the desired outcome with very little effort.
Thanks for such a detailed approach.
One QQ about #c. Is this technically possible even if your SH and Application servers are not in the same domain? I mean one in Splunk Cloud and another in private network?
The reason behind exploring this option vs running scripted input is because the data I want to get from UF is needed very rarely at random point in time. So don't see a value in indexing it at regular intervals.
But I think I need to balance between solution complexity vs data availability here. Anyways, thanks once again for your comments.
An application server capable of orchestrating this action needs to be able to communicate with your UFs and be reachable by the SH.
Again this is all totally feasible, but if you are considering exposing endpoints across the internet, be very certain that the application is sufficiently robust to resist misuse, especially if it acts as a gatekeeper to run scripts on your endpoints.
To be clear..
Nothing about my proposed solution sounds like a good idea to me! - I certainly would not be happy with this approach, so I wasn't trying to say thats how you "should" do it. 🙂
The approach I would take is the scripted input and I would make sure that the output is as concise (small, in bytes) as possible. As you say there is a tradeoff between indexed data which may never be used, and functionality.
If the use case must have the onclick() functionality and based on @nickhillscpl highlighted concerns, you can explore the Hybrid search approach. Where you will setup a SH on prem and have it in a secured zone.