THANKS for the ESCU companion app hint. That's quite a good idea alongside an automatic merge concept I'm developing and producing a report for Analyst what to do. thanks for that and will mark as answered ... View more

thanks @livehybrid . Upvoted. I almost figured it out, but in a slightly different manner. I'm got an ansible setup for URL interaction and automation. The 'contentctl build' will produce artefact similar to a Splunk app with `savedsearches.conf` and other things like `analyticsstories.conf` contentctl build --path content --app.title MY_DETECT --app.appid DA-ESS-MY_DETECT --app.prefix MY --app.label MY Then i'm using the ansible automation which interacts with saved/searches and other endpoints to insert it back. Two things i'm still figuring out is it is slow once the savedsearches have 50+ searches as it runs one by one contentctl new : this option doesn't accept ALL parameters like search, name which means a user input is required Any chance for automation can detect if a savedsearch is Changed, then only insert   Update:  Able to insert into system after contentctl using REST api "saved/searches".  Though the type is specified as 'ebd' (event-based detection), while it is inserted into Splunk, it becomes a 'saved search' type !! any solutions/recommendations for this? ... View more

Faced the same issue. Though it shows 401 Authorized, the search  in "_internal" index showed its a  "JsonWebToken validation failure". This happens mostly when you got Splunk cloud and there is a maintenance or restart happening. The API bearer token you/service account using may need a kick. So please try 'logout' (user from Splunk console) or expire it from Splunk and relogin/retry. Hopefully that will fix it.  For us, it was a suddent stoppage of 3 or 4 Tokens at same time which gave the hint ... View more

thanks for the draw.io icons ... View more

Interactive in the sense, you can click in each and it can drill down into further details. The main reason for diagram as a code is, we can maintain in version control ... View more

so what you are saying,  just configure 'indexer clusters' in Each cloud environment and then use a 'SHC' from any of the cloud to search the 'indexer clusters' in ALL cloud environments? You sure it won't causes latency at time of SH aggregation? A diagram would be really appreciated ... View more

Agree Guiseppe. I was thinking if anyone in the community have done similar >> if yes you have to create a multisite cluster, one for each different Cloud, configuring replications between them and configuring Search Heads, in each Cloud, to search in all the clusters. Creating SHC and replications is not the problem; but the key is for the end-user, they have to Search from a single Search Head cluster and it should search in ALL the clusters in multi-cloud. (This could be SHC on top of another SHCs in each cloud or preferably directly to indexers on each clouds if feasible) Org. doesn't need data to be replicated between clouds, but we can create clusters within each cloud provider for Replicator factor. ... View more

I understand this is quite old, but please have a try like below.. - name: "SEARCH: Run Search and register Sid" uri: url: "{{splunk_ssl_uri}}:{{splunk.port.mgmt}}/services/search/jobs" method: POST user: xxxx password: xxxx body: - [ search, "{{splunk_search}}"] status_code: [200, 201] force_basic_auth: yes validate_certs: no body_format: form-urlencoded return_content: true vars: - splunk_search: "search index=_internal | stats count" register: splunk_search_register ... View more

great ideas. This is exactly I was looking for. Will get in touch with Sales on these topic/ideas ... View more

thanks for the idea. upvoted. Do you have a rough idea if it is cheaper vs the default licensing methodology? or is it more of a question for Splunk Sales? ... View more

Well, in my experience, many of the clients/users still don't understand their entire estate ! By the time, we reach someone would have estimated the data and said.. oh, its only 1000 windows & linux, so would be 10GB per day, while in reality the data would then start from "auditd", "applications" and far exceed 10x the initial estimate and suddently no funding available. I personally feel, Splunk should have a secondary license to cater for trivial/less-important data and collection and when it is required to search, there should be charged separately. Many customers never realise the value until unless they see it. ... View more

Agreed. I understand the current license structure, but was checking if there was a other type of license within Splunk or if any of you have guys done it in a different way Yeah, we have options to store outside Splunk. Of course, we may look into such a architecture using other products (as filesystem storage is bit clunky). There are few advanced thoughts as well, to store raw data in other tools and send a summary information to Splunk. So different avenues/thoughts, but wanted to see if there is any clever way within Splunk before venturing to other tools ... View more

Unfortunately the savedsearch logic is not within our control but by client. Also it may impact other elements like webhook/emails etc. if the manipulation happens at search level.   ... View more

Remember we are talking about "custom" applications and my estimate is around 1200+ applications/sourcetypes and understanding every bit and data types is not what the Splunk team at client site is not pursuing. Also the format of logs and version changes all the time and hence the small splunk content team can never catchup. So it has to be outsourced to the Application SME. I'm external consultant who will be called only during important or urgent requirements, so not fully into TA development these days. I've recommended few options that organisation should pursue logging standards as JSON so atleast key-value fields are present ... View more

Thanks Giuseppe. I've configured ES at client site and have all been working good for few years. But in practice client is not able to map everything to CIM as lot of effort is required and their core Splunk team cannot do all application style data thus relying on application team who don't have an idea about CIM or splunk fields. This goes back to customising use-cases which is available in backend (currently we couldn't figure out how to get the savedsearches/use-cases from Splunk ES Content update ) to modify it to fit non-CIM fields. I feel other large companies will also experience the same as not everyone would be able to map all fields to CIM and has to rely on SME providing search based on raw data which they understand.       ... View more

that's really not good.  >> is restricted from performing the following types of tasks... Installing apps and modifying app configurations almost cancels the whole point of automation then? How do you guys control the search-time and custom TA configurations in Splunk cloud & Version control them? Manually upload them and give to  Splunk support? ... View more

There are two parts to this - Technical side (Technically pooling of license will solve/fix the  issue) - Support and Legal side   My humble opinion is NEVER loose  out on the perpetual license though the marketing and salesmen from your organisation and everyone would like to  see  it  gone 🙂   My suggestion is - use  the perpetual to long term/stagnant set of  dataset, clusters which needs no support.  So have a pool with perpetual license and  apply to the indexers which  are part of pool - Try seeing  if you  need your  DEV/TEST server those licenses which  may not need support at all.  - If nothing happens, you could ask Splunk Rep to merge it and give term license  for you   ... View more