All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

we are have multiple splunk cluster and will need to generate a diag file everytime for search head or indexer..  so need to automat the process of generating the diage and upload in splunk support ... See more...
we are have multiple splunk cluster and will need to generate a diag file everytime for search head or indexer..  so need to automat the process of generating the diage and upload in splunk support case automatically.  i have script to generate a file and enter the case but spplunk support is will need api or some connection to login and search the case and upload the diag.   
Hi @vempatisuresh  Applying two set(body, ...) statements sequentially results in only the last one ("extracted") being set as the body. This eliminates your intended transformation. Try the foll... See more...
Hi @vempatisuresh  Applying two set(body, ...) statements sequentially results in only the last one ("extracted") being set as the body. This eliminates your intended transformation. Try the following: processors: transform/logs: log_statements: - context: log statements: - set(body, ParseJSON(body)["message"]) Pipeline inclusion: service: pipelines: logs: processors: [transform/logs, ...] For more info check out the docs at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @harishsplunk7  I wanted to check, are you using Windows or Linux UFs? UFs do not have Python installed as part of the Splunk deployment, therefore Python might not be best approach for this? ... See more...
Hi @harishsplunk7  I wanted to check, are you using Windows or Linux UFs? UFs do not have Python installed as part of the Splunk deployment, therefore Python might not be best approach for this?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @krishna821  Most of the REST API endpoints you're likely using for on-premise are also available in Cloud. The SplunkCloud REST API docs are at https://docs.splunk.com/Documentation/SplunkCloud... See more...
Hi @krishna821  Most of the REST API endpoints you're likely using for on-premise are also available in Cloud. The SplunkCloud REST API docs are at https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTREF/RESTprolog  You will need to ensure your egress IP is allow-listed on your Splunk Cloud environment as by default this is restricted. If you are not an admin on the Splunk Cloud platform then you will need to speak to your admin team to setup the allow-listing. For more information check out https://docs.splunk.com/Documentation/SplunkCloud/9.3.2411/Config/ConfigureIPAllowList Note: I would recommend using Token authentication over user/password login. If your Splunk Cloud instance is using SAML/SSO authentication then you will need to use a token.   Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
For the "auxiliary" servers (although CM is very important for cluster operations) the sizing hugely depends on a scale. You can have a TB-sized environment which still serves only a few dozens of UF... See more...
For the "auxiliary" servers (although CM is very important for cluster operations) the sizing hugely depends on a scale. You can have a TB-sized environment which still serves only a few dozens of UFs from DS so you can make this DS really small (6CPU would suffice; I've seen such environments) but you could as well have several thousands of UFs pulling from DS. Anyway, with DS you can significantly lower the server's load by increasing the polling period at the cost of increased "latency" of changes to deployed apps. CM also grows with the size of your environment. TB/day scale is still relatively moderate so it shouldn't need 24vCPUs for that. 
Thanks, I'll look into this and confirm the behavior.
In fact, if its specific data sources which you want to send to different places then you wont need to touch props/transforms - instead you can set _TCP_ROUTING in your inputs.conf stanzas, setting t... See more...
In fact, if its specific data sources which you want to send to different places then you wont need to touch props/transforms - instead you can set _TCP_ROUTING in your inputs.conf stanzas, setting the value to be the output group that you want to use, for example: == inputs.conf == [monitor:///some/path/someFile.log] index=someIndex sourcetype=myAppLogs _TCP_ROUTING=myOnPremOutputGroup [monitor:///some/path/IIS/logs] index=iis_logs sourcetype=iis:logs _TCP_ROUTING=mySplunkCloudOutputGroup Also worth reading https://community.splunk.com/t5/Getting-Data-In/Issue-with-default-outputs-when-TCP-ROUTING/m-p/509716  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Flobzh  Yes you can achieve this with multiple output groups in your outputs.conf and then props/transforms.conf to filter as required. For more details documentation and examples check out htt... See more...
Hi @Flobzh  Yes you can achieve this with multiple output groups in your outputs.conf and then props/transforms.conf to filter as required. For more details documentation and examples check out https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Routeandfilterdatad  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Luckily, we didn't have to tackle this one. It was way more important to us to move the content created than to remove it later.
Hey,  Thank you for assisting. 1PB is incredible. I though 1.7 TB was a lot. I am not too sure about instance type, I am reaching out to see. As far as Smart Store, we aren't using anything of the s... See more...
Hey,  Thank you for assisting. 1PB is incredible. I though 1.7 TB was a lot. I am not too sure about instance type, I am reaching out to see. As far as Smart Store, we aren't using anything of the sort i believe. We just have retention policy to roll over cold/frozen data. Ill look over the docs regarding the smart store.    In regard to min requirements, a lot of our administration servers, Deployment Server, some of the servers handling syslog data, DMZ heavy forwarders, Cluster Managers, etc, all have around 6-8 Cores, and roughly around 6 CPUs.  The Server pictured below/above, is of our Azure  Cluster Manager. It manages a cluster, and may index itself, not sure, but it only has 8 CPU and 4 cores. Should all servers at least meet the minimum requirements? Especially with our ingestion load? I would imagine so. I can work with Support to answer any specific questions as I already have an ODS case open to handle this Splunk version upgrade. This RHEL upgrade is being pushed due to RHEL 7's support expiring.  12 physical CPU cores, or 24 vCPU at 2 GHz or greater speed per core. 12 GB RAM. A 1 Gb Ethernet NIC, optional second NIC for a management network.  
Hello, Is it possible to have only 1 Universal Forwarder installed on a Windows server and this UF sends data to 2 different Splunk instances Ex: 1- Source: IIS logs -> Dest = SplunkCloud 2- Sour... See more...
Hello, Is it possible to have only 1 Universal Forwarder installed on a Windows server and this UF sends data to 2 different Splunk instances Ex: 1- Source: IIS logs -> Dest = SplunkCloud 2- Source: event viewer data -> Dest = On Premise Splunk Enterprise If yes can you point to an article that help setup this? Other possible constraint: we have a deployment server that should allow to setup both flow.   Thanks for your help
Can anyone give me idea or script python to generate a diag file in splunk using python script login to splunk support portal and enter the case number Upload the file automatically 
Hi!   @isoutamo looped me in b/c he knows I'm currently in an Azure environment that's doing ~1Pb/day First, are you using SmartStore to offload older events to blob storage?  If so, around 1TB/da... See more...
Hi!   @isoutamo looped me in b/c he knows I'm currently in an Azure environment that's doing ~1Pb/day First, are you using SmartStore to offload older events to blob storage?  If so, around 1TB/day you're going to want to start thinking about splitting up your cluster because Azure throttles blob upload/download.  That WILL cause latency problems.  And there's also a whole bunch of SmartStore tuning you'll need to consider to minimize cache thrashing.   If you're not using SmartStore then the math goes a completely different way. Generally, what instance types are you using?  We've evaluated the following and find them are more than capable at our scale dasv5-series  dasv6-series  lsv3-series  ebsv5-series  edsv5-series  edsv6-series  If you ARE using SmartStore keep in mind that theres no concept of hot/cold,  just local disk/remote store so some of the faster local NVME may not scale up to what you need for your local cache and going for systems that don't have that and instead can scale your attached disk for your local cache is the way to go. That was our situation which is why we chose instances that don't have local disk, but allow lots of disks to be attached. If you AREN'T using SmartStore, then you'll want to look at the other instance types and leverage the NVME local disk for hot/warm and teh attached disk as cold. Beyond that, it's just a matter of picking the right size of your instance types to meet your SF/RF needs and data Ingest/Search load.   SmartStore/blob storage is really the piece that makes Azure unique.  Let me know if you are using it and we can discuss how to go about splitting your storage account(s) and possibly splitting your cluster.
I am ,    I am working alongside the unix team, as they have a better understanding of storage and resource requirements than I do. But this upgrade is long overdue.  Thanks for the assistance. I ... See more...
I am ,    I am working alongside the unix team, as they have a better understanding of storage and resource requirements than I do. But this upgrade is long overdue.  Thanks for the assistance. I think I have enough for the required requests I am filling out. 
Based on that information your cluster sizing is not even a near the enough  I’m quite sure that your environment needs something else than cpu too. In Azure there are some other limitations and r... See more...
Based on that information your cluster sizing is not even a near the enough  I’m quite sure that your environment needs something else than cpu too. In Azure there are some other limitations and recommendations which you must handle with your current volumes. Let’s see if we can get some people who have worked more with bigger Azure splunk Installations?
Yes it is. I wasn't around when these were spun up, but now that it's up to me to fix it, I want to make sure we wont run into these issues for a few years.  If we already have 10 Physical indexers t... See more...
Yes it is. I wasn't around when these were spun up, but now that it's up to me to fix it, I want to make sure we wont run into these issues for a few years.  If we already have 10 Physical indexers that handle most of the data, would I need the 96 vCPU for the other 6 in Azure, I have to consider costs with this as well.    Thanks 
You could access SCP’s REST api, but you must enable it first. Here is instructions how to do it https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud
Hey,  Thanks for your response. With this azure cluster, we receive latency alerts all of the time. Comparing the installed and available resources to the recommended, we are under resourced. We hav... See more...
Hey,  Thanks for your response. With this azure cluster, we receive latency alerts all of the time. Comparing the installed and available resources to the recommended, we are under resourced. We have a clustered environment, two clusters at a different location for both the Search Heads as well as the Indexers. We have 10 Physical indexers, each with about 60TB of storage and 48 cores and 96 CPUs. Compare these indexers to the Azure ones, which have 4 Cores and 8 CPUs.   Our Azure cluster has been giving us latency warning for a few years now, even after a few upgrades. Now that I am more comfortable with our environment, I want to finally upgrade the cpu and memory to the recommended values.    We have 6 Azure Indexers, all of which have latency issues at some point or another. We have about 100 indexes, our top 3 sources ingest 600GB daily, and we average about 1.7 TB a day.    To sum up, these are under resourced, and they need more CPU.    Thanks
Yeh I have played with those modes and those helps in many cases and currently you can do things which haven’t been possible e.g. with 7.x versions. Those are defined here https://docs.splunk.com/Doc... See more...
Yeh I have played with those modes and those helps in many cases and currently you can do things which haven’t been possible e.g. with 7.x versions. Those are defined here https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/PropagateSHCconfigurationchanges But in curiosity how you manage those in this case that you could push initially all configurations from old SHC and later so that when user removed e.g. alerts from SHC by GUI and then it didn’t pop up after new apply without that you remove it first from deployer after initial push?
Hello, This is Krishna and I have been some POC about accessing Splunk logs through Rest API's. I was successful in calling the Rest API's through Spunk Enterprise version but in my company we have ... See more...
Hello, This is Krishna and I have been some POC about accessing Splunk logs through Rest API's. I was successful in calling the Rest API's through Spunk Enterprise version but in my company we have Splunk Cloud and so unable to call Rest API's as how I was able to do in Splunk Enterprise edition. I would like to know the details of how I can call Splunk Rest API's for Cloud edition. Below are my findings On my local instance of Splunk when I hit the below url it lists all the services available https://localhost:8089/services(it asked me for admin credentials which I provided) in which I am interested in the https://localhost:8089/services/search/jobs  so would like to call the similar ones for Cloud version   Thanks in Advance.