All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

You can find earlier discussions on this topic within Answers. It's... a bit tricky. There are at least several different things at play here. 1. The technicalities - as Splunk brings with it most ... See more...
You can find earlier discussions on this topic within Answers. It's... a bit tricky. There are at least several different things at play here. 1. The technicalities - as Splunk brings with it most of the things it requires and generally uses just the "bare OS" level of what is provided by OS, there shouldn't be a problem with running different OS versions (even different distros - I run a combined CentOS/SuSE environment for some time). It should work. 2. The maintainability - as different distros and different releases have some different mechanisms (like different startup scripts, different ways of configuring the system and so on), a mixed environment is much more prone to errors and misconfiguration. 3. The supportability - the official docs say that all cluster members must run the same "OS and version". And here's where it gets really tricky - there is no single official explanation what this means so while technically it could just mean that all boxes must be Linux-based and they must be running 64-bit OS version (and that could really be the bare minimum to make the cluster work), it can also be understood as "all boxes must use the same Linux distro and they all must be running the same release". So long story short - from the technical point of view it usually doesn't make much of a difference whether you're running RHEL9 across your whole environment or if some boxes are still at RHEL8 (if you already have a RHEL8 environment and want to migrate to RHEL9, you will have at some point a situation when some boxes are already migrated and some are not) but if you raise a support case and support finds out that you have a mixed setup, they might want to tell you "get your environment in order and align your OS versions".
Hi @Abass42  From my experience you can run a Splunk cluster (indexer or search head) with a mix of RHEL 7 and RHEL 8 hosts, including having your cluster manager on RHEL 8 while some peers remain o... See more...
Hi @Abass42  From my experience you can run a Splunk cluster (indexer or search head) with a mix of RHEL 7 and RHEL 8 hosts, including having your cluster manager on RHEL 8 while some peers remain on RHEL 7, as long as all Splunk nodes are running the same supported Splunk version.  The underlying OS version does not affect Splunk clustering compatibility, provided both OS versions are supported by the Splunk version in use, although it becomes much more complicated if mis-matching underlying OS e.g. Windows Vs Linux! This is from a technical standpoint though, While mixed OS versions may be supported during migration periods, the recommended long-term state is to standardize all nodes in the cluster on the same, newer supported OS version. You run the risk of being in an un-supported state if you remain in a mixed version state. OS differences alone should not impact search or cluster management functionality, Splunk communicates via supported network protocols, not OS-specific mechanisms.  You have possibly already seen this but its worth reviewing https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/9.2/plan-your-splunk-enterprise-installation/system-requirements-for-use-of-splunk-enterprise-on-premises  I would also recommend that after migrating all nodes to RHEL 8, to revalidate ulimits etc. Avoid running mismatched Splunk software versions across cluster nodes/IDX where possible to avoid different performance across different nodes. If you use custom scripts or apps, validate their dependencies (Python, OS libraries) for OS compatibility.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
I am upgrading from RHEL 7 to RHEL 8 in light of end of support for Red Hat. We have a clustered environment. We have two sites per cluster for each the SH and Indexer Cluster. All splunk servers are... See more...
I am upgrading from RHEL 7 to RHEL 8 in light of end of support for Red Hat. We have a clustered environment. We have two sites per cluster for each the SH and Indexer Cluster. All splunk servers are on 9.2.0.1.     My question is, can we run a RHEL 8 Cluster Master and have a mixed environment of RH 8 and RH 7 servers within the cluster? I know there is a hierarchy for the servers, but i wasn't sure to what extent the OS affected the application.    With the upgrade, I might have:   RHEL 8 Indexer Cluster Manager while Indexers themselves are on RHEL 7. RHEL 8 SH cluster Manager while SH may be on RHEL 7. Depending on how the in-place upgrade goes, determines how many servers I upgrade at once. These are all Azure servers or VmWare servers.    Would any search functionality for any of the search peers be affected by differing OS versions?   Thank you for any clarity. 
@livehybrid , I tried the same query as you suggested, not sure why it is giving me data only for May month | tstats summariesonly=false dc(Message_Log.msg.header.message-id) as Blocked from dat... See more...
@livehybrid , I tried the same query as you suggested, not sure why it is giving me data only for May month | tstats summariesonly=false dc(Message_Log.msg.header.message-id) as Blocked from datamodel=pps_ondemand where (Message_Log.filter.routeDirection="inbound") AND (Message_Log.filter.disposition="discard" OR Message_Log.filter.disposition="reject" OR Message_Log.filter.quarantine.folder="Spam*") earliest=-6mon@mon latest=now by _time | eval Source="Email" | eval MonthNum=strftime(_time, "%Y-%m"), MonthName=strftime(_time, "%b") | stats sum(Blocked) as Blocked by Source MonthNum MonthName | xyseries Source MonthName Blocked | addinfo | table Source [| makeresults count=60 | streamstats count as month_offset | addinfo | eval start_epoch=info_min_time, end_epoch=info_max_time | eval start_month=strftime(start_epoch, "%Y-%m-01") | eval month_epoch = relative_time(strptime(start_month, "%Y-%m-%d"), "+" . (month_offset-1) . "mon") | where month_epoch <= end_epoch | eval month=strftime(month_epoch, "%b") | stats list(month) as search ]  
Also remember that you should not have your environment sized "tightly" - RF and SF should be smaller than your number of indexers. Otherwise your cluster will not be able to rebalance data in case o... See more...
Also remember that you should not have your environment sized "tightly" - RF and SF should be smaller than your number of indexers. Otherwise your cluster will not be able to rebalance data in case of indexer failure.
Ok. Back to square one - what (in terms of business goal of your search, not technical means you're trying to use) is your search supposed to achieve?
Hello, colleagues. I'm using an independent stream forwarder installed on Ubuntu 22.04.05 as a service. After updating to 8.1.5 bytes_in, bytes_out, packets_in, packets_out are always equal to zero... See more...
Hello, colleagues. I'm using an independent stream forwarder installed on Ubuntu 22.04.05 as a service. After updating to 8.1.5 bytes_in, bytes_out, packets_in, packets_out are always equal to zero. If I stop the service and change /opt/streamfwd/bin/streamfwd from 8.1.5 to 8.1.3 and start sert service again, everything is ok.  Anybody run into this? thanks. { [-] app_tag: PANA-L7-PEN : ххххххххх bytes_in: 0 bytes_out: 0 dest_ip: x.x.x.x dest_port: 55438 endtime: 2025-05-28T15:01:26Z event_name: netFlowData exporter_ip: x.x.x.x exporter_time: 2025-May-28 15:01:26 exporter_uptime: 3148584010 flow_end_reason: 3 flow_end_rel: 0 flow_start_rel: 0 fwd_status: 64 input_snmpidx: 168 netflow_elements: [ [+] ] netflow_version: 9 observation_domain_id: 1 output_snmpidx: 127 packets_in: 0 packets_out: 0 protoid: 6 selector_id: 0 seqnumber: 2278842767 src_ip: x.x.x.x src_port: 9997 timestamp: 2025-05-28T15:01:26Z tos: 0 }
Hello, colleagues. I am using independent streamfwd as a service installed on Linux Ubuntu 22.04.05. Streamfwd gets settings from the stream app and gets the indexers list. Everything is ok, streamf... See more...
Hello, colleagues. I am using independent streamfwd as a service installed on Linux Ubuntu 22.04.05. Streamfwd gets settings from the stream app and gets the indexers list. Everything is ok, streamfwd balancing data between all indexers, but if I made a push from the master node to the indexers cluster, and the indexers are rebooting, data balancing breaks after that streamfwd sending data just to one indexer. I can't find how to fix this. Please help thanks
Did you resolve this issue @gazoscreek ? I have same problem when upgrade from 9.2 to 9.4 currently.
Hi Rich, since i am breaking them into separate columns - i used this using if condition | eval TwoXXonly=if(status_code>=200 and status_code <300,1,0) | eval FourXXonly=if(status_code>=400 and s... See more...
Hi Rich, since i am breaking them into separate columns - i used this using if condition | eval TwoXXonly=if(status_code>=200 and status_code <300,1,0) | eval FourXXonly=if(status_code>=400 and status_code <500,1,0) | eval FiveXXonly=if(status_code>=500 and status_code <600,1,0) | stats sum(TwoXXonly) as Total_2xx, sum(FourXXonly) as Total_4xx,sum(FiveXXonly) as Total_5xx by date_only, org,cId,pPath, apie,apiPct,envnt | table list of fieds say for ex; in my data today i dont have 300 events but if they show up tomorrow - do i need to explicitly filter them out as i dont need them at all  - i have not used the status_code in by clause just confused - should i use the filter to explicitly exclude 300 ?
Hello, Hereby a new update on this case Some weeks ago we have upgraded Splunk to 941. After the upgrade we receive erros when executing Splunk commands like splunk show .. or btool Failed to c... See more...
Hello, Hereby a new update on this case Some weeks ago we have upgraded Splunk to 941. After the upgrade we receive erros when executing Splunk commands like splunk show .. or btool Failed to calculate cpu count from cgroup location="V2:/sys/fs/cgroup:/user.slice/user-570057916.slice/session-9.scope:/sys/fs/cgroup:/user.slice/user-570057916.slice/session-9.scope:" Because the Splunk version 941 added support for cgroups V2, we removed the workaround. Since then the original issue is back: adhoc hanging of systemctl If anyone has the issue, i really would like to know. Regards, Harry
I'd use a separate field that contains the status codes of interest.  Something like this | eval status=case(status_code<300 OR status_code>=400, status_code) ``` Other values of status_code set sta... See more...
I'd use a separate field that contains the status codes of interest.  Something like this | eval status=case(status_code<300 OR status_code>=400, status_code) ``` Other values of status_code set status to null``` | stats count by status
Hello @mbjerkeland_spl  Indeed Azure/Microsoft suggests to make use of Virtual Network Flow Logs, as the NSG Flow Logs will soon be deprecated.  We have trouble in extracting fields from the Vi... See more...
Hello @mbjerkeland_spl  Indeed Azure/Microsoft suggests to make use of Virtual Network Flow Logs, as the NSG Flow Logs will soon be deprecated.  We have trouble in extracting fields from the Virtual Network Flow Logs. We tried with props.conf and transforms.conf but no chance to get the fields correctly (for example, only the first source IP is extracted, i.e., src_ip = 50.110.12.0) Is there any way to get the fields correctly? [mscs:vnet:flow] LINE_BREAKER = \}([\r\n]s*,[\r\n]s*){ SEDCMD-remove_header = s/\{\s*\"records\"\:s*\[\s*//g SEDCMD-remove_footer = s/\][\r\n]\s*\}.*//g SHOULD_LINEMERGE = false KV_MODE = json TIME_PREFIX = time\":\" REPORT-tuples = extract_tuple [extract_tuple] FIELDS = time,src_ip,dst_ip,src_port,dst_port,protocol,traffic_flow,traffic_result,traffic_encryption,packets_out,bytes_out,packets_in,bytes_in MV_ADD = 1 SOURCE_KEY = flowRecords.flows{}.flowGroups{}.flowTuples{} Example of log: {"time":"2025-05-28T13:58:52.3816960Z","flowLogGUID":"78er54d6-9e30-493e-84c535-0dae4835a3c3","macAddress":"6044FDGB513","category":"FlowLogFlowEvent","flowLogResourceID":"/SUBSCRIPTIONS/ABC-CDE-EFG-GHI-23XCCX/RESOURCEGROUPS/MYRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKWATCHERS/NETWORKWATCHER_MYREGION/FLOWLOGS/FLOWLOGNAME","targetResourceID":"/subscriptions/fdsfdsf-5534-42449-ddfds33-9718c766ed3f/resourceGroups/rggroup/providers/Microsoft.Network/virtualNetworks/vnet","flowLogVersion":4,"operationName":"FlowLogFlowEvent","flowRecords":{"flows":[{"aclID":"00000000-0000-0000-0000-000000000000","flowGroups":[{"rule":"PlatformRule","flowTuples":["1748440682030,50.110.12.0,10.0.0.1,23456,28701,6,I,B,NX,0,0,0,0"]}]},{"aclID":"fdf34-ff40-4bd1-9803-fdsfd54345","flowGroups":[{"rule":"DefaultRule_AllowInternetOutBound","flowTuples":["1748440669062,10.0.0.1,43.156.12.150,47212,443,6,O,C,NX,15,9521,11,7741","1748440672094,10.0.0.1,43.245.25.152,46512,443,6,O,E,NX,12,2377,10,8287"]}]}]}}
Thank you Rick, exactly what i was looking for..  can i give you another scenario - just guide please i have a field in the same index i dont have to show it in the table but i have to use a case st... See more...
Thank you Rick, exactly what i was looking for..  can i give you another scenario - just guide please i have a field in the same index i dont have to show it in the table but i have to use a case statement to sum or count the number of transactions status_code this will have values like 200, 201, 300, 302, 400,401, 500,502 i only need the count of events for  all 200  all 400 all 500 only  (dont need the one for 300) trying to get this into case statement
You can use the strptime and strftime functions to do that. | eval date=strftime(strptime(<<someField>>, "%a %d %b %Y %H:%M:%S:%3N %Z"), "%m/%d/%Y") where <<someField>> is the name of the field con... See more...
You can use the strptime and strftime functions to do that. | eval date=strftime(strptime(<<someField>>, "%a %d %b %Y %H:%M:%S:%3N %Z"), "%m/%d/%Y") where <<someField>> is the name of the field containing the date value shown.  
Hi ,  I have this scenario where i am getting data from one of the index with 2 other specified filters like index=index_logs_App989 customer="*ABC*" org in ("Provider1","Provider2") i have one f... See more...
Hi ,  I have this scenario where i am getting data from one of the index with 2 other specified filters like index=index_logs_App989 customer="*ABC*" org in ("Provider1","Provider2") i have one filed with the date values as below Tue 27 May 2025 15:26:23:702 EDT  - from this i have to take out the time part and convert it into date like 05/27/2025  - so that i can use this to aggregate at the date or day only ... any guidance please      
Hello, colleagues. After upgrading Splunk Stream 8.1.5 stopped parsing bytes_in, bytes_out, packets_in, packets_out, they are always equal to zero...  { [-] app_tag: PANA-L7-PEN : xxxxxxxxxxxxx b... See more...
Hello, colleagues. After upgrading Splunk Stream 8.1.5 stopped parsing bytes_in, bytes_out, packets_in, packets_out, they are always equal to zero...  { [-] app_tag: PANA-L7-PEN : xxxxxxxxxxxxx bytes_in: 0 bytes_out: 0 dest_ip: x.x.x.x dest_port: xxx endtime: 2025-05-28T15:01:26Z event_name: netFlowData exporter_ip: x.x.x.x exporter_time: 2025-May-28 15:01:26 exporter_uptime: 3148584010 flow_end_reason: 3 flow_end_rel: 0 flow_start_rel: 0 fwd_status: xx input_snmpidx: xx netflow_elements: [ [+] ] netflow_version: 9 observation_domain_id: 1 output_snmpidx: xxx packets_in: 0 packets_out: 0 protoid: 6 selector_id: 0 seqnumber: 2278842767 src_ip: x.x.x.x src_port: 9997 timestamp: 2025-05-28T15:01:26Z tos: 0 } I am using an independent streamforwarder with streamfwd installed as a service on linux ubuntu 22.04.5 If I stop the service and replace the streamfwd file with the old version 8.1.3 and start the service again, everything is ok Anybody run into this?  Thanks!
I’m not sure if this helps, but changes made on the search head cluster captain do appear in the bundle under /opt/splunk/var/run.
It seems that you are not using {0} in your query input. Also can you post the sanitized code for the code block and the full entry for the data path of the 0 input?
As the error message describes, you are trying to delete a playbook from a read-only repository. If you are importing it directly from the Splunk security content github repo, then you cannot delete ... See more...
As the error message describes, you are trying to delete a playbook from a read-only repository. If you are importing it directly from the Splunk security content github repo, then you cannot delete the playbook and would be better off removing the repo in your Source Control settings. If it is cloned to a repo you control, then you need to uncheck the "read only" setting for that repo.