×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question
PT_crusher
Explorer
Member since: ‎07-15-2021
‎10-15-2024
Community Statistics
Posts 7
Solutions 1
Karma Given 0
Karma Received 0
Member Since ‎07-15-2021
Activity Feed
View All

Re: transform ingested HEC JSON log as regular log

by in Splunk Enterprise
‎10-15-2024 01:08 PM
‎10-15-2024 01:08 PM
raw endpoint is not an option because it is not supported by the logging-operator ... View more

transform ingested HEC JSON log as regular log

by in Splunk Enterprise
‎10-15-2024 08:28 AM
‎10-15-2024 08:28 AM
Looking for props.conf / transforms.conf configuration guidance. The aim is to search logs from a HTTP Event Collector the same way we search for regular logs. Don't want to search JSON in the search heads. We're in the process of migrating from Splunk Forwarders to logging-operator in k8s. Thing is, Splunk Forwarder uses log files and standard indexer discovery whereas logging-operator uses stdout/stderr and must output to an HEC endpoint, meaning the logs arrive as JSON at the heavy forwarder. We want to use Splunk the same way we did over the years and want to avoid adapting alerts/dashboards etc to the new JSON source OLD CONFIG AIMED TO THE INDEXERS (using the following config we get environment/site/node/team/pod as search-time extraction fields)   [vm.container.meta] # source: /data/nodes/env1/site1/host1/logs/team1/env1/pod_name/localhost_access_log.log CLEAN_KEYS = 0 REGEX = \/.*\/.*\/(.*)\/(.*)\/(.*)\/.*\/(.*)\/.*\/(.*)\/ FORMAT = environment::$1 site::$2 node::$3 team::$4 pod::$5 SOURCE_KEY = MetaData:Source WRITE_META = true   SAMPLE LOG USING logging-operator   { "log": "ts=2024-10-15T15:22:44.548Z caller=scrape.go:1353 level=debug component=\"scrape manager\" scrape_pool=kubernetes-pods target=http://1.1.1.1:8050/_api/metrics msg=\"Scrape failed\" err=\"Get \\\"http://1.1.1.1:8050/_api/metrics\\\": dial tcp 1.1.1.1:8050: connect: connection refused\"\n", "stream": "stderr", "time": "2024-10-15T15:22:44.548801729Z", "environment": "env1", "node": "host1", "pod": "pod_name", "site": "site1", "team": "team1" }   ... View more

Low Raw To Index Ratio (_audit)

by in Splunk Enterprise
‎09-23-2021 05:12 AM
‎09-23-2021 05:12 AM
We were investigating some indexes that have low RAW to Index Ratio and came across _audit whose RAW to Index Ratio is 0.81:1. At first glance, _audit seemed a good candidate to learn how to find out if an index has high cardinality and what we can do about it (like tsidx reduction along with bloomfilters). First is not frequently searched to test tsidx reduction or bloomfilters, moreover, it is an index that everyone has in their Splunk installations so, we could benefit from common knowledge.  We came accross the following numbers about cardinality by taking a sample of the data and using tstats and walklex: earliest latest number of events keywords in lexicon min number of events per keyword keywords with min number of events percentage of keywords with min number of events 18/09/2021 00:00 22/09/2021 24:00 5721945 6764698 10 176669 2,61%   Just by looking at the above table it is hard to tell if we are in front of an index whose data changes a lot or not. What is considered a high cardinality index? It would be awesome to have some reference numbers but i was not able to find them anywhere. Q1: Do we have any reference numbers that once compared to, would unequivocally tell us either or not the bucket is an  high cardinality one? Nonetheless, should we expect Raw to Index Ration to drop bellow 1:1? Then we went through and inspected the size of the tsidx files against the size of the buckets indexer vm bucketsize_bytes tsidxsize_bytes bucketsize_megabytes tsidxsize_megabytes A 1317675655 933241652 1257 890 B 1321231309 892793498 1260 851 C 1464189620 1003103122 1396 957 D 1538519922 1045951037 1467 997 E 901792050 609003289 860 581 F 1417458990 929185810 1352 886 G 1591446741 1167724482 1518 1114 H 1497574135 1009380670 1428 963 totals 11049888422 7590383560     Results show that the tsidx files take around ~69% of the overall disk space needed to store the _audit index in the indexers Q2: Once again, is this a sign of high cardinality? Q3: Lastly, any SEGMENTATION config that is commonly applied to _audit index?    ... View more

Re: Correct way of testing forwarder_site_failover...

by in Splunk Enterprise
‎08-03-2021 09:05 AM
‎08-03-2021 09:05 AM
Turns out the it is up to the clustermaster to loose connectivity to the indexers. Clustermaster will them instruct the forwarder to failover. Hope it helps ... View more

Re: Correct way of testing forwarder_site_failover...

by in Splunk Enterprise
‎07-16-2021 10:13 AM
‎07-16-2021 10:13 AM
07-16-2021 17:07:58.143 +0000 DEBUG IndexerDiscoveryHeartbeatThread - Connector::runCookedStateMachine in state=eV3ConnectDone for <site2 indexer>:9997 07-16-2021 17:07:58.143 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState Ping failover connection to idx=<site2 indexer>:9997 successful. 07-16-2021 17:07:58.143 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState enable the failover to 4 07-16-2021 17:07:58.944 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is enabled to fail over 07-16-2021 17:07:58.944 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState current connection status 10 07-16-2021 17:07:58.945 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is not failing over to the failover site because the primary site is up 07-16-2021 17:08:01.850 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is enabled to fail over 07-16-2021 17:08:01.850 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState current connection status 10 07-16-2021 17:08:01.850 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is not failing over to the failover site because the primary site is up 07-16-2021 17:08:04.754 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is enabled to fail over 07-16-2021 17:08:04.755 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState current connection status 10 07-16-2021 17:08:04.755 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is not failing over to the failover site because the primary site is up 07-16-2021 17:08:07.611 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is enabled to fail over 07-16-2021 17:08:07.611 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState current connection status 10 07-16-2021 17:08:07.611 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is not failing over to the failover site because the primary site is up 07-16-2021 17:08:07.913 +0000 WARN TcpOutputProc - Cooked connection to ip=<site1 indexer>:9997 timed out 07-16-2021 17:08:10.518 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is enabled to fail over 07-16-2021 17:08:10.518 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState current connection status 10 07-16-2021 17:08:10.518 +0000 DEBUG IndexerDiscoveryHeartbeatThread - TcpOutputGroupState is not failing over to the failover site because the primary site is up   After setting IndexerDiscoveryHeartbeatThread to DEBUG i confirmed my suspicious. Forwarder is not failingover because it thinks site1 is up netstat -tunap | grep "<ip of the forwarder VM>" <------- no ESTABLISHED connections in any of site1 indexers ... View more

Re: Correct way of testing forwarder_site_failover...

by in Splunk Enterprise
‎07-16-2021 08:00 AM
‎07-16-2021 08:00 AM
So I followed the link and explicitly set the polling_rate to 10 in the cluster master. This would give me a polling cycle interval of roughly 4 minutes. Regardless of this fact I just see the logs getting filled with  07-16-2021 15:55:33.324 +0100 WARN TcpOutputProc - Cooked connection to ip=<multiple site1 indexer IPs>:9997 timed out If i bump the TcpOutputProc logs to DEBUG the state machine is transitioning between  Connector::runCookedStateMachine in state=eInit and Connector::runCookedStateMachine in state=eV3ConnectTimedOut but still, all attempts are made to site1, i don't feel that the failover is triggered at all. If we change the server.conf of the forwarder and set the site to site2 it is able to send the logs via site2 indexers 😕 ... View more

Correct way of testing forwarder_site_failover fea...

by in Splunk Enterprise
‎07-15-2021 09:52 AM
‎07-15-2021 09:52 AM
We have a multi-site installation of Splunk and would like to test if the forwarder_site_failover is working properly. In the forwarders output.conf we have the following       [indexer_discovery:master1] pass4SymmKey = $secretstuff$ master_uri = https://yadayada:8089 [tcpout:group1] indexerDiscovery = master1 useACK = false clientCert = /opt/splunk/etc/auth/certs/s2s.pem sslRootCAPath = /opt/splunk/etc/auth/certs/ca.crt [tcpout] forceTimebasedAutoLB = true autoLBFrequency = 30 defaultGroup = group1       As far as the yadayada clustermaster server goes, we have the following config:       /opt/splunk/etc/apps/clustermaster_base_conf/default/server.conf [clustering] (...) /opt/splunk/etc/apps/clustermaster_base_conf/default/server.conf forwarder_site_failover = site1:site2       One thing that I was trying to figure out was the need to explicitly set site2:site1 or if the existing configuration was enough for failing over from site1 to site2 and from site2 to site1. My approach was to shut the connection between the forwarder and the site1 indexers by setting iptable rules in the indexers that DROP the connections from the forwarder.       #e.g. iptables rule iptables -I INPUT 1 -s <forwarder ip> -p tcp --dport 9997 -j DROP #forwarder splunkd.log 07-15-2021 16:20:41.729 +0000 WARN TcpOutputProc - Cooked connection to ip=<site1 indexer ip>:9997 timed out       The iptables rules didn't make the forwarder failover so, i wonder if the failover process only kicks when the clustermaster loses the visibility over the indexers. In the live setup this seems more risky and less flexible. What is the recommended approach to perform this kind of testing?   ... View more
Contact Me
Online Status
Offline
Date Last Visited
‎10-15-2024 05:03 PM