About jordanking1992

jordanking1992

We recently started working with metrics data. The application is sending metrics events with the dimensions: component, deployment_id, timestamp_seconds_from_epoch and metrics names: change_committed, release We are trying to calculate the duration between deployment_id's from the time a change was committed (change_committed) and released (release) based on timestamp_seconds_from_epoch (which is a timestamp in epoch time) We thought the transaction command would be helpful but since we arent leverage _time and instead using a custom time field called timestamp_in_epoch_time, we are having some trouble figuring out the best approach. Here is what we currently have: | mstats avg("change_committed") as change_committed prestats-true WHERE "index"="statsd" span=auto BY deployment_id | table _time deployment_id | append [ | mstats avg("release") as release prestats-true WHERE "index"="statsd" span=auto BY deployment_id | table _time deployment_id Example metrics events: change_committed:1,timestamp_seconds_from_epoch:1651096172,deployment_id:28020 release:1,timestamp_seconds_from_epoch:1651097000,deployment_id:28020 How can we track the duration of timestamp_seconds_from_epoch between a change_committed and release event for each deployment_id?

jordanking1992 · ‎03-22-2022

Hello, Working with a team that is sending some custom paramters via metrics data. They are trying to include a dimension that contains a data, but Splunk is not accepting of the date. release:1,component:test,team:TestTeam,repo_branch:master,version:3,eventTimestamp:2022-03-22T14:46:41.048881800 My guess is that Splunk doesn't like the colon's in the timestamp but a bit unsure. The team wants to be able to send time within the metrics for later analysis using eval commands after indexing. Is there a best practice for including a time dimension/value within metrics data? (i.e epoch/UNIX time)

jordanking1992 · ‎09-20-2021

No worries. Thanks again. Will look into post-processing of dashboard panels. Thanks!

jordanking1992 · ‎09-20-2021

Hey richgalloway, Appreciate the recommendations. Can you elaborate more on how adding more SH's will reduce the indexers CPU usage? We currently have a 3 node search head cluster tied to a 11 node indexer cluster.

jordanking1992 · ‎09-20-2021

Hello All, Our environment consists of an indexer cluster scaled for 1 TB of data per day. On average, we have about 30 users logged running ad-hoc searches and about 40 scheduled searches running along side those queries. For 360 days of the year, the average CPU of our indexer cluster is no higher than 25%. But for 1 week of the year during the thanksgiving time range, we have about 65 users logged in, running ad-hoc queries, and loading multiple dashboards to monitor sales data during this time of the year. During this week of the year, the CPU on our indexers stays consistently at 90%-100% which we have attributed to many users loading dashboards with many panels simultaneously along with normal ad-hoc and scheduled searching. My question is, what recommendations are our there for combating this increased usage and prevent the CPU from being pegged at 100% for 1 week of the year? We are thinking about limiting the amount of searches each user is allowed to run concurrently but fear that many users will complain that their searches are queued. Any suggestions are much appreciated. Respectfully,

jordanking1992 · ‎04-20-2021

My concern is that if enough indexes are storing the data longer than the expected retention, do we rely on maxVolumeSize to start deleting events if the disk starts to fill up?

jordanking1992 · ‎04-19-2021

Does this mean that eventually the bucket will move to frozen when the "newest" event is more than the the frozentimeperiodinsec setting? Is there a way to prevent this behavior so that all indexes have the data age of the frozentimeperiodinsecs setting? Thanks, J

jordanking1992 · ‎02-16-2021

Thanks richgalloway. My goal is upgrade the indexer cluster without the end user seeing warnings when a peer goes down for the upgrade. Since removing the quarantine task, things are a little better,however, I am still occasionally get the "connection refused for peer=x" when the peer goes down via 'splunk offline' and a search was ran at the same time. Is it nearly impossible to perform an indexer cluster upgrade without a few "connection refused" warnings when a search is ran during a peer being down?

jordanking1992 · ‎02-16-2021

Hello, It seems that my current process of quarantining a search peer and then running 'splunk offline' causes searches to become zombified. "This search has encountered a fatal error and has been marked as zombied." Is it best practice to quarantine the peer before or after running the splunk offline command? I know that running 'splunk offline' graceful haults new searches from reaching that indexer but for some reason, I think there is an interference when quarantining the host first and then running 'splunk offline'. Thoughts?

jordanking1992 · ‎02-16-2021

Hey manjunathmeti, Have you used the 'target = log' or 'target = none' settings? Although the message.conf.spec file says they are keys that can be used, there are no examples of them used in any of the messages.conf default. I have the following in my messages.conf on all three search heads under '/opt/splunk/etc/system/local/messages.conf' and still get the warning messages on my search heads when a quarantined indexer restarts. [DISPATCHCOMM:EXCLUDED_QUARANTINED_PEERS] message = One or more peers has been excluded from the search because they have been quarantined. Use "splunk_server=*" to search these peers. This might affect search performance. severity = info target = log Any ideas? J

jordanking1992 · ‎02-13-2021

Disregard. To silence that message, you can set the target option to "none" in messages.conf. Thank you for the solution to my question. Respectfully, J

jordanking1992 · ‎02-13-2021

Hey manjunathmeti, Thanks for the recommendation, however, it looks like quarantining the peers helped quiet the errors above but now the user sees the following: One or more peers have been excluded from the search because they have been quarantined. Use "splunk_server=" to search the peers. This may affect search performance Do you know of anyway to silence this message?

jordanking1992 · ‎02-12-2021

Hello, I have an automated upgrade plan that does the following: Puts the cluster in maintenance mode splunk enable maintenance-mode Goes 1 by 1 on each of the 3 indexer peers and runs: splunk offline Extracts upgrade tar file to necessary location runs splunk start and accepts license and answers yes. repeats for the next peer Disables maintenance mode. I am trying to upgrade the peers without the end users seeing messages but unfortunately users see things like the following: Unable to distribute to peer named X because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. ^ Even though the peer is Up according to the Cluster Master Connection Refused for peer=X ^ Which seems like the search heads are sending queries or still have an established connection with the peer. I would expect the search head to know that a peer is down and not communicate with it till it indexes have been validates and deemed searchable. Anyone have recommendation on making the indexer upgrade as seamless to the end user as possible? Things tried: adjusted the restart_timeout, quiet_period, and decomission_node_force_timeout on the cluster master Thanks, J

jordanking1992 · ‎08-10-2020

All, I am trying to send to from an external forwarder to a DMZ Heavy Forwarder that is behind a firewall w/ 9997 open. In my attempt to send data, I am getting "TCP output processor has paused the data flow" and unable to get to get results to the DMZ forwarder. I have similar setup with HEC and Syslog that works just fine but for traffic on 9997, Splunk is refusing to send. The box is ping-able and 9997 is open. The firewall is set up to route to the DMZ host as well. Anyone have some ideas on what I may be doing wrong or things to consider when setting up a Splunk HF in a DMZ?

jordanking1992 · ‎01-17-2020

I am facing the same issue. Any solution?

jordanking1992 · ‎12-04-2019

Thank you so much. No more headaches!

jordanking1992 · ‎12-04-2019

We are trying to identify how much of our data is impacted by the latest timestamp bug. I was wondering if there was a query that used regex to search _raw for events that have 2 digit years. This will greatly help us analyze the risk of doing a last minute production upgrade...

jordanking1992 · ‎12-04-2019

Is there a query that can search the _raw events for sources that are using 2 digit years?

jordanking1992 · ‎12-04-2019

Thanks for the response. This is what i was suspecting. The reason for posting is because this time of year we have excessive use of Splunk. Teams put up all there dashboards for Holiday monitoring and we are getting this message. What you described with dashboards and panels is exactly whats happening hear. As an admin, how do you manage multiple users loading the same dashboard with 12 searches (thus triggering many concurrent searches)? I explain to the users that this is why searches are queued but they do not seem to understand it haha.

jordanking1992 · ‎12-04-2019

We are receiving the message "This instance is approaching the max concurrency searches" on our search head. Usually if we are hitting concurrency for a role, it will mention "...max concurrency for this role.". From what i've read, when the instance hits max concurrency, it has to do with the system wide concurrency setting Splunk calculates based on CPUs. Does the max_search_per_cpu need to be applied to the limits.conf of the indexers or the search heads? Thanks,

Posts	44
Solutions	0
Karma Given	10
Karma Received	4
Member Since	‎03-07-2018

Online Status	Offline
Date Last Visited	Friday

How to Calculate time difference with _metrics eve...

Is there a best practice for using a time dimensio...

Many Users Running Dashboards Simultaneously Cause...

Quarantine a Peer before or after running splunk o...

Upgrade Indexer Cluster w/ out search Warnings on ...

External Forwarder sending to DMZ Heavy Forwarder ...

Identify Two Digit Year Timestamp Events Query (20...

Is the max_searches_per_cpu setting based on CPU's...

Indexer _internal size under /opt/splunk/var/lib i...

Dashboard with multiple searches and multiple user...

How to Calculate time difference with _metrics eve...

Is there a best practice for using a time dimensio...

Re: Many Users Running Dashboards Simultaneously C...

Re: Many Users Running Dashboards Simultaneously C...

Many Users Running Dashboards Simultaneously Cause...

Re: Data age in splunk

Re: Data age in splunk

Re: Quarantine a Peer before or after running splu...

Quarantine a Peer before or after running splunk o...

Re: Upgrade Indexer Cluster w/ out search Warnings...

Re: Upgrade Indexer Cluster w/ out search Warnings...

Re: Upgrade Indexer Cluster w/ out search Warnings...

Upgrade Indexer Cluster w/ out search Warnings on ...

External Forwarder sending to DMZ Heavy Forwarder ...

Re: splunk addon for aws does not list s3 buckets

Re: Identify Two Digit Year Timestamp Events Query...

Identify Two Digit Year Timestamp Events Query (20...

Re: How can we verify the patch that has been prov...

Re: Is the max_searches_per_cpu setting based on C...

Is the max_searches_per_cpu setting based on CPU's...