About mitag

mitag · ‎05-06-2020

Existential question here... 🙂 What is the appropriate mechanism in Splunk to have multiple (potentially hundreds) of alerts that are based on the latest events, rather than real-time or timeframe searches, while keeping our Splunk deployment sane and simple? (Is it even possible?) Example: I need an alert when a volume (disk) breaches an 80% used space threshold, and need it within 30 seconds of when Splunk gets an event. (Then similar alerts for NAS and SAN volumes, CPU, memory, interface utilization, and a whole bunch of other metrics.) Setting up a few dozen of these realtime searches and respective alerts brings our cluster to its knees. Attempting to set up auto-refreshing and fast dashboards with the same metrics simply knocks it out. Whereas doing something like this in Solarwinds or Datadog - piece of cake, including statistics-based metrics (e.g. if a metric exceeds a 30-minute baseline by more than 20% over the last 3 minutes). Is Splunk not the right product for the task? If it is, what is the technical term for my problem, how can it be solved in Splunk, and would you be so kind as to point me to where it's discussed? Thanks! P.S. Please read the question fully and please abstain from attempting to answer what you might think the question is. The question is specific enough: What is the appropriate mechanism in Splunk to have multiple (potentially hundreds) of alerts that are based on the latest events, rather than real-time or timeframe searches, while keeping our Splunk deployment sane and simple? To rephrase it a bit: in an environment where the ability to set up fast alerts on a large number of metrics is important, is Splunk (OOTB) the right product for the task? If so, what are the mechanisms to accomplish that? (Because OOTB, Splunk is not suitable for it - at least not in my experience.) P.P.S. Is metrics such a mechanism? If so what would be a good resource to get those set up and running fast? (Digging through the documentation tells me it's not a streamlined, easy experience.)

mitag · ‎05-05-2020

Thank you - appreciate the kind words. Doesn't seem to be working though (probably something simple). (Can't seem to post an image... Here is the link to the two screenshots. Hopefully this works.)

mitag · ‎05-05-2020

Does this look right? (Feels weird - as if I am doing two very similar transforms one after another - i.e. doesn't feel efficient.) sourcetype=WinHostMon source="disk" [ search sourcetype=WinHostMon source="disk" | stats min(storage_used_percent) as min, avg(storage_used_percent) as avg, max(storage_used_percent) as max, by host, Name FileSystem DriveType | eval delta = max - avg | where delta>20 | sort - max delta avg | table host Name ] | timechart max(storage_used_percent) by host

mitag · ‎05-05-2020

Sorry for the delay! The sourcetype is the standard sourcetype=WinHostMon . Searching for Type=Disk or source=disk would give you disk stats. Events look like this: Type=Disk Name="C:" DriveType="fixed" TotalSpaceKB=116859900 FreeSpaceKB=62318744 FileSystem="NTFS" (host = ws2016_016 source = disk sourcetype = WinHostMon ) (If you'd like, I can send you a sample of raw events.) They are sampled every 5-15 minutes. Some additional fields are calculated - e.g. for the above single event these fields are: storage 114120.99609375 storage_free 60858.1484375 storage_free_percent 53.32774031126161 storage_used 53262.84765625 storage_used_percent 46.67225968873839 My specific case is this: on several of our hosts, the boot disk ("C:") went full (from about 45% to 100% within minutes, then after 15-45 minutes - back to normal). I need to do a report that only shows those hosts and volumes that had a spike, and plot those spikes over time. We could of course just search for all hosts with volumes close to full (say, over 90%) - but that does not isolate the spikes correctly as some volumes have been close to full for a while. So I am thinking: calculate min, average and max storage_used_percent for each volume, calculate the delta (difference) between max and avg for each volume / host; List hosts and volumes where that delta is over a threshold (say, 20%) run a timechart command just on those volumes and hosts. With the following search I am getting a list of hosts and volumes that had a spike: sourcetype=WinHostMon source=disk FileSystem!="SNFS" | stats min(storage_used_percent) as min avg(storage_used_percent) as avg max(storage_used_percent) as max by host, Name FileSystem DriveType | eval delta = max - avg | where delta>20 | sort - max delta avg | fields Name host Now, how do I pipe the results into a timechart (or any other plotting mechanism)? Thanks!

mitag · ‎04-29-2020

Need a report that: Lists volumes with significant disk usage spikes over a given timeframe. Plots those disk usage spikes over time. P.S. Not interested in volumes with high percentage of used disk space - only in those that had a spike of say more than 20%. I am assuming I'd need to: List volumes that had such a spike by calculating max and average values for e.g. UsePct for a volume and then leaving only those with the delta > 20; Run a timechart or something similar on those volumes. Blanking out on how to do that and would appreciate your help - thanks! P.P.S. This is as far as I've gotten - and it seems to correctly ID volumes with usage spikes (updated May 5): sourcetype=WinHostMon source=disk FileSystem!="SNFS" | stats min(storage_used_percent) as min, avg(storage_used_percent) as avg, max(storage_used_percent) as max, by host, Name FileSystem DriveType | eval delta = max - avg | where delta>20 | sort - max delta avg The above produces the full stats table for all hosts and their volumes that had a spike; adding | fields host Name to it would produce just the hosts and volume names; the question remains: what is the best way to plot storage_used_percent on those volumes over the timeframe of the search? P.P.P.S. Bonus points for streamlining the above search and making it faster; generally a streamlined mechanism for pinpointing anomalies (spikes, unusual deviations or volatility) on any available metrics - such as CPU, memory, disk and network utilization. (I have yet to properly configure Splunk infrastructure apps - perhaps such mechanisms are included in those.)

mitag · ‎04-27-2020

Thank you - this works, and I learned a couple of things! P.S. A simplified version of your answer: Base Search: sourcetype="tomcat:vantage" | eval "Field Extraction Error(s)" = if(isnull(message),"Present","Not Present") | stats sparkline count by "Field Extraction Error(s)" | addcoltotals labelfield="Field Extraction Error(s)" label="Total Events" Post-process searches: | search "Field Extraction Error(s)" IN ("Present") | table count | table "Field Extraction Error(s)" sparkline count | search "Field Extraction Error(s)" IN ("Total Events") | table count | search "Field Extraction Error(s)" IN ("Not Present") | table count

mitag · ‎04-24-2020

This is a follow up to my "simplifying a (field extraction error) dashboard?" question earlier today, and the new question is: How do I structure my base and post-process searches to produce single value visualizations for the three calculated stats values: (1) count of events with errors, (2) w/o errors, and (3) the total - in addition to this? sourcetype="tomcat:vantage" | eval "Field Extraction Error(s)" = if(isnull(message),"1","0") | stats sparkline count by "Field Extraction Error(s)" In other words the result should be something like this, where the single value visualizations are produced using base and post-process searches, as opposed to separate ones: Thanks!

mitag · ‎04-24-2020

Possible to use the results of the same search in multiple panels on the same dashboard, and with different visualizations for them? (By the "same search" I mean: run it once, present results in several places via different means.) Reason: make it faster, use less resources. Example: All four panels of the above dashboard use basically the same search that checks if a field message was extracted, and reports the stats highlighting the number of events where that field is not present. Notes: - Field message should be present in all events; if it's not - it's a field extraction error. - The error is not necessarily the result of a bad field extraction regex - it could also be the result of a malformed event, event breaking too soon, etc. - The top right panel is all that is needed - yet the other panels do help - I'd like to keep them there - although not at the expense of running multiple redundant searches. The search: sourcetype="some_sourcetype" | eval "Field Extraction Error(s)" = if(isnull(message),"present","not present") | stats sparkline count by "Field Extraction Error(s)" Possible? Thanks!

mitag · ‎04-16-2020

Put the master in maintenance, rolling reboot of all indexers; checking bucket status between reboots to ensure integrity. That said - I can't remember now if I also tried manual rolling service restarts. (Automatic rolling restart from the master wasn't available due to that degraded index.)

mitag · ‎04-14-2020

Maybe not in Splunk docs or CIM repo - but there must be other people logging java applications using log4j with a similar log structure - and extracting fields. The question wasn't, "does Splunk offer a CIM for log4j?", it was "LOG4J CIM?" with the implication of, "how do I zero in on the best possible approximation of such a CIM?". If you search the interwebs for pieces and bits of the logs I posted - you'll see a bunch of people using similar log structures. It's those people using Splunk that I am hoping to engage with this question, not those who would say "Splunk can't help you" or "I can't help you". Thank you for the understanding 🙂

mitag · ‎04-14-2020

many thanks - but not finding much there... E.g. not seeing anything relevant to threads, java classes, namespaces, java traces... Appreciate your looking into it.

mitag · ‎04-14-2020

Thanks - this isn't it. The logs are already in Splunk - forwarded via SUFs - so I don't need a TCP input to get them into Splunk. What I need to extract fields and name them in line with common standards - i.e. using a CIM for log4j if one exists - or using field names that others are using. I probably didn't write my question very clearly - will see if I can revise it. Thanks for looking into it!

mitag · ‎04-14-2020

Thanks Pavel! This part of CIM is what I am looking for: The CIM helps you to normalize your data to match a common standard, using the same field names and event tags for equivalent events from different sources or vendors. I.e. field names that are in line with common standards - or at least with what others do with log4j events. So yes, it is a CIM that I am looking for and not a CIM parser - even if field names are just a small part of what a CIM is. (I probably didn't write my question very clearly - revised it - hope it's clearer now.) Thanks for looking into it!

mitag · ‎04-13-2020

A number of applications and services in our environment use LOG4J for logging. Is there a CIM (Common Information Model) for LOG4J log types - or perhaps just the accepted / standardized field names? (The idea is to properly set up field extraction correctly the 1st time so we don't have to do it again in the future.) In other words what I need first and foremost are standardized field names for this log type - and if there is anything else that needs to be done to have a clean and performant field extraction for this log type that'll last a while w/o needing major revisions that might wreak havoc on existing dashboards, reports and searches. Event examples: 2020-04-13 15:20:53,379 ERROR [com.somejavaapp.exec.Server] (pool-1-thread-1) - Caught exception producing output java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) Show all 15 lines ... 2020-04-13 15:20:53,379 ERROR [com.somejavaapp.exec.Server] (Thread-149821) - Exception while sending progress data java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) Show all 8 lines ... etc... So perhaps the field names should be as follows? _time 2020-04-13 15:20:53,379 severity? log_level? ERROR java_class? [com.somejavaapp.exec.Server] java_class_package? com.somejavaapp.exec java_class_package_namespace? com.somejavaapp thread? (pool-1-thread-1) (Thread-149821) message? Caught exception producing output exception? java.net.SocketException: Connection reset by peer: socket write error java_traces? at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) Thanks!

mitag · ‎04-09-2020

same story as @hlouwes: "The lookup table 'windowsapps' is invalid." when running the search inside the app. (Job settings show "App: `splunkappwindowsinfrastructure`".)

mitag · ‎04-09-2020

Cool! (Please mark my answer "accepted" if it worked for you.)

mitag · ‎04-09-2020

My understanding is that visualizations plot numerical values. For something to show up in your visualization, define what you need to visualize, and extract - or generate - numerical values representing that.

mitag · ‎04-09-2020

Restarting all indexers (not just the one with errors) was what resolved the issue for me. The root cause is still a mystery. Suspecting a bug or a misconfiguration: that degraded _metrics index you see in the screenshot above is not supposed to exist - we aren't collecting any metrics to my knowledge, and searching for them via e.g. | mcatalog values(_dims) WHERE index=* produces no results. _metrics index did seem to exist prior to this issue - but does not exist now in our environment; it's unclear why it was created originally.

mitag · ‎04-03-2020

Done it before, done it again just now - didn't help.

mitag · ‎04-02-2020

After updating a bucket replication policy and doing a rolling restart of cluster indexers, one of the indexers seems stuck in this state: Question: where do I go, what do I do, to figure out what's the root cause and how to fix it? Cluster status in plaintext: - Search Factor Not Met - Replication Factor Not Met - One of three indexers: Fully Searchable: No, Status: Pending. - One out of 12 indexes shows with Searchable and Replicated Data Copies (the rest seem fine) Under "Indexer Clustering: Service Activity", "Snapshots" - a number of "pending" tasks that seem to be stuck and never moving to "in progress" status: - "Fixup Tasks - In Progress (0)" - "Fixup Tasks - Pending": -- Tasks to Meet Search Factor (4) -- Tasks to Meet Replication Factor (6) -- Tasks to Meet Generation (6) Tasks to Meet Search Factor (4) Bucket Index Trigger Condition Trigger Time Current State _metrics~34~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics does not meet: sf & rf Waiting 'target_wait_time' before search factor fixup _metrics~34~64AE7236-EE5E-4EEE-AEBF-203F149FCB61 _metrics does not meet: primality & sf & rf Waiting 'target_wait_time' before search factor fixup _metrics~35~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics does not meet: sf & rf Waiting 'target_wait_time' before search factor fixup _metrics~35~64AE7236-EE5E-4EEE-AEBF-203F149FCB61 _metrics does not meet: sf & rf Waiting 'target_wait_time' before search factor fixup Tasks to Meet Replication Factor (6) Bucket Index Trigger Condition Trigger Time Current State _metrics~34~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics does not meet: sf & rf Waiting 'target_wait_time' before replicating bucket _metrics~34~64AE7236-EE5E-4EEE-AEBF-203F149FCB61 _metrics does not meet: primality & sf & rf Waiting 'target_wait_time' before replicating bucket _metrics~35~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics does not meet: sf & rf Waiting 'target_wait_time' before replicating bucket _metrics~35~64AE7236-EE5E-4EEE-AEBF-203F149FCB61 _metrics does not meet: sf & rf Waiting 'target_wait_time' before replicating bucket _metrics~36~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics non-streaming failure - src=64AE7236-EE5E-4EEE-AEBF-203F149FCB61 tgt=4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 failing=tgt _metrics~37~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 _metrics non-streaming failure - src=9B5D3504-81B2-4DCC-BF4D-F7ED811A3571 tgt=4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 failing=tgt ... etc. Some of the errors on the indexer(s): 04-03-2020 07:55:10.100 -0700 ERROR TcpInputProc - event=replicationData status=failed err="Close failed" host = bvl-mit-splkin2source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 04-03-2020 07:55:10.100 -0700 WARN BucketReplicator - Failed to replicate warm bucket bid=_metrics~37~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 to guid=4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 host=10.101.128.89 s2sport=9887. Connection closed. host = bvl-mit-splkin1source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 04-03-2020 07:55:10.097 -0700 WARN S2SFileReceiver - event=processFileSlice bid=_metrics~37~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 msg='aborting on local error' host = bvl-mit-splkin2source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 04-03-2020 07:55:10.097 -0700 ERROR S2SFileReceiver - event=onFileClosed replicationType=eJournalReplication bid=_metrics~37~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 state=eComplete src=64AE7236-EE5E-4EEE-AEBF-203F149FCB61 bucketType=warm status=failed err="bucket is already registered, registered not as a streaming hot target (SPL-90606)" host = bvl-mit-splkin2source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 04-03-2020 07:55:10.089 -0700 WARN BucketReplicator - Failed to replicate warm bucket bid=_metrics~36~4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 to guid=4C2AF0DE-E42F-489B-92FB-2CA3FC68AC85 host=10.101.128.89 s2sport=9887. Connection closed. host = bvl-mit-splkin3source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd Additional notes: Output of /opt/splunk/bin/splunk list peer-info on the peer: slave base_generation_id:651 is_registered:1 last_heartbeat_attempt:0 maintenance_mode:0 registered_summary_state:3 restart_state:NoRestart site:default status:Up /opt/splunk/etc/master-apps/_cluster/local/indexes.conf on CM (successfully replicated to peers via /opt/splunk/bin/splunk apply cluster-bundle 😞 [default] repFactor = auto [_introspection] repFactor = 0 [windows] frozenTimePeriodInSecs = 31536000 coldToFrozenDir = $SPLUNK_DB/$_index_name/frozendb [wineventlog] frozenTimePeriodInSecs = 31536000 coldToFrozenDir = $SPLUNK_DB/$_index_name/frozendb Details: Splunk Enterprise 8.02 mostly default settings Thank you!

Posts	100
Solutions	5
Karma Given	120
Karma Received	6
Member Since	‎01-04-2019

Online Status	Offline
Date Last Visited	a week ago

event categorization with rex: best practices?

limited formatting options when using fieldformat?

deprecated 'stats' command syntax notification on ...

sparkline length (width) out of control?

windows command line parsing (aspera ascp.exe)

"This search uses deprecated 'stats' command synta...

extracting fields from "event_message" "context: "...

default '/opt/splunk/etc/deployment-apps/Splunk_TA...

Choosing "settings" - "indexer clustering" takes m...

Suitability of Splunk for real time monitoring and...

Suitability of Splunk for real time monitoring and...

Re: report on disk usage spikes?

Re: report on disk usage spikes?

Re: report on disk usage spikes?

report on disk usage spikes?

Re: post-process search to get a single value?

post-process search to get a single value?

simplifying a (field extraction error) dashboard?

Re: splunk indexer: status: "pending", fully searc...

Re: LOG4J CIM?

Re: LOG4J CIM?

Re: LOG4J CIM?

Re: LOG4J CIM?

LOG4J CIM?

Re: After Updating the Add-on for Windows receive ...

Re: Why do I not see data in my search's visualiza...

Re: Why do I not see data in my search's visualiza...

Re: splunk indexer: status: "pending", fully searc...

Re: splunk indexer: status: "pending", fully searc...

splunk indexer: status: "pending", fully searchabl...