About JustinSC

JustinSC · ‎07-26-2022

I can attest that removing this stanza after the migration will cause Splunk to revert to the mmapv1 storage engine on restart. It will fail to start the kvstore and you cannot re-migrate the kvstore even if you add the setting back. Now I have to figure out how to clean everything out and start over.

JustinSC · ‎03-25-2022

I've got a lot of LDAP users who no longer exist (500+ folders in /etc/users). What's the proper way to clean this? If I just delete the folder won't the search head cluster just resync it from the captain's running configuration? Also, someone pushed these using the Deployer when we migrated to 8.x. If I remove all user folders from the /etc/shcluster/users folder of the Deployer will anything bad happen next time I push? I have no desire to manage user settings with the Deployer; just push shcluster apps.

JustinSC · ‎03-18-2022

I had a situation where I wanted to know if the mstats p90(cpu) over 5 minutes of a host was above a certain value; but needed to extend it to 10 minutes for some hosts. I figured rather than make two searches I could use span=5m and search back 10 minutes: (Search Window: -10m@m to @m) | mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m Except this was often producing 3 events per host, because unless I'm mistaken mstats span always aligns to UTC 0, so if I'm running the search on a minute not divisible by 5 (say every 3 minutes) I'll end up with 3 data points per host instead of 2. So I thought, maybe using prestats + bin + stats will work; I can get 10 samples and use bin aligntime=earliest to force them to just 2 time bins. I think this works, a quick check says that the P90 values are the same up until 4 decimal places if the times are aligned: (Search Window: @h-10m to @h) | mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m | join host, _time [| mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m | bin _time span=5m aligntime=earliest | stats p90(_value) AS p90B BY host, _time ] | where round(p90A,4) != round(p90B,4) So this search should work for any two 5 minute intervals aligned to any minute of the day. | mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m | bin _time span=5m aligntime=earliest | stats p90(_value) AS p90 BY host, _time | where p90 > 80 | stats list(p90), count by host | where count == 2 OR match(host, "prod") I ended up not needing it when I realized my alert was already locked to every 5 minutes. Has anyone else tried doing this? Know a better way without creating two searches?

JustinSC · ‎11-23-2021

We've got the Splunk App for Infrastructure inputs for Windows metrics deployed to our universal forwarders. Metrics are all working fine except in one situation; if any performance counter value is 0 then Splunk isn't recording the value. For example, if a disk free space counter reaches 0, it'll drop off all charts. Likewise application pool request queues don't record a 0 value if there's nothing in them. I've got plenty of custom metrics I've done (faking statsd protocol) that can write 0 just fine, it seems to only be the perfmon metrics from SAI that have the issue. Has anyone else encountered this and know what the fix is?

JustinSC · ‎11-22-2021

Thanks! As I thought about it further I must have been doing a search after something like... | rex max_match=0 "ComputerName=(?<ComputerName>.+?)\b" What is the $s in the FORMAT example you gave? Was that meant to be $2? I should have also mentioned I have control of the scripts writing these logs and could write out the computer names in a denser format. I could also just generate more distinct events (repeat the event per computer name instead of shoving them all in one event) but I'm not sure what's worse; requiring the regex in the transform or generating more log data. This gave me an idea too; perhaps I could call out the fields I want parsed as multivalue by giving them a suffix like ComputerName[]=Server1 ComputerName[]=Server2; unless there's a smarter way.

JustinSC · ‎11-22-2021

I've got a situation that I thought I understood but clearly don't. I have logs that look like this: 2021-11-22 14:00:00 Event=InventoryComplete ComputerName=Server1 ComputerName=Server2 ComputerName=ServerN I thought that ComputerName would automatically be a multivalue field due to there being multiple copies of that Key=Value pair and I'd be able to search any of the values. And I thought there are instances where this works automatically, but it's not right now. | search sourcetype=inventory_audit ComputerName=Server1 ```works``` | search sourcetype=inventory_audit ComputerName=Server2 ```no results``` | search sourcetype=inventory_audit "ComputerName=Server2" ``` forcing text search works``` Is there something I can do to make these events implicitly multivalue? Ideally for the entire sourcetype regardless of the specific field name, as this sourcetype covers a wide variety of audit logs with different object classes.

JustinSC · ‎08-18-2021

I never came up with a solution. I stopped using mcollect and now use the Summary Indexing feature with Metrics index type, which I believe requires Splunk 8.1. For some reason I never setup any metrics to use a host dimension, but perhaps this feature preserves it. I believe something like this would work, or at least it's worth a shot: sourcetype=mysourcetype host=* "Events to count" | timechart limit=0 span=1m count by host | untable _time host "My.Metric.Name"

JustinSC · ‎04-08-2021

Thanks! This has been very helpful. I thought setting the upper limit to "now" would help ensure that if a server somehow ends up a few too many seconds ahead of Splunk (and therefore _time is greater than _indextime) it wouldn't be excluded, since the next search window would exclude the event as _indextime is now outside the window. Is that right or is my logic bad?

JustinSC · ‎04-07-2021

Thank you. The sliding windows are what I currently use for converting events to metrics (i.e. running a report from -4m@m to -3m@m and mcollect the results). It's nice to know that's a recommended way to handle those. For using the indextime is this right? sourcetype=somesourcetype event=SomeEvent earliest=-10m@m latest=now | where _indextime >= relative_time(now(), "-1m@m") AND _indextime < relative_time(now(), "@m") Is there any reason I can't use the previous minute (@m) for the upper bound of indextime, or should I always go back a bit?

JustinSC · ‎04-07-2021

I'd like to have an alert that throttles per result, but triggers only once per schedule run (instead of once per host). Or solve this problem some other way. For example if a service restarts we'll pick up an event from the log and send an email, but if 10 restart at the same time we don't want 10 emails, just one. We want some sort of throttle on the alert because: We're monitoring Production systems and we need the alert to fire within a minute or two of the event occurring. So we're using a schedule like * * * * *, or maybe */5 * * * * depending on the criticality of the event. We can't guarantee that the event is delivered to Splunk on time. On occasion someone will be messing with the SAN and indexing will get delayed by a couple minutes, or perhaps the host rebooted and the event won't be sent for another few minutes. We're stuck with this environment for the time being. This (to my understanding) prevents us from using a simple schedule like * * * * * and earliest=-1m@m because we could miss events that don't arrive within the minute. Therefore the best way to ensure we don't miss events has been increase the range of the search, for example to -5m@m. However this means we'll catch the same event multiple times if our schedule is still * * * * *. Hence wanting to exclude the events we've already alerted on once. Is there a good way to approach this problem? I have similar concerns about using summarization searches where I want the summary results quickly but don't trust the events to be searchable quickly enough. Thank you.

JustinSC · ‎02-23-2021

I've got some events I'm converting to metrics using mcollect with a scheduled report. Does anyone know how to get the metrics to store the original host from the logs instead of whichever indexer they get sent to? This is the query I'm using to populate the metrics: sourcetype=mysourcetype host=* "Events to count" | bin _time span=1m | stats count AS _value BY _time, host | eval metric_name="My.Metric.Name" | mcollect index="prod_metrics" The host field in the metrics ends up being a random indexer from our cluster. I know I could always rename host as server, but if possible I'd like to use the expected field name since all our other natively populated metrics are by host.

Posts	11
Solutions	0
Karma Given	3
Karma Received	1
Member Since	‎02-23-2021

Online Status	Offline
Date Last Visited	‎09-21-2022 02:35 PM

How to cleanup etc/users folder on search head clu...

How do I use mstats and changing aligntime to set ...

Splunk App for Infrastructure Windows not collecti...

Automatically generating multivalue fields in gene...

Trigger alert once for all results but exclude res...

Storing the correct host dimension with mcollect

Re: How to complete a KVStore Migration