About Jarohnimo

Jarohnimo · ‎08-25-2017

I disabled the real time search around noon today and noticed the skipped searches ratio was already down to 20% I know Enterprise Security is their flagship program but please support ITSI a little better. I'm starting to feel I know more than most of their premier Splunk engineers. I had to once wait a few weeks for a senior engineer to go to ITSI class to learn the tool prior to helping me. The support for this program is very low and I hope they don't get rid of it

Jarohnimo · ‎08-25-2017

Hi rPhillips, Where did you get that information? Seeing how when we search google/ splunk on ITSI Event Grouping there is literally NOTHING. I would like to just turn this off if I can't figure out what the benefit is? This message is being fired off on the indexer (not the WFE) so Im not sure if i need to have this enabled. Please let me know

Jarohnimo · ‎08-25-2017

Most of these searches i found are system generated. Perhaps from a Enterprise app that goes haywire with the scheduling. I'm not sure why this isn't better controlled by splunk as out of the box functions are being skipped.

Jarohnimo · ‎08-25-2017

I have this exact same problem. Only occurs on the indexer, the search head seems to work fine. The indexer has alot of issues (it was the previous ITSI box) but now that i have a Dedicated sh it runs from there... Dirty move, i simply deleted the services once i migrated the content to a new box. wasn't sure if that was the "proper" way to ridding the content but i got a feeling i need to turn something off on the indexer

Jarohnimo · ‎08-25-2017

When is ok to disable one of these offending real time searches? I know for ITSI it needs to be installed on both Search and indexer but i'm not sure why on the indexer ITSI event grouping (noteable events) always gets skipped but not on the WFE. I wish i could just turn it off on the Indexer to bypass all these skipped events but i'm unsure if doing this will break something on the search head.

Jarohnimo · ‎08-25-2017

I'm getting 100% Skip on something that i don't even know that i need "Itsi event grouping" this is on the indexer. I could care less as the indexer job isn't to do noteable events. It's very frustrating trying to find good documentation around these skip ratios and why they are set this way out of the box with splunk apps.

Jarohnimo · ‎08-25-2017

I have a 3 node setup, 1 indexer and 2 search heads. one search head has ITSI (ITSI is also on the indexer). The index is the one that's giving me all these skipped ratios Under the User Management Console, when we click on "Schedule Activity" it shows SA-ITOA 241 executions and that it's 100% skipped. Now at one Point I had ITSI exclusively on this particuar box as it was our stand alone server. I have since migrated ITSI to a new box and only have ITSI on the indexer now for functional purposes. So perhaps ITSI has some running processes that it doesn't need? It says its reason: The maximum number of concurrent searches has been reached (241). I'm very weak in my understanding of Skip Ratio and why, as i didn't manually setup any searches. Why it's failing now is beyond me, I'd imagine there some maintenance that I'm suppose to be doing that I'm not. I have read some of the splunk documentation around this but i haven't grasped the understanding. perhaps someone can break this down for me. my goal is to eliminate all unneeded/ unused scheduled reports, alerts etc... I'm tripping all over the skip ratio and i'm not sure what I'm getting out of it.

Jarohnimo · ‎08-24-2017

I'm getting this warning too (Windows 2012 R2) and spent half the day finding out how much i don't know about linux. Splunk is built on linux, it works better on linux, linux is king ... now that we have established that. They need to fix this or do a better job at making sure the product works correctly on both Windows and Linux platforms. or perhaps only support linux and give up on the Windows Platform.. either way whatever you put out it should work and be applicable.

Jarohnimo · ‎06-29-2017

Roughly 160gb a day. Definitely don't want to adjust the queue if just fixing the storage is the real issue I'd imagine the current queue would fair well with 3000 iops vs the 80 iops we have now according to splunk

Jarohnimo · ‎06-28-2017

Thanks Definitely make sense. For that reason I guess it would make sense to isolate your alerts and scripts on one box and another box for ad-hoc searches? To me the overarching issue I have is index queues. It's very high. Parsing queue sometimes but always index queue. We have very slow storage it seems. At one point to it worked great. It seems the more alerts and searches we set up. The more stuff doesn't work and now this index queue seems to be the biggest issue. I have a 3 server setup . 1 indexer 12 CPUs 24gbs of ram slow disk for 2 TB 2 search heads 16cpus 12 GBS of ram.. disk whatever... Lol This is my set-up. We only have 5 or 6 People searching most of our load is from ITSI or scheduled searches and alerts.. perhaps we are stressing out system too much with the amount to of searches or the effeciency but I truly believe if we had good iops on our indexer it could process these requests much faster and alerts would fire off easily

Jarohnimo · ‎06-28-2017

I ran a report to show me the queues and I see some queues go up to 90. I assume this is 90 seconds wait time... Horrid!! Anyone with experience can tell me if this level of latency on the disk can cause alerts to never fire off? I noticed after I reboot my box in .. then it will fire off alerts... For a lil while...

Jarohnimo · ‎06-28-2017

I have a 2 TB Indexer 12 CPUs, 12GBs of memory. We didn't get a chance to have a say in the storage teir and i imagine we have the slowest storage immaginable. Definitely not the 800 IOPS splunk requires. I'm noticing 20 second spike all the time with my indexqueue and also spikes at times with my parsingqueue Can someone tell me if these problems could cause data loss or things such as alerts not firing off? I'm noticing now that some of the results i look for in splunk are not there even though splunk is monitoring those logs. it takes a LONG time for splunk to show that the info is in there. I'm pretty sure the answer is to get Splunk on faster Storage... i just would like someone to explain the queues thanks.

Jarohnimo · ‎06-09-2017

So instead of limiting my results I filter it with where.. nice!! My end game is an alert, So the way I assume it will work is if any host has their CPU above 95%, In real time search that -5 is t starting point and the now is present, so as long as the value remains for atleast 5 minutes Splunk will alert me at the 5 minute mark? Thanks

Jarohnimo · ‎06-09-2017

Hello all, I know it's possible to find values via greater than, less than or equal to in your search queries. Is it possible to add more arguments to request that the value has to be that way for at least 5 minutes, if so then return the results. In the end i wish to have an alert be sent out if any CPU is above 95% for more than 5 minutes, send an alert. We don't want an alert if its just hitting the ceilling and coming back down, we want it if it's on fire (a real problem). I was trying something like index=blah counter=% Processor Time" Sourcetype="perfmon:processor" Value > 95 span=5m | timechart etc.etc.etc... I believe it's possible but i just need help getting the syntax correct. Thanks in advance.

Jarohnimo · ‎03-08-2017

Import and access are two very different things I'm SharePoint. I think your confused on what I'm talking about. User profile sync can import all the profiles but doesn't automatically give access to any site in SharePoint. Thank you

Jarohnimo · ‎02-09-2017

So I've setup Splunk IT Service Intelligence (ITSI) and am interested in setting up Email Alerts with ITSI (if possible) So far I've been looking into the Multi-KPIAlerts via: https://docs.splunk.com/Documentation/ITSI/2.5.1/User/CreateMulti-KPIAlerts. This creates correlated searches that ultimately create "Notable events" but what about email alerts? I see the benefit of Multi-KPIAlerts but is there anyway to get an email sent each time a notable event is happens? Or is ITSI able to send out email alerts period?

Jarohnimo · ‎02-02-2017

Thank you for your response! From the remedy it appears I shouldn't have only copied the pieces of search and users over to the new server that I absolutely needed to begin with. Weather I delete each Report/Dash via GUI or do it the way described above, it's seems tedious. But going through this way seems like I have to use extra brainpower making sure I don't nick the wrong thing out of the local.meta file. Thanks again!

Jarohnimo · ‎01-31-2017

So I have mass copied the search app from Server A to Server B (Along with the users directory) to basically copy over all the reports and dashboards.. however now I'd like to delete about 100 reports from Server B. Can someone help me identify the appropriate files to delete while on the server within the directories (C:\program files\splunk\etc......). Say I had a report called: Test, what files do I need to delete from within the server to accomplish this?

Jarohnimo · ‎01-31-2017

Still feels like I'm piecing it together because I am. I wish their was published information on how to use these guys because right now I'm guessing. I don't use heavy forwarders only universal and to me this seems way more complicated than it needs to be. I feel we should be able to tag our servers (while setting up entities) where they will know which ones an app servers, which is sql, which is VM host etc....

Jarohnimo · ‎01-27-2017

I've see the TA folders but didn't pay them much mind as I never had to do anything with them. My setup has been pretty much all basic. How does one have the logs go through the splunk built TA's? And Were you able to locate any documentation that explains all of this. Before I read your response I thought it was easy enough going to.. Settings>Tags>List by tag name and go in there and manually create tags.. oshost=* ^ Or can this be used as a work around if the other way is a pain!?

Jarohnimo · ‎01-27-2017

I've see the TA folders but didn't pay them much mind as I never had to do anything with them. My setup has been pretty much all custom. How does one have the logs go through the splunk built TA's (I hope to God it's not hard)? And Were you able to locate any documentation that explains all of this. Before I read your response I thought it was easy enough... Setting>Tags>List by tag name and go in there and manually create tags.. oshost=* ^ Or can this be used as a work around if the other way is a pain!?

Jarohnimo · ‎01-27-2017

Did you have to tag all of your servers "virtualization" and if so how did you know to tag them that? Can you provide a link for the initial setup, I've been looking but cannot find. I suspect the reason mine isn't working is because I haven't tagged my server with the predefined ITSI tags but I cannot find any documentation anywhere that explains how to setup ITSI with it's tags for non generic search.

Jarohnimo · ‎01-27-2017

So when setting up a new Service in Splunk IT Service Intelligence, it allows you to select a Generic KPI or Select from a list of pre-defined KPI's provided by sSplunk (Thank you very much). I'm new to the Technology, so I'm still figuring it all out, but I feel there may be missing pieces of instruction on how to properly configure ITSI (When using non generic KPI's). Each time I would try to use the non-generic KPI say for example: Under Operating System: Network Utilization, I select: Network Utilization: Total Packets/second (in/out). it doesn't return any results. In the beginning of the setup it allows you to click "Generated Search" It will show all the servers in your farm, but then it also shows tags at the top (that's when I gasped and begin to have hope!... so I'm here now trying to confirm the right way to do this) I've been following Splunk's Guide here: http://docs.splunk.com/Documentation/ITSI/2.5.0/Configure/HowtocreateKPIsearches to try and configure this properly, but there's no documentation on how to get this to work. My suspicion is that you have to manually tag each server to match all the predefined tags associated with the KPI you have so that when you use those KPI's it will return results. Is this the case? If so can someone please provide me a link from Splunk that explains in detail this process? Trying to do things by the book on the setup as ITSI is not the easiest to setup and we need for it to be extra accurate! Thanks! One reason why I didn't move forward with manually creating the tags an adding my servers was because I'm unsure if ITSI was suppose to create those tags once ITSI was installed and then I just go in there and plug in the name of my servers.

Jarohnimo · ‎01-26-2017

Thank you for your insight, but perhaps a compromise can be made. It appears that you and I may work in similar fields (not saying anymore) however this isn't a negotiable tasks. Many of the points you've made I've made and the direction is Sally forth... Dealing with SharePoint People, they import all 40k user no problem, why can't Splunk do it?... So here I am.. . Question I have is, just because all users and groups are impprted doesn't mean all users have access right?. It's still controlled through mapping each group to a role in splunk and we'd like to do that based on request. Mapping each LDAP isn't practical to the org I work for But You've made some great points. I will regurgate them in a meeting today.

Jarohnimo · ‎01-26-2017

Your awesome!

Posts	164
Solutions	7
Karma Given	45
Karma Received	22
Member Since	‎04-01-2015

Online Status	Offline
Date Last Visited	‎11-01-2022 08:50 AM

Test SHC running out of mem every day.

Why must I upgrade to 7.2 first before 8.1

Scripts Directory & upgrades

Converting Splunk Instance from Windows to Linux

Need help writing input stanza for maillog

Creating Index time fields for your HF's

Determining what data is in use/ accessed vs what'...

Else if like not working with json type data

Dashboard Tokens and PDF Delivery help

Juniper firewall search time field extraction not ...

Re: Why are real time searches not running and get...

Re: Why is the itsi_event_grouping scheduled searc...

Re: Why do we have 75,000 skipped scheduled search...

Re: Why is the itsi_event_grouping scheduled searc...

Re: Why are real time searches not running and get...

Re: Why is there a high Skip Ratio showing in Sche...

100% Skip Ratio - ITSI event Grouping

Re: Where is the correct configuration file locate...

Re: Slow Indexer Problems

Re: Slow Indexer Problems

Re: Slow Indexer Problems

Slow Indexer Problems

Re: If CPU is > 95 for more than 5 minutes < -- ho...

If CPU is > 95 for more than 5 minutes < -- how do...

Re: How to use my LDAP strategy to add all users i...

Can Splunk IT Service Intelligence (ITSI) send ema...

Re: Where do you manually delete certain reports o...

Where do you manually delete certain reports or da...

Re: Splunk IT Service Intelligence: How to get on-...

Re: Splunk IT Service Intelligence: How to get on-...

Re: Splunk IT Service Intelligence: How to get on-...

Re: Splunk IT Service Intelligence 2.4.1: How to a...

Splunk IT Service Intelligence: How to get on-Gene...

Re: How to use my LDAP strategy to add all users i...

Re: How do I exclude a certain set of servers in s...

Are you a member of the Splunk Community?