Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About thejeffreystone
thejeffreystone
Path Finder
Member since:
07-25-2016
09-27-2021
Community Statistics
| Posts | 30 |
| Solutions | 4 |
| Karma Given | 10 |
| Karma Received | 6 |
| Member Since | 07-25-2016 |
Activity Feed
- Karma Re: How to create a table using dedup to show one entry for each application name and create a multivalue field? for dmaislin_splunk. 06-05-2020 12:48 AM
- Karma Re: Splunk-sdk-python installation failed on windows 7 64-bit for anshanno. 06-05-2020 12:48 AM
- Karma Re: Splunk IT Service Intelligence: How is the Importance Value for KPIs calculated that contribute to the Service Health Score? for appache. 06-05-2020 12:48 AM
- Karma Re: Splunk OS Host Module Server uptime for hunters_splunk. 06-05-2020 12:48 AM
- Karma Re: average count events for javiergn. 06-05-2020 12:48 AM
- Karma Re: Need to rename just one header for AlexeyNL. 06-05-2020 12:48 AM
- Karma Re: Splunk ITSI Certification for cmerriman. 06-05-2020 12:48 AM
- Karma Is there a concise guide for migrating an ITSI search head to a search head cluster? for csprice. 06-05-2020 12:48 AM
- Got Karma for Re: Splunk IT Service Intelligence: How are service dependencies calculated into the health score of the service?. 06-05-2020 12:48 AM
- Got Karma for Re: Splunk IT Service Intelligence: How are service dependencies calculated into the health score of the service?. 06-05-2020 12:48 AM
- Got Karma for Re: Splunk IT Service Intelligence: How are service dependencies calculated into the health score of the service?. 06-05-2020 12:48 AM
- Got Karma for Re: After installing Splunk SDK for Python, why am I getting "ImportError: No module named spluklib.client" in my python code?. 06-05-2020 12:48 AM
- Got Karma for Re: Python SDK connection error: socket.gaierror: [Errno 11004] getaddrinfo failed. 06-05-2020 12:48 AM
- Got Karma for Re: Shared search for ITSI and glass panel. 06-05-2020 12:48 AM
- Karma Re: What is the best way to get list of index in my splunk for Ayn. 06-05-2020 12:46 AM
- Posted Re: ITSI License on Installation. 07-11-2017 05:06 AM
- Posted Re: Splunk IT Service Intelligence: How are service dependencies calculated into the health score of the service? on Splunk IT Service Intelligence. 02-03-2017 10:20 AM
- Posted Re: Splunk IT Service Intelligence: How to get on-Generic KPI's (Pre configured KPI source searches) to work? on Splunk IT Service Intelligence. 01-31-2017 07:48 AM
- Posted Re: Splunk IT Service Intelligence: How to get on-Generic KPI's (Pre configured KPI source searches) to work? on Splunk IT Service Intelligence. 01-27-2017 10:28 AM
- Posted Re: Splunk IT Service Intelligence: How to get on-Generic KPI's (Pre configured KPI source searches) to work? on Splunk IT Service Intelligence. 01-27-2017 09:07 AM
Topics I've Started
No posts to display.
07-11-2017
05:06 AM
I want to say the minimal license was 250mb. But at any rate, I believe you can match your current license. Our current ITSI license matches our Splunk Cloud license.
... View more
02-03-2017
10:20 AM
I think kind of glossed over the dependencies / KPIs. You are right, you have no sliders for a service that is a dependency of another service. A dependency in that case is considered a minimum health score so it is treated as if it is weighted at 11.
... View more
01-31-2017
07:48 AM
Yeah. It has to do with Splunk wanting ITSI to use CIM Compliant data and using the TAs to ensure it is.
... View more
01-27-2017
10:28 AM
So most of the TAs are just field extractions, and tags. So if you are using a heavy forwarder in your setup just install the TAs on the heavy forwarder. As long as your applications are using the default source paths or log names/formats the heavy forwarder will process it. You will also probably need them on your search head so it is aware of the eventtypes if there are any, and the props and transforms if you are not using a heavy forwarder.
You could do it the manual way. But I would dissect the TA and see what all it is doing. Is suspect you will need to create more than just tags, but the TA will have a props.conf and transfoms.conf that will tell you what it would do, and then you could manually recreate it.
I am by far not the expert in this, but going through the same thing. My problem is everyone has heavily customized the app logs that the TAs wont help so we are having to do custom KPIs.
... View more
01-27-2017
09:07 AM
My understanding is that to use those prebuilt KPIs you need have those logs go through the Splunk built TAs which will make sure everything is CIM compliant. For example if you want your Apache web logs to be tagged correctly then you should be using the Splunk Add-on for Apache to process those logs.
Just had a conversation with my SE about that this week.
... View more
01-17-2017
04:01 AM
Sorry for the late response. That is one I haven't quite solved myself. I have issue with how the services roll up, so if I have a service dependent on other services I typically just write a new base search.
... View more
01-10-2017
11:13 AM
I grouped mine based on how I thought they would affect the service. The ones that directly impacted service got grouped together and the ones that indirectly or where the impact was conditional based on other factors got grouped together. My use case was audio and video stream metrics so it was a little different.
I had a base search with 7 KPI metrics in it. And what I did was created two new metrics and added them to the base search that determine if the entity is in either a critical state or a warning state. For example, the critical metric looks at three of my KPIs that I consider to have a major impact to my service and sets the critical KPI metric to 1 if any of those KPIs is "failing." I sum those up based on the service. So if all the entities in my service are failing then the sum would be 9.
Because I didn't want to lose the visibility of the other KPIs I left the other 7 KPIs in my base search. Then in my service I weighted the two new KPIs (critical and warning) normally (10 and 5) and set the other 7 to 1 so they were just informational. So now my health score is determined by the two new metrics. And when I see the service is degraded I click on it in service analyzer I see immediately what my problem KPIs are because they are included in the service.
... View more
01-10-2017
09:11 AM
3 Karma
The service dependencies are weighted by that slider for 0-10 and I believe it is just almost like a percentage of total the total. So if you have 5 KPIs and wight them all as 10 each one will not move the dial much. From what I have found its hard to get that Health score dialed in.
If you have a critical KPI or one that 1 to 1 impacts the over health though you should really consider making it an 11.
But I agree, its hard to get the weighting dialed in. Event if you have them all spread out on the weighting each one effects the over health score very little. You would need multiple KPIs going bad to really make the health score dip.
What I have done to get the health score dialed in is combine KPIs. So for example if I have 6, I group 3 into a critical KPI that gets weighted high, and the other 3 into another group that gets a low weight. Then the health score is a bit closer to what I want it to show. That way if anyone of my critical KPIs tripped the health score is reflected where as if they were individual all three would have to get tripped.
... View more
10-27-2016
05:11 AM
If you are on *nix, then you might be able to edit /etc/init.d/splunk
I suspect if you run 'splunk enable boot-start' you will overwrite your changes though. But in there you could add your script to the splunk_start(), and splunk_restart()
... View more
10-21-2016
02:21 AM
I think that example might show the issue. If you had just one entity and one KPI using that threshold I think what will happen is when the KPI on a single entity hist 501ms, the service (aggregate) is going to pass the low threshold and change colors but the entity is going to still be below the low threshold. When the entity hits 1001ms, the service is going to pass the medium threshold but the entity will just pass the low and so on. I'll have to do some testing.
In mine, the entity/aggregate thresholds are reversed. Low alert on the entity is >500, and on the aggregate of the entities it is 1000 since one entity being in a bad state won't necessarily affect the service, but will affect the entity.
... View more
10-20-2016
04:47 AM
You could create a modular input using the python SDK. It is not really hard. Here is a basic example: https://github.com/kospol/splunk_simplePythonModularInput
Then your script can just be an app inside splunk.
... View more
10-19-2016
09:20 AM
I haven't seen that error, but the unable to edit error makes me think the user Splunk is running under may not have permission to modify that file. Is the SA-ITOA directory owned by the same user Splunk is running as?
... View more
10-11-2016
09:10 AM
Does the entity thresholding need to be adjusted? That sounds like a strange issue. My thresholds for my entities though are way lower since the values on those KPIs will be less than the service.
What is the value of your KPIs? Is it a percentage or a count? And how is the calculation per entity and service?
... View more
10-05-2016
05:26 PM
1 Karma
I'm guessing your environment doesn't know where the splunklib dir in the sdk is located. Not sure how to set that up on windows. I'm sure python on windows came with pip though.
Try to install via pip:
from: http://dev.splunk.com/view/python-sdk/SP-CAAAEDG
Use pip:
pip install splunk-sdk
Also, I do not think it works with python3
... View more
10-05-2016
11:47 AM
This might help too:
https://answers.splunk.com/answers/134553/how-to-delete-data-index-reset-start-from-scratch.html
... View more
10-05-2016
04:55 AM
So you see data in preview but not in sourcetypes, and you have the TA on the remote server. Sounds like the sourcetype and possibly field extractions might not have been created either. Especially if you can see the data in a query outside the app. Either that or it could be a permissions issues I guess if your account doesn't have access to some of the data, but that seems unlikely since you see the data in preview.
... View more
10-04-2016
11:03 AM
Thats strange. I just installed it on a new 6.5 instance and the only thing I had to do was enable the metrics I wanted to see in the Splunk_TA_nix.
... View more
10-04-2016
10:34 AM
I typically leave it since going forward all the data is right. But I guess it depends on how long it was not breaking properly. I have never done it but I believe you could delete the data in your index, delete the fishbucket on your universal forwarder, and just reingest the data. If it is a lot of data then that might be intensive.
... View more
10-04-2016
10:14 AM
As of right now I think it would have to be via a script or email. You can build a multi-kpi alert, then under the correlation search, set it to fire off an email or script. You might be able to point one of those to service now to achieve what you are trying to do.
... View more
08-16-2016
11:29 AM
No waste. I learned a new way of doing something.
... View more
08-16-2016
11:21 AM
haha...I was working on my response and didnt see you have found it. Glad you got it.
... View more
08-16-2016
11:13 AM
or perhaps a firewall/routing issue somewhere. I was suspecting dns when I first read it, but you already disproved that. It's been a while since I encountered that in python. I did some testing in one of the example scripts and can't get the same error. So perhaps in this context its not resolution but something else. I think it is for sure in your connection attempt.
When I pass bad host names I get:
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
I'm using python 2.7.11 for reference and the same version of the sdk.
So now I am not so sure. I tested connecting to splunk cloud and to a local instance using the .splunkrc file and the stail.py example in the sdk.
I couldn't get your script to work. But I modified the info.py script provided to provide the connection info in the script. Not sure if this will format right:
from splunklib.client import connect
from splunklib.results import ResultsReader
# Create a Service instance and log in
service = connect(
host='host.com',
port=8089,
username='admin',
password='mypass')
content = service.info
for key in sorted(content.keys()):
value = content[key]
if isinstance(value, list):
print "%s:" % key
for item in value: print " %s" % item
else:
print "%s: %s" % (key, value)
print "Settings:"
content = service.settings.content
for key in sorted(content.keys()):
value = content[key]
print " %s: %s" % (key, value)
... View more
08-16-2016
08:58 AM
1 Karma
I believe that error means it couldn't resolve/reach your host.
... View more
07-25-2016
06:16 AM
Cool.
Yeah, then you just need to know how you want to define your KPIs and how you want those to effect your service.
Sounds like then you could use your 4-8 systems as entities. And then each entity is going to have 16-20+ KPIs that will be used to calculate the health of those entities.
Then your base search could just output a table with the headers like:
host service1 service2 service3
Each service# column would be a metric you could create a KPI around.
... View more
07-25-2016
05:17 AM
1 Karma
I am not sure what you consider a system, so I just assumed system = host for what I created below.
Just create a base search or searches that gathers your metrics/KPIs by entity. Your entities in this case could be the 16-20 services or it could be the hosts those services are running on. Either way should work as long as long as the base search outputs the metric values by whatever you want to use as your entity.
For example you could setup System1 as a service, and then make your entities the 16-20 services under that service like "system1_service1", "system1_service2" and then your base search gathers your metrics and outputs them in a table of rows where "system1_service1" has some columns of metrics (metric1, metric2 - if each of those service has multiple KPIs) and lookups by entity and filters by entity under service. Then for "system_service1" you can set up some KPIs that use the single base search and pull a single metric for each one of those "systemX_serviceX" entities. On your glass table you could then put all 8 systems as individual services or combine them into a higher service and display the health using those KPIs
Or if those 16-20 services each have just one KPI/metric you could go simpler, and just list your 8 systems as entities, and add those to a service called MyApp. Then create a base search, or multiple that outputs a table with your columns for each of the 16-20 services that contains the metric you want to capture for that service. Then filter the base search by host. Then create a KPI for each of those 16-20 services.
I've just started playing around with ITSI myself, but have already had use cases that used both the approaches above. It really comes down to defining what is a service, entity, and KPI/metric you want to display.
... View more
