About Silmarillion197

Silmarillion197 · ‎11-04-2021

In WildFly application servers there is the /metrics REST endpoint. What is the best way to get the data provided from WildFly /metrics into splunk? What we have found is the "REST API Modular Input" App (https://splunkbase.splunk.com/app/1546/), but this will cost 99$ per connection, and we have 200+ different WildFly Servers - comparing that to Prometheus + Grafana, which comes for free and has such an API "out of the box". So it would be hard to justify this solution. But as we already have a splunk environment we would like to keep that, so there must be a better solution for this, which costs less than this REST API Modular Input API". We have a Splunk forwarder on all WildFly servers, so it should be possible to grab the data somehow and push it to splunk. We have also seen the "Splunk Add-on for Java Management Extensions" AddOn, but this seems like re-inventing the wheel, as the data necessary for monitoring is already provided in the /metrics endpoint. And opening a production server for remote JMX access seems odd - as JMX can do anything to that server, not just performance monitoring, which feels like a severe security breach, and JMX security and Beans change from release to release. Who can help?

Silmarillion197 · ‎10-22-2020

It is not my management. We are using standard splunk forwarder management for rollouts: https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Forwardermanagementoverview And I`m not aware of a possibility to get those servicenames right from the splunk forwarder management, because this would require a mapping on the splunk forwarder manager that is aware which service belongs to which host. That is far more complicated to manage, than just storing the servicename once as an environment variable on the corresponding server. Ok, as you are saying that we can`t use an environment variable in the inputs.conf, I think we have to create a scripted solution of our own for rollout, such as ansible script. That rollout script solution would make the splunk forwarder management useless for us.

Silmarillion197 · ‎10-21-2020

Hi, we have 180+ machines with different services, which send their data using a splunk forwarder to different indexes. To keep this scenario manageable we use a splunk management instance to rollout inputs.conf and outputs.conf on each of these splunk forwarders. This scenario woks fine, as long as the same index is used for all the services and data for all services arrive in that "superindex". But it is obligatory to seperate the indexes, as data comes from different services (requirement). So I thought we could establish an environment variable on each linux system, which keeps the servicename and then refer to that servicename as an index in the inputs.conf For example like so: [monitor:///var/log/service/*/service.log] sourcetype = sc:$SERVICENAME:service:log disabled = 0 index = $SERVICENAME That way, we could still rollout the inputs.conf using the splunk manager and only have to set up this environment variable $SERVICENAME once on each machine. But it seems that the environment variable isn`t recognized in the inputs.conf, as on Splunk Indexer there is the message "Search peer splunk-indexer has the following message: Received event for unconfigured/disabled/deleted index=$servicename with source=[...]" So it seems that the environment variable $servicename was not resolved to the value which was set on the machine. Is there a different way to make index in inputs.conf flexible for each machine and nevertheless keep the rollout system, or can it be done using environment variables and we did it wrong somehow?

Silmarillion197 · ‎08-04-2020

Excellent. This works like a charm. With this solution, we can show the number of returned results outside a XML table. Great. Thank you.

Silmarillion197 · ‎05-22-2019

This seems to be correct. But I have to figure out, on how to create a lookup table. At this time my created lookup table can`t be found in splunk search. It seems that a lookup has to be defined in the transformation.conf so that splunk can find ist (this file I can only change together with the splunk administrator). Thy so far.

Silmarillion197 · ‎05-22-2019

Sorry, this doesn t work for me, as day of the week in my table is a row and not a column. Day of week is nevertheless shown in the chart as x-axis. If there is no event for a certain day of the week, you can t make splunk to invent one (only with makeresults which is not a suitable solution).

Silmarillion197 · ‎05-17-2019

Hello, I`m trying to find a solution for this problem. The result of the following SPL query should show every day of the week in a diagram. So the diagram should show Mon ... Tue ... Thu ... Wed ... Fri ... Sat ... Sun (x-axis) always, even if there is no data at all (seems that if no avg or no sum can be aggregated then the specific day of the week will not show up). => I have tried to fillnull, value=0 or coalesce ifnull but this is not working for me, as the day of week with no result is never shown. _time is overwritten because events loaded into splunk are deferred sometimes, so I use the real START_TIME of Event as base for calculations. This is working so far, except if a "day of week" has no data: index="rw_trail_complete" | eval StartEpoch=strptime(START_TIME, "%Y-%m-%d %H:%M:%S.%Q") | eval _time=StartEpoch | eval EndEpoch=strptime(END_TIME, "%Y-%m-%d %H:%M:%S.%Q") | eval DayOfWeekName=strftime(_time, "%a") | eval DayOfWeekNumber=strftime(_time, "%u") | eval Dur = EndEpoch - StartEpoch | bucket _time span=1d | search SCRIPT_NAME=$scriptName$ | eventstats count AS "Num Events per day" by DayOfWeekNumber | table DayOfWeekName, DayOfWeekNumber, "Num Events per day", Dur | stats avg(Dur) AS "Avg Dur per Day of Week" by DayOfWeekNumber, DayOfWeekName, "Num Events per day" | table DayOfWeekName, "Avg Dur per Day of Week", "Num Events per day"

Posts	7
Solutions	0
Karma Given	4
Karma Received	0
Member Since	‎05-17-2019

Online Status	Offline
Date Last Visited	‎11-15-2021 06:44 AM

WildFly Metrics

Need environment variable for index ?

Avg Day of Week does not show up, when no result f...

WildFly Metrics

Re: Need environment variable for index ?

Need environment variable for index ?

Re: How to add final total count of results withou...

Re: Avg Day of Week does not show up, when no resu...

Re: Avg Day of Week does not show up, when no resu...

Avg Day of Week does not show up, when no result f...