There are some tools that sort of let you do this - including capturing the Splunk configuration changes (all the conf files) into GIT. Here is an old, but still insightful, video from a Splunk Conference some years ago: https://www.youtube.com/watch?v=-NIME9XRqlo ... View more

Can you offer any advice on filtering out (or merging) the two concurrent sessions? Based on this answer: https://community.splunk.com/t5/Splunk-Search/Can-you-help-me-identify-when-end-time-overlaps-start-time-for/m-p/419529 and @VatsalJagani's answer I came up with: index=VPN sourcetype=VPNlog | stats earliest(_time) as session_start, latest(_time) as session_end by Username, SessionID | sort 0 session_start | streamstats current=false last(session_start) as next_start_time by Username | eval overlap=if(next_start_time<session_end,"yes","no") | convert ctime(session_start) ctime(session_end) ctime(next_start_time) Which when sorted by username now produces a column showing overlapping sessions. But I'm stuck merging these overlaps into one block of time: my thinking is that if a session starts at 2:15pm and last 30 minutes, and another starts at 2:25pm and lasts 35 minutes, these should be merged into one event starting at 2:15pm lasting for 45 minutes. I think the Streamstats command, using range() and 'reset_before', would do it but because of the earlier stats command I've lost the _time from the data? Am I heading in the right direction here do you think? ... View more

Thank you both @VatsalJagani  and @bowesmana  - it's great you both came up with working solutions using different approaches. I've learnt something new from both of them. However they both have a catch: @bowesmana - Using your technique with Streamstats I get results up to a point then get the message "The maximum window size (10000) was reached." When this happens I start getting blanks in the data - the results start containing durations of zero. This only happens when applying it across the last 24 hours, shorter periods or less staff are fine, so I assume it's the number of events in the raw data it's analysing. @VatsalJagani  - Your solution almost works perfectly except that remote working staff can seem to have more than only session ID simultaneously. I'm not really sure how this can happen but it seems the VPN records the same staff member, on the same computer, with more than one session ID at the same time. This makes the total very inaccurate as it ends up counting the same timespan more than once. I experimented with using combinations of private and public IP as the secondary BY clauses but for a remote worker these don't change much (if at all) so I end up with counting from the very start of the first session to the end of the last. ... View more

Thanks for this addition information. As you recommended I have changed the entity removing the ‘site’ field from the alias section and recreating it in the ‘Info fields’ section – as below: I then opened the Service and from the Entities Tab, removed ‘site’ as an alias and recreated it as an Info field – so it looks like the following. It does successfully match the correct entity too. In the KPI the ‘entity filter field’ is set to ‘site’. But now the KPI does not work! In the data I can manually search and find ‘[rest of the base search] site=UserInterface’ via search. If I expand the Generated Search for the KPI, open it in search, and reduce the search just back to the base search and the rest lookup I get nothing. So it seems that the generate_entity_filter function isn’t picking up the field 'site' since I changed it to ‘info’. What am I doing wrong here?   ... View more

Hi and thanks for the quick response. Yes - could you give more details on the lookup and entity import search you mention in your reply? Here is a list of servers, IIS site names, and Service Name across both environments. The whole setup is bigger than this but hopefully gives you an idea. Note: to further confuse things the IIS site name is reused on different servers for different purposes. Environment Service Name IIS Site Name Server Host Name Non-production User Interface UserInterface QAUI11       QAUI12   Data Processing DataProcessing QAAPP11       QAAPP12   Image Processing DataProcessing QAAPP15       QAAPP16   Reporting ReportingInterface QAUI11       QAUI12         Production User Interface UserInterface PRDUI11       PRDUI12   Data Processing DataProcessing PRDAPP11       PRDAPP12   Image Processing DataProcessing PRDAPP15       PRDAPP16   Reporting ReportingInterface PRDUI11       PRDUI12   Ideally I would like to be able to create a service in ITIS for each item in the 'Service Name' column which then has KPIs monitoring the performance of IIS for only the servers and IIS sites that serve that service. ... View more

Again thanks for the informative reply. As suggested I opened the generated search and removed everything after '| `aggregate_raw_'. This I found was returning no results at all. So I experimented further with parts of that search query, which started out like: eventtype=microsoft_iis_web | eval 4xx_error=if(status>=400 AND status<500, status, null()), 5xx_error=if(status>=500 AND status<600, status, null()) | search [| rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" | return $value ] From here I took just the rest query and experimented with the entity_alias_filtering_fields argument, starting with: | rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" It returns `no_entities_matched`. However when I changed entity_alias_filtering_fields from "s_sitename" to "host" it returns the value configured for host in the Services entities tab. From here I went on to experiment with other alias's in the Entities section of the service - only the host field ever works, and only for hosts ITSI is aware of (so if I changed the Alias for host to add a hostname that doesn't exist it wouldn't be returned by the above rest query). But after this I tried something further - I looked up the entity in the 'Entities' list (from the Configure menu in ITSI) and added an alias for what I was hoping to filter the entity by. Suddenly the rest call above started working, and after re-creating the KPI that started working correctly too! So it seems the cause of this 'fault' was my mis-reading of the documentation - I had been assuming that mentions of 'entities' when creating KPIs refer to what's in the Entities tab of the Service, but in fact refers to an alias configured against the Entity as a whole in ITSI. ... View more

Thanks for the reply and insights. I didn't know that the 'Entity Split Field' doesn't have to reference an entity defined in the service - it can just reference any field you know will be in your data. However one issue still continues to bug me and I can't quite work out a solution: as I understand it when creating a KPI in a service if 'Filter to Entities in Service' is enabled the 'Entity Filter Field' is used to filter the data to only that which contains both the Entity field name and the value you associated with it as an alias in the entities of the service. I find that this only works for me if the Entity Filter Field of 'host' is used, anything else I use never matches anything (so the KPI is always null). Does the Entity Filter Field need to be declared somewhere else in the service maybe? Or does this field have restrictions like it's case sensitive or can't contain not alphanumeric characters? Thanks For reference: I am doing this in ITIS version 4.4.5 in Splunk Enterprise version 8.0.6. ... View more

Hi Stefan, I've faced the same problem trying to deploy the Splunk App for Infrastructure to Windows via a deployment server - Splunk provide no "all Splunk" way of doing it. I think they expect you will be using another tool to deploy your forwarders and config (something like Ansible or SCCM I suppose). However I worked out another way to deploy this config which does use the deployment server and then set the metadata on a per-machine basis via a scripted input with the "interval" configured to -1 (run only once). Create a new app on your deployment server - for example "Windows-Base-Monitoring". Under this create two folders: 'local' and 'bin'. In the bin folder create a PowerShell script to open the inputs.conf file, look for stanza's titled "[perfmon://...]" and then update the _meta field with info specific to that host. The following script does this and inserts a _meta line containing the hosts OS version, IP address, hostname. ## InjectMetaConfigs.ps1 # Collect the hosts meta data essentials: # extract os & ip info $os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version $ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address # Add OS & ip info as dimension # If these dimensions are to be updated, user has to rerun this script $metadata = "_meta = os::" + "`"" + $os_info.Caption + "`"" $metadata = $metadata + " os_version::" + $os_info.Version $metadata = $metadata + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`"" $metadata = $metadata + " entity_type::Windows_Host" # Path to the Perfmon collection TA app: # ($SplunkHome is defined by Splunk before this script is run.) $perfmonApp = $SplunkHome + "/etc/apps/Windows-Base-Monitoring" # The path to the input.conf file to be edited. $inputsConfLoc = $perfmonApp + "/local/inputs.conf" $regex = '(?<=\[perfmon\:\/\/\w+\])' $inputfile = Get-Content ($inputsConfLoc) -Raw $outputfile = [regex]::replace($inputfile, '(?<=\[perfmon\:\/\/\w+\])(\r\n|\n)(?!_meta)', ($1 + "`n" + $metadata +"`n")) Set-Content -path $inputsConfLoc -Value $outputfile Now you can go ahead and create an inputs.conf file in the local folder with all the Perfmon configs set for everything you want to collect - however without the "_meta" lines as these will be set by the PowerScript after it's been deployed to the universal forwarders.  In this file also define a PowerShell scripted input  which runs the above script once only (interval = -1). For example: # inputs.conf # The following powershell script is run just once per restart (internal = -1) and if it finds the Perfmon stanza's below # doesn't contain '_meta' statements immidately after the [perfmon..] line adds it, completing the system info for the # it's running from. [powershell://InjectMetaConfig] script = . "$SplunkHome\etc\apps\Windows-Base-Monitoring\bin\InjectMetaConfigs.ps1" interval = -1 disabled = false ### Perfom monitoring ## CPU [perfmon://CPU] counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time; disabled = false instances = _Total mode = single interval = 60 object = Processor useEnglishOnly = true sourcetype = PerfmonMetrics:CPU index = metrics ## Memory [perfmon://Memory] counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes; disabled = false interval = 60 object = Memory useEnglishOnly=true sourcetype = PerfmonMetrics:Memory mode = single index = metrics  When you deploy this app to your universal forwarders the deployment server handles sending the configuration and PowerShell script out. The UF then restarts itself and reads the new inputs.conf config, as it does this it runs the PowerShell script which updates the inputs.conf file with the _meta lines.  You might need to restart the UF's after this has happened, however in my experience I discovered the inputs.conf is updated before the Perfmon monitoring starts. Hopefully this provides some inspiration to fixing your deployment! Eddie ... View more

Hi Greg, Thanks for your response. I haven't really got this working still (I have a solution using Cribl to change the data later on) - I'm still struggling to get access to the heavy forwarder.  However I agreed with your advice  that the the props and transforms files need to go on the Heavy Forwarder. When I finally get access I'll give it a try. Thanks ... View more