I recorded a successful connection to Splunk Cloud via Curl with NetMon and it captures the Client Hello, Server Hello, Handshake, Cipher change successfully. So it does seem like the UF does a Client Hello, nothing comes back, and it just sends the data down the line anyway. Weird! ... View more

@PickleRick  I got confirmation that the proxy I'm using doesn't require a HTTPS connection to us it and will accept connections. to the proxy, that are not secured (and also it's a CONNECT type proxy - it'll just allow the traffic to tunnel through without inspecting it). As you suggested I used NetMon to monitor the traffic. Here's a summary of what I saw: Traffic leaving the server with the UF on it is going to the proxy server (so the proxy config is working!) The Splunk UF knows its talking to a CONNECT proxy – I can see it opens a connecting (in plan text) asking to connect to ‘CONNECT http-inputs-tennent.splunkcloud.com:443 HTTP/1.1’ And then back from the proxy server comes a confirmation: ‘HTTP/1.1 200 Connection Established’. Immediately after that a TLS handshake: Hello Client is sent. After this I’m not sure what’s going for a couple of message – there are some acknowledgements and resets messages. Then Splunk sends a packet (unencrypted, I assume, as I can see it) with an HTTP payload that starts… POST https://http-inputs-tennent.splunkcloud.com/services/collector/s2s HTTP/1.1..TE: trailers, chunked..Host: http-inputs-tennent.splunkcloud.com..Accept-Encoding: gzip..Content-Length: 5868..Content-Type: application/octet-stream..Authorization: Splunk OUR-VALID-HEC-TOKEN …. [and then a long blob of binary data that I guess is S2S data] Then following some acknowledgements, a keep alive message, and then about 45 seconds later a message back from the proxy server saying the connection has timed out: HTTP/1.0 504 Gateway Timeout.. Ten seconds later it starts the same cycle again. So I’m not sure at all why this is failing – expect the possibility the Splunk UF is thinking it’s successfully talking to Splunk Cloud through the proxy server before the proxy server has decided it’s ready to send traffic on to the Splunk Cloud? ... View more

I'm pretty sure the proxy is an CONNECT type proxy. I say this because when I try the connection using Curl with the verbose flag it actually shows it's 'CONNECT tunnel' before confirming it's connected to the proxy server: Example connection:  curl.exe https://http-inputs-tennant.splunkcloud.com:443/services/collector/raw ^ -H "Authorization: Splunk abff0c6f-670f-4911-bb85-eea72e49f6af" ^ -H "Content-Type: application/json" ^ -d "{\"index\": \"test_index\", \"sourcetype\": \"HEC-test\", \"event\": \"Send event from Curl via the proxy server.\"}" ^ --proxy http://proxyserver.example.com:443 ^ -v The output from Curl when starts... * Host http-inputs-tennant.splunkcloud.com:443 was resolved. * IPv6: (none) * IPv4: x.x.x.x * Trying x.x.x.x:443... * CONNECT tunnel: HTTP/1.1 negotiated * allocate connect buffer * Establish HTTP proxy tunnel to http-inputs-tennant.splunkcloud.com:443 > CONNECT http-inputs-tennant.splunkcloud.com:443 HTTP/1.1 > Host: http-inputs-tennant.splunkcloud.com:443 > User-Agent: curl/8.13.0 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection Established < Proxy-Agent: Zscaler/6.2 < * CONNECT phase completed * CONNECT tunnel established, response 200 * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server did not agree on a protocol. Uses default. * Connected to proxyserver.example.com (x.x.x.x) port 443 * using HTTP/1.x > POST /services/collector/raw HTTP/1.1 and then it does the connection and sends the data to Splunk Cloud.   Does this give any other ideas? Though I still suspect the issue is some sort of incompatibility or mis-configuration between the UF and the proxy server though. ... View more

Here's a couple of ways of getting a list of Windows services and the status of these services into Splunk: Windows Host Monitoring In the inputs.conf file add a stanza like this: [WinHostMon://Service] interval = 600 disabled = 0 type = Service index = windows This will collect a list of services, and status, every 10 minutes, from the system running the Splunk Forwarder. More documentation here: https://docs.splunk.com/Documentation/Splunk/9.1.1/Admin/Inputsconf#Windows_Host_Monitoring WMI Create a WMI.conf file and add the following stanza: [WMI:WindowsServiceState] interval = 600 wql = select Name, DisplayName, Description, State from Win32_Service disabled = 0 index = windows This will collect the same data as the previous example, however its more customisable - for example you can use WMI to narrow down to collecting data on only specific services, or even querying a remote server. More documentation here: https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Wmiconf ... View more

There are some tools that sort of let you do this - including capturing the Splunk configuration changes (all the conf files) into GIT. Here is an old, but still insightful, video from a Splunk Conference some years ago: https://www.youtube.com/watch?v=-NIME9XRqlo ... View more

Can you offer any advice on filtering out (or merging) the two concurrent sessions? Based on this answer: https://community.splunk.com/t5/Splunk-Search/Can-you-help-me-identify-when-end-time-overlaps-start-time-for/m-p/419529 and @VatsalJagani's answer I came up with: index=VPN sourcetype=VPNlog | stats earliest(_time) as session_start, latest(_time) as session_end by Username, SessionID | sort 0 session_start | streamstats current=false last(session_start) as next_start_time by Username | eval overlap=if(next_start_time<session_end,"yes","no") | convert ctime(session_start) ctime(session_end) ctime(next_start_time) Which when sorted by username now produces a column showing overlapping sessions. But I'm stuck merging these overlaps into one block of time: my thinking is that if a session starts at 2:15pm and last 30 minutes, and another starts at 2:25pm and lasts 35 minutes, these should be merged into one event starting at 2:15pm lasting for 45 minutes. I think the Streamstats command, using range() and 'reset_before', would do it but because of the earlier stats command I've lost the _time from the data? Am I heading in the right direction here do you think? ... View more

Thank you both @VatsalJagani  and @bowesmana  - it's great you both came up with working solutions using different approaches. I've learnt something new from both of them. However they both have a catch: @bowesmana - Using your technique with Streamstats I get results up to a point then get the message "The maximum window size (10000) was reached." When this happens I start getting blanks in the data - the results start containing durations of zero. This only happens when applying it across the last 24 hours, shorter periods or less staff are fine, so I assume it's the number of events in the raw data it's analysing. @VatsalJagani  - Your solution almost works perfectly except that remote working staff can seem to have more than only session ID simultaneously. I'm not really sure how this can happen but it seems the VPN records the same staff member, on the same computer, with more than one session ID at the same time. This makes the total very inaccurate as it ends up counting the same timespan more than once. I experimented with using combinations of private and public IP as the secondary BY clauses but for a remote worker these don't change much (if at all) so I end up with counting from the very start of the first session to the end of the last. ... View more

Thanks for this addition information. As you recommended I have changed the entity removing the ‘site’ field from the alias section and recreating it in the ‘Info fields’ section – as below: I then opened the Service and from the Entities Tab, removed ‘site’ as an alias and recreated it as an Info field – so it looks like the following. It does successfully match the correct entity too. In the KPI the ‘entity filter field’ is set to ‘site’. But now the KPI does not work! In the data I can manually search and find ‘[rest of the base search] site=UserInterface’ via search. If I expand the Generated Search for the KPI, open it in search, and reduce the search just back to the base search and the rest lookup I get nothing. So it seems that the generate_entity_filter function isn’t picking up the field 'site' since I changed it to ‘info’. What am I doing wrong here?   ... View more

Hi and thanks for the quick response. Yes - could you give more details on the lookup and entity import search you mention in your reply? Here is a list of servers, IIS site names, and Service Name across both environments. The whole setup is bigger than this but hopefully gives you an idea. Note: to further confuse things the IIS site name is reused on different servers for different purposes. Environment Service Name IIS Site Name Server Host Name Non-production User Interface UserInterface QAUI11       QAUI12   Data Processing DataProcessing QAAPP11       QAAPP12   Image Processing DataProcessing QAAPP15       QAAPP16   Reporting ReportingInterface QAUI11       QAUI12         Production User Interface UserInterface PRDUI11       PRDUI12   Data Processing DataProcessing PRDAPP11       PRDAPP12   Image Processing DataProcessing PRDAPP15       PRDAPP16   Reporting ReportingInterface PRDUI11       PRDUI12   Ideally I would like to be able to create a service in ITIS for each item in the 'Service Name' column which then has KPIs monitoring the performance of IIS for only the servers and IIS sites that serve that service. ... View more

Again thanks for the informative reply. As suggested I opened the generated search and removed everything after '| `aggregate_raw_'. This I found was returning no results at all. So I experimented further with parts of that search query, which started out like: eventtype=microsoft_iis_web | eval 4xx_error=if(status>=400 AND status<500, status, null()), 5xx_error=if(status>=500 AND status<600, status, null()) | search [| rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" | return $value ] From here I took just the rest query and experimented with the entity_alias_filtering_fields argument, starting with: | rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" It returns `no_entities_matched`. However when I changed entity_alias_filtering_fields from "s_sitename" to "host" it returns the value configured for host in the Services entities tab. From here I went on to experiment with other alias's in the Entities section of the service - only the host field ever works, and only for hosts ITSI is aware of (so if I changed the Alias for host to add a hostname that doesn't exist it wouldn't be returned by the above rest query). But after this I tried something further - I looked up the entity in the 'Entities' list (from the Configure menu in ITSI) and added an alias for what I was hoping to filter the entity by. Suddenly the rest call above started working, and after re-creating the KPI that started working correctly too! So it seems the cause of this 'fault' was my mis-reading of the documentation - I had been assuming that mentions of 'entities' when creating KPIs refer to what's in the Entities tab of the Service, but in fact refers to an alias configured against the Entity as a whole in ITSI. ... View more

Thanks for the reply and insights. I didn't know that the 'Entity Split Field' doesn't have to reference an entity defined in the service - it can just reference any field you know will be in your data. However one issue still continues to bug me and I can't quite work out a solution: as I understand it when creating a KPI in a service if 'Filter to Entities in Service' is enabled the 'Entity Filter Field' is used to filter the data to only that which contains both the Entity field name and the value you associated with it as an alias in the entities of the service. I find that this only works for me if the Entity Filter Field of 'host' is used, anything else I use never matches anything (so the KPI is always null). Does the Entity Filter Field need to be declared somewhere else in the service maybe? Or does this field have restrictions like it's case sensitive or can't contain not alphanumeric characters? Thanks For reference: I am doing this in ITIS version 4.4.5 in Splunk Enterprise version 8.0.6. ... View more