RCA for impact between 10:57 pm - 11:40 pm - Only tier1 and WWW in CLT-A were enabled at 10:57 pm to resolve the hard down condition, Tier3 was still in BRF. Tier3 in CLT-A was enabled at 11:40 pm. There is a dependency of user-security service in Tier3 on public tools and shared apps in Tier1. They all have to be in the same datacenter for user-security local coherence cache to work.
There are no short term action items (including the OCI F5 dashboard proposed at the RCA call) at this time to address this problem.
Strategic solution for WebCenter Switching discussed in the RCA call -
A) AppViewX - being rolled out by next month and Will have visibility from F5 perspective. The tool will show the state of F5, eliminate unknowns when commands are unsuccessful on F5.
B) Operational Control Framework - replacement for OCI tool. The framework will be utilizing AppViewX and will have visibility on state of operations. Project just got approved and is in initial phase. Tentative time of completion is end of this year and roll out starts Jan 2017.
F5 software upgrade to V11 is being tracked through PRB75373.
The data exists like this in a formatted way when we pull out the same in splunk
RCA for impact between 10:42 pm - 10:57 pm -The root cause could not be identified. There is a suspicion that WebCenter switching did not work as expected. However, there are no logs either on F5 or OCI tool to show where the failure was. Analysis was done to see if Tier1 disable commands fired from OCI tool show up on F5 side. Publictools, shared, iwc, m and retirementatwork Tier1 VIPs disable command in CLT-A do not show up on F5 in splunk. F5 team has confirmed that there are no splunk logging issues as other Tier1 VIPs disable command can be seen on F5 in splunk. F5 logs are already rolled over on the device and not available to further review the problem. RCA for impact between 10:57 pm - 11:40 pm - Only tier1 and WWW in CLT-A were enabled at 10:57 pm to resolve the hard down condition, Tier3 was still in BRF. Tier3 in CLT-A was enabled at 11:40 pm. There is a dependency of user-security service in Tier3 on public tools and shared apps in Tier1. They all have to be in the same datacenter for user-security local coherence cache to work. There are no short term action items (including the OCI F5 dashboard proposed at the RCA call) at this time to address this problem. Strategic solution for WebCenter Switching discussed in the RCA call - A) AppViewX - being rolled out by next month and Will have visibility from F5 perspective. The tool will show the state of F5, eliminate unknowns when commands are unsuccessful on F5. B) Operational Control Framework - replacement for OCI tool. The framework will be utilizing AppViewX and will have visibility on state of operations. Project just got approved and is in initial phase. Tentative time of completion is end of this year and roll out starts Jan 2017. F5 software upgrade to V11 is being tracked through PRB75373.
Please help me out with options i have to provide an update whether its doable or not
Thanks
Krishna
... View more