Related outage(s) : Southern Nodes : https://coolideas.co.za/announcements/64004
Frogfoot/Thinkspeed is currently experiencing the following issue:
Start Date & Time: 31/12/2024 23:10
Affected Area/s: Western & Eastern Cape
Impact: Customers may experience degraded connectivity.
Frogfoot technicians are currently investigating. There is no ETR at the moment.
Update (03/01/2025 @ 14:54):
Frog foot confirmed that this outage was resolved on 03/01/2025 @ 14:54. Please restart your devices and contact our Support Team at 010 593 2665 or support@coolideas.co.za should you still experience connection issues.
Update 03/01/2025 @ 12:33pm
-- The on-site tech and our CORE team have rebooted the switch, they are currently completing the required checks. Stability will be monitored over the next hour and we will provide further updates.
Update 03/01/2025 @ 11:05am
-- Tech has been on site since 9.30 am and the core team has been investigating remotely, we are awaiting further updates on the investigation.
Update 03/01/2025 @ 09:08am
-- We have identified a loop on the network which seems to be originating from the table view switch. The core team have disabled that backhaul as a temp fix. We are monitoring for stability. A tech is en route to site and the ETA is 20minutes. Core team will work remotely with the tech to implement a permanent fix.
Update 03/01/2025 @ 08:33am
-- We discovered that there is an the underlying fault that triggered the juniper issues which had not yet been identified. Our change last night resolved the juniper issues, and with the junipers stabilized we have managed to identify the underlying issue. Core team is now working on resolving the underlying issue.
Update 03/01/2025 @ 05:32am
-- The routers have been rebooted and the Core team is busy performing some checks. Further updates will be shared.
Update 03/01/2025 @ 04:59am
-- The Core team has completed moving over some services to the new routers and are now performing a reboot of the routers, once the reboot has been completed they will conduct some checks
Update 03/01/2025 @ 03:03am
-- New device configurations are still underway, some services have been moved over to the new routers and we are expecting to perform a reboot once we have completed enough to balance the load. Further updates will be provided as we progress.
Update 03/01/2025 @ 00:21am
-- New hardware devices have been installed and CORE teams are busy setting up and installing configs. Once completed we will perform a reboot under a more balanced load and then perform checks.
Update 02/01/2025 @ 22:58pm
-- The installation process has begun and is estimated to take approximately 2 - 3 hours to complete. Afterward, we'll conduct additional checks on sample sets to check for stability while the Engineering team reviews the new data.
Update 02/01/2025 @ 21:34pm
-- We are currently experiencing reoccurring impact affecting some services. The juniper investigations into a permanent fix are still ongoing. In the interim, our core team is installing two additional juniper routers that will enable us to balance load, and reboot the problematic junipers under less strain. Techs are already on-site completing the pre-work
Update: 02/01/2025 @18:56pm:
-- Stability has been introduced for some clients through traffic adjustments and the shutdown of two backhauls, implemented by the Engineering Team. Efforts are ongoing with the Vendor and Frogfoot Engineering to investigate and implement a permanent solution.
Update: 02/01/2025 @17:03pm:
-- T3 is pulling logs to continue to investigate. More updates to follow.
Update: 02/01/2025 @15:16pm:
-- Additional logs collected by the Vendor indicate that there is an underlying issue on the FPC's that needs to be resolved before any firmware upgrade is implemented. It is highly likely that resolving this underlying issue will also resolve the intermittency issues and negate the need for any additional firmware upgrades. Vendor and Frogfoot Engineering Teams are actively troubleshooting and we will provide more updates as they progress.
Update: 02/01/2025 @13:41pm:
-- Team is still busy with an internal investigation
-- We are sourcing the latest updates from the team
-- Teams is still finalizing the checks for the upgrade developed with the Vendor. Some risks have been identified which could cause further impact and the Vendor has added a more senior engineering team to assist with this troubleshooting. More updates will be provided in the next 30 - 45 minutes.
Update: 02/01/2025 @13:28:
Based on the insights from Frogfoot's vendor and ongoing analysis, they are now preparing to implement a targeted software upgrade on our core routers. This upgrade is being carefully planned to ensure network stability while addressing the problem. Each upgrade will take approximately one hour, with minimal additional impact to current services.
Update: 02/01/2025 09:58am:
-- The upgrade developed jointly with the Vendor has been prepared and is currently undergoing final checks. Once ready we will begin the upload to the affected devices which will take around an hour per device (2 devices in total). We will provide more accurate timeframes as we begin the processes.
Update: 02/01/2025 08:23am:
-- We have identified a bug that is causing the impact, and we are going to upgrade the 2 Junipers in CPT. We are currently in planning for the upgrade. More updates to follow.
Update: 02/01/2025 06:58:
-- The fault is still being investigated. There is currently a joint troubleshooting with the hardware vendor. We are awaiting an update on the outcome of the investigation.
-- Sourcing an update from the teams. Feedback will be shared as soon as possible.
-- Investigation still ongoing. Fault has been escalated internally on juniper side.
-- Teams are in the background continuing with isolation and resetting individual routing instances that are showing flooding with the aim of reducing service impact.
Update: 02/01/2025 06:46am:
-- Sourcing an update from the teams. Feedback will be shared as soon as possible.
-- Investigation still ongoing. Fault has been escalated internally on juniper side.
-- Teams are in the background continuing with isolation and resetting individual routing instances that are showing flooding with the aim of reducing service impact.
-- Investigations are still ongoing internally. More updates will be shared.
Update: 02/01/2025 00:20am:
-- The fault is still being investigated. There is currently a joint troubleshooting with the hardware vendor. We are awaiting an update on the outcome of the investigation.
Update: 01/01/2025 23:41pm:
-- ISPs are monitoring services and sharing samples of the clients that are still impacted
-- T4 is working through the samples to isolate the issue
Update: 01/01/2025 21:06pm:
-- Techs have received the replacement line card and are en route to site
-- Tech is on site, installation will commence shortly
Update: 01/01/2025 21:06pm:
-- The spare line card has been collected at our office and is enroute to site for installation
-- Team is onsite with the line card waiting on core team to assist with the installation, we are pending ETTR for the installation. Further updates will be shared
-- The tech is busy changing the line card
-- The replacement line card which was installed turned out to be faulty. We are in the process of sourcing a new one from the warehouse.
Update: 01/01/2025 14:00:
--We are observing an ongoing flap on the Juniper, our core team is currently investigating
--Techs are onsite hard rebooting the junipers to try and resolve the issue
--The reboot of R2 is now complete. R1 has just restored, we are waiting for R2 to fully stabilize before bringing the interfaces up. Thank you for your continued patience as we work through this process. We will keep you updated on further developments
Update: 01/01/2025 09:21:
-- Logged to T3 team for investigation
--Teams are currently investigating impact.
--Core team has been alerted to investigate further. More updates to be shared
--Busy renaming routing instance for additional flooding that the team is seeing on the network.
--Tier 3 team still seeing flooding on the network, Core is assisting with investigating further.
Affected Areas
Rondebosch, Cape Town CBD, Bellville, Rynfield, Pinelands,
Montague Gardens, King Williams Town, Matroosfontein, Knysna, Mossel Bay,
Goodwood, George, Blue Downs, Kempton Park, Franschhoek, Blouberg,
Somerset West, Atlantis, Richards Bay, Swellendam, Yzerfontein,Port Elizabeth, East London:
#Capetown #EasternCape #Thinkspeed #Frogfoot #Outage
Frogfoot Official Statement: 02/01/2025 @ 13:28pm
