Update - We are continuing to monitor for any further issues.
Sep 20, 10:24 PDT

Update - We are seeing some services restored and are working to reconfigure and reset all our equipment.
Work in the Vancouver and Newark datacenters is underway, and techs will reach out to each location to restore and confirm connectivity.

Oppy staff: if you are available today and need system access, or have worked with the tech team to change a configuration on your laptop/PC, could you please email ITHelp@oppy.com with your contact number (telephone number) so we can reach out individually in priority order throughout the day today. Also, if you are available to go into an Oppy office to assist in resetting equipment, please note that in your email.
Sep 20, 00:35 PDT

Update - Information from our Telco is that all repairs are complete and all underlying services should be restored.

We are working to reconfigure and test our datacenter operations and bring normal services up. Will update as we restore each area.
Sep 20, 00:13 PDT

Update - This will be the last update for this evening unless there is a significant change.
The Telco is working on a low signal level between Everett, WA and Bellingham, WA.
Unless the services are restored overnight, no further updates will be posted until early Sunday morning.
Sep 19, 17:46 PDT

Update - The Telco Network Operations Center has found the location of a bad fiber splice that was made/damaged whilst they were fixing another issue yesterday. A splicing technician is on-site and making the repair now.

Optimo is available for staff that are physically in one of the offices we have brought online (Vancouver, Newark, Manfredi, WCT and CPS), and plugged into the network via a wired connection (not wifi).

Citrix is currently not working for out-of-office Optimo users. The resolution we thought was working is still not allowing a connection. We are attempting a few more configurations to get this working.

Discussion planning for full and permanent cutover to Newark secondary datacenter is complete. Given the risk and technical efforts required, and the current status of our Telco in repairing this current issue, and the requirement to turn off the Vancouver datacenter to begin the transition, we will continue on with Vancouver and not switch to the D/R site.

Our EDI synchronization discussion has determined we will hold for a couple more hours before looking further at this due to the risk with some trading partners and the way their network interacts with ours. The concern is in losing transactions. Given how close the Telco appears to be in fixing this issue, we will hold on making any EDI changes at this time.

NOTE:
Mail is available via https://mail.oppy.com (webmail)

Note, after you enter your User id and Password, you need to approve the "Authenticator" notification "You have received a sign in verification request" that will pop up on your smartphone. Suggest you have your smartphone in front of you and unlocked before you try to sign in to webmail. Mail via your smartphone will not be available until our main network is restored.
Sep 19, 16:06 PDT

Update - Mail is available via https://mail.oppy.com (webmail)

Note, after you enter your User id and Password, you need to approve the "Authenticator" notification "You have received a sign in verification request" that will pop up on your smartphone. Suggest you have your smartphone in front of you and unlocked before you try to sign in to webmail.

Sites online are Vancouver, Newark, Manfredi, WCT and CPS. Looking into Chino. Calgary office itself does not have a working internet connection so reaching out to individual staff.

There was a problem today in getting Citrix to operation on Oppy computers, but a resolution has been found and working to bring online priority staff now.

Discussion planning for full and permanent cutover to Newark secondary datacenter is complete. Final decision to be made at 4 pm, along with an EDI synchronization discussion.
Sep 19, 14:03 PDT

Update - Telco repair crews are now on-site at the third location in Washington state.
This is good news and hopefully we should soon get an ETA of the fix.
Sep 19, 12:52 PDT

Update - Update received from Telco. Based on the information provided I would make an educated guess that we should have more concrete information on time to repair in about 1 hour (1:30 pm Pacific). It appears the delay in finding this last issue has been resolved and work is now progressing on the actual outage itself.

Vancouver, Newark, Manfredi, CPS, offices are connecting to our secondary datacenter in Newark, and individual staff and locations have been set up with Citrix to access Optimo (tech staff are all currently engaged with end users so will update details when they are available).

Email is currently being synchronized from outside Oppy into our Vancouver datacenter. Staff in Vancouver/Newark/Manfredi offices (likely CPS also-unconfirmed) should see mail start to flow, and mail within Oppy to other Oppy staff is working. Soon we will broadcast a message when all other staff can access email, but it is not ready just yet for that access.
Sep 19, 12:25 PDT

Update - Tech staff continue to bring staff and locations online via our secondary datacenter in Newark.
While this is underway, investigation is ongoing with regards to some potential risks to bringing email online. Options are currently being investigated.

The main involved Telco (Level3/CenturyLink/Lumen) management are engaged with the lower level fiber service providers to determine a current status. The remaining situation seems to involve more than simply damaged fiber optic cables, but no further details have been provided as of 10:50 a.m. (Pacific).
Sep 19, 10:49 PDT

Update - Continuing to wait on further update from Telco and field repair crews.

We have enabled some services in our secondary datacenter in Newark, and are routing traffic through that datacenter and into Vancouver. Completely switching to our Newark datacenter will be a decision made late this afternoon.
Leveraging a portion of our secondary datacenter is confirmed to work in restoring Citrix connections, and techs are working on connecting each person/site in priority order. A small configuration change is required to make this connection, so techs are reaching out individually to staff.

Once key staff and locations are online we will determine if external email can be enabled without degrading the performance of the Optimo/Citrix connections.
Sep 19, 09:55 PDT

Update - We received a somewhat ambiguous update from our Telco. They continue to communicate with the field crews to provide a deeper update and some type of estimated time of repair. Expecting an update in 10 minutes.

Meanwhile our 3rd backup network was brought up overnight and techs have been doing configuration to bring Citrix access up, and focus on specific locations. For now we are prioritizing getting Optimo access either in-office or via Citrix working. Email within Oppy should work for those physically in Vancouver/Newark offices, but we have not worked on external email in/out until we are sure that this 3rd backup network can handle the load from Optimo. Once that is stable we will look at turning up other services.
Sep 19, 08:55 PDT

Update - Optimo is accessible "in office" at Newark, Manfredi, and Vancouver offices.
Next we are working to give access in priority order WCT, LVC, GOL, RAN, VIS, CGY, CPS, MAR, MZN, AWS, FRO.
This is all in parallel with waiting for a network update at 8:30 a.m. today.
Sep 19, 07:42 PDT

Update - Optimo is accessible "in office" at Newark, Manfredi, and Vancouver offices. Next we are working to give access to TWC, then CPS, PIC and Calgary staff. This is all in parallel with waiting for a network update at 8:30 a.m. today.
Sep 19, 07:39 PDT

Update - The Network Operations Center expects an update from the field office working on the repair to give an update soon after 8:30 am (Pacific).
Sep 19, 06:56 PDT

Update - A second fiber cut location was located at 2 am (Pacific) and crews were onsite to to repair. As of 6:15 am (Pacific) the Telco is working to contact the local on site telco for an update.
Throughout the night we have been looking at several other possible options to restore some services in the interim. We are pursuing those options while we await the repair to be completed.
Sep 19, 06:24 PDT

Update - After the fiber repair, the network operations center still did not see traffic flowing.
Further investigations appear to suggest a second damaged site that technicians are now at attempting to diagnose and repair.
Sep 18, 19:39 PDT

Update - From Telco:
The fiber has been repaired and service validations are currently in progress. Once all services are confirmed restored and stable, a final notification will be sent.

Oppy services should be back up very soon.
Sep 18, 17:38 PDT

Update - As of 4:45 pm (Pacific) from Telco: Excavation teams and replacement fiber have arrived on-site, and excavations have commenced. Excavation crews expect the damaged fiber to be exposed within the hour. The fiber damage has been identified due to 100 feet of the fiber being removed by an unknown source.
Sep 18, 16:47 PDT

Update - From Telco 3:20 p.m. (PST): The local provider has isolated the fiber issue 800 feet from the test site in Everett, Washington. Cable crews are on-site standing by preparing for excavators to arrive and expose the damaged fiber and further assess the damage. The estimated time of arrival provided for excavation crews is 00:00 GMT (5 p.m. Pacific). The local provider has investigated to migrate services; however, no options are available.
Sep 18, 15:20 PDT

Update - Update from the Telco company on their portion of the fiber span from Vancouver to Seattle, WA. A technician is currently at the location and trying to diagnose the underlying cause of the problem. From their update it seems to be more complicated than a simple "cut fiber line".
Sep 18, 14:16 PDT

Update - Telco company is working to repair the fiber cut and restore network services. This affects Optimo, Email, and all remote connections into the Vancouver datacenter. ETA is unknown, but in the past this type of repair has taken several hours. A guess (estimate) would be later this afternoon (Sept 18) for a repair.

Telco generally gives update every 60 minutes. Most recent update is they are awaiting status information from the field engineers involved in the repair.

*** CASCADED EXTERNAL NOTES 18-Sep-2020 18:27:38 GMT From CASE: 19699046 - Event
The Transport Network Operations Center is currently communicating with the local provider who is working to obtain an update from the underlying provider.
Sep 18, 11:39 PDT

Update - We are continuing to monitor for any further issues.
Sep 18, 11:37 PDT