NOTE: This article holds archived status update messages. If you'd like to know about the current status of Canvas, click here.

----

CANVAS STATUS UPDATE ARCHIVES

In reverse chronological order--newest first, oldest at the bottom

----

Tuesday, April 9, 2013

As of 9:40 PM MT on April 9th, we've resolved the issue causing some users to sporadically receive errors when attempting to access pages in Canvas. We'll provide an incident report tomorrow. Please contact Canvas Support if your users experience anything else out of the ordinary this evening.

Tuesday, April 9, 2013

Starting at 8:50 PM MT on April 9th, less than 10% of Canvas users began receiving sporadic error messages when trying to access some pages in Canvas. Our operations team is reviewing the issue; we'll post an update shortly.

Monday, March 11, 2013

As of 11:30 PM MT on March 8, Canvas was functioning as normal again for all users. (We emailed admins at that time.) This morning, we sent an incident report to admins at institutions where users were impacted. Less than 0.5% of page view requests for users at these institutions were affected.

Friday, March 8, 2013

Beginning at about 10:15 PM MT, a small percentage of Canvas users received error messages when trying to access pages in Canvas. Our engineering team is currently working to solve this problem. Errors are appearing intermittently. In most cases, if users reload or try again to access the page on which they received the error, they will be successful. We'll post an update shortly.

Tuesday, February 26, 2013

Between about 11:40 AM and 12:10 PM MST today, a very small fraction of Canvas users (<1%) received page error messages when trying to access or use Canvas.

Friday, February 22, 2013

On February 21, starting at 7:03PM MST, Canvas was unavailable to most customers for approximately 25 minutes. The root cause of the issue was failure of a drive in one of the databases that provides cluster-wide metadata, but the outage was extended by complications with another database component. Canvas admins at each institution have been provided with some more technical details of the outage, as well as our plans for future mitigation. Please contact the local Canvas admin at your institution if you'd like more information on this incident. 

Thursday, February 21, 2013

We found an issue with one of the databases that serves metadata for all of Canvas. Our operations team has routed around the issue and are investigating the root cause. Canvas was back online at 7:30pm MST. We will have a post mortem by end of day tomorrow.

Thursday, February 21, 2013

Our monitoring systems noticed service interruption beginning at 7:05 pm MST today. Our operations and engineering teams are working on the issue currently. We will have an update by 8:00 pm MST.

Wednesday, February 13, 2013

Between 10:30 PM and 10:40 PM MT this evening, some Canvas users could not log in. We are gathering information on the cause of the issue and will provide an update by email to administrators at affected institutions tomorrow (2/14).

Wednesday, February 13, 2013

Between 8:54 PM and 8:58 PM MT this evening, less than 10% of Canvas users could not log in. Access is now restored for all users.

Tuesday, February 12, 2013

Between 10:05 PM and 10:15 PM MT this evening, some Canvas users could not log in. We will provide an update to affected institutions tomorrow about today's brief access issues.

Tuesday, February 12, 2013

Starting at about 6:45 PM MT this evening, about 10% of users could not log into Canvas for between five and ten minutes. 

Tuesday, February 12, 2013

Starting at about 12:00 PM MT, about 10% of users could not log into Canvas for between five and ten minutes. 

Thursday, January 31, 2013

For five minutes (from 11:57 AM MT to 12:02 PM MT) this morning, less than 20% of users could not log into Canvas.

Wednesday, January 23, 2013

Starting at about 4:49 PM MT this afternoon, about 20% of users could not log into Canvas for between five and ten minutes. Our Operations team quickly resolved the problem and access was restored for all users shortly after 5:00 PM MT.

Sunday November 25, 2012 8:33 PM MDT

Page loads are back to normal in Canvas. Thanks for your patience! Please contact Support if you see anything out of the ordinary.

Sunday November 25, 2012 8:00 PM MDT

Slowness continues for Canvas users. Some have reported page timeouts, too. Our Ops team continues to work on the problem.

Sunday November 25, 2012 7:15 PM MDT

Some users are experiencing slow page loads in Canvas right now. Our Operations team is working on it; we'll post updates as we learn more.

Saturday November 17, 2012 2:30 AM MDT

Canvas experienced a ten-minute outage Saturday morning between 2:30 and 2:40 AM Mountain Time. Failover and backup procedures functioned as designed and service was quickly restored. No data was lost.

Monday November 5, 2012 5:00 PM MDT

The beta refresh is complete. You should now be able to log into your beta instance of Canvas. Thanks for your patience.

Monday November 5, 2012 11:15 AM MDT

**NOTE: This status update refers to beta instances of Canvas only.**

We're continuing to work on the beta refresh. It's probable that beta sites will not be accessible for the rest of the day. We'll post an update when they're accessible again.

Please note that, since we performed a release on Saturday, code in beta will match code in production this week.

Monday November 5, 2012 6:45 AM MDT

**NOTE: This status update refers to beta instances of Canvas only.**

Our regularly scheduled beta site refresh is taking longer than usual this morning. Beta sites will not be accessible until the refresh is complete--probably by 9:00 AM MT. We'll post another update when the refresh is complete.

Wednesday October 31, 2012 8:30 AM MDT

Crocodoc, the third-party document previewing tool used for some file types in Speedgrader, will be performing maintenance on Sunday, November 4, at 9am MT. The maintenance is scheduled to last 30 minutes. Any homework submitted during the maintenance window will be sent to one of the other previewing tools built into Canvas (Scribd or Google Preview).

Friday October 26, 2012 2:30 PM MDT

Planning ahead for Hurricane Sandy

Overview

The East Coast is about to be hit by a major hurricane, called “Hurricane Sandy.” Meteorologists are predicting that this will have a serious impact in the Northeast and/or Mid-Atlantic regions, which is where Canvas is currently hosted by Amazon Web Services. Current predictions are that it will land as soon as Sunday night, going through Monday and Tuesday.

There is more information about Hurricane Sandy here: http://www.weather.com/news/weather-hurricanes/tropics-scenarios-us-threat-20121022

If the storm develops as predicted, we anticipate that at least some portion of AWS will be negatively affected, perhaps even with complete power losses in some data centers. We will be closely monitoring the situation and working around any failures as quickly as possible.

We are also preparing for the possibility that the entire region becomes unavailable. Per our Disaster Recovery Plan, if it becomes clear that the AWS Eastern region will be down for an extended period, we will move Canvas traffic to data centers on the West Coast.

We will keep you notified in the event of any disruption to Canvas services.

LDAP IP Address Update (LDAP login customers only)

As we communicated in the most recent Canvas Status Update, we are updating our list of IP addresses that needs to be whitelisted by firewalls for LDAP access. This is in order to improve login performance for schools that have LDAP configured. This is also required in the event that we need to move traffic over to the West Coast.

The new and complete list of IP addresses that should be whitelisted for Canvas LDAP traffic will be emailed directly to the designated Admin at each Canvas institution.

We encourage Admins to have your institution’s firewall administrator add these IP addresses to the whitelist for your LDAP server as soon as possible. We do not plan to begin using the newly added IP addresses for at least four weeks, and we will notify you again before that happens. In the unlikely event that we are forced to move Canvas traffic to the West Coast, the new IP addresses will be in use immediately, and you will receive notification about that.

Wednesday October 24, 2012 5:00 PM MDT

We’ve had several problems with our chat feature over the past few months. Chat in Canvas is powered by a third-party tool, but it’s our responsibility to make sure it works. We are sorry for the frustration these problems have caused for students and instructors lately. This isn’t acceptable, and we’re going to fix it.
                
We are actively seeking a new chat tool. We will re-deploy chat by the end of the year at the latest, but we plan to have good news for you much sooner.

For now, we are turning off the chat tool in Canvas. We think it’s better to remove the feature entirely and tell you why than to leave it in when we can’t vouch for its performance.

We suggest using Etherpad collaborations (which include a chat component) as a temporary stand-in while we’re working on a new chat tool. This isn’t a perfect solution, but it should be an adequate workaround for many situations.

Monday October 22, 2012 2:00 PM MDT

Monday October 22, 2012 1:30 PM MDT

Monday October 22, 2012 12:15 PM MDT

Wednesday October 19, 2012 9:50 AM MDT

Users are not able to start or join conferences in Canvas this morning. Our engineers are working with Big Blue Button, our conference tool provider, on a solution. We’ll post updates as we learn more.

---

Wednesday September 26, 2012 11:05 AM MDT

Monday September 24, 2012 8:45 PM MDT

Saturday September 15, 2012 12:30 AM MDT

Our scheduled maintenance this evening was successful. Affected Institutions were notified last week and were affected for less than an hour tonight. Please contact Support with any questions or if anything appears out of the ordinary.

Tuesday September 11, 2012 1:00 AM MDT

Our Operations and Engineering teams have finished maintenance work addressing the slow page-loads some Canvas users experienced periodically yesterday. We’ll watch performance closely as traffic increases this morning.

Monday September 10, 2012 9:15 PM MDT

Monitoring shows the slowness reported by some of our users this evening has passed. We are working with our operations team to improve performance overall. Some of the improvements will require about an hour of maintenance for a handful of institutions this evening. Institutions to be affected will be contacted directly by their account manager. We will provide an update when maintenance is complete tonight.

Monday September 10, 2012 8:00 PM MDT

Monitoring and user feedback are showing that a portion of the Canvas community is experiencing slow page loads again. Our Operations and Engineering teams are working to address the short-term symptoms and plan next steps. We’ll post updates throughout the process. Thank you for your patience. Our next update will be posted at 9:00 PM MT or sooner.

Monday September 10, 2012 2:30 PM MDT

Canvas is now operating normally for all institutions. We’re keeping a close eye on system status and will provide another update later if necessary. Please contact Support if you experience anything out of the ordinary.

Monday September 10, 2012 1:15 PM MDT

Canvas has operated normally for most institutions since 12:30 PM MT. A few are again experiencing some slowness since about 1:00 PM MT. We will continue to monitor our systems closely and we’ll post another update in an hour (about 2:30 PM MT).

Monday September 10, 2012 12:15 PM MDT

Some Canvas users experienced sporadic slow page loads for about an hour starting at 11:15 AM MDT. Our operations team has taken some measures and page loads are now back to normal. We’ll provide an update on our status within an hour, by 1:15 PM MDT.

Friday September 7, 2012 10:00 AM MDT

We'll be doing maintenance on Canvas on Friday, 9/14 starting at 11 PM MT. The work we’re doing will only affect a few Canvas institutions, and any given institution should only be affected for an hour. We’ve contacted the LMS administrators at these institutions directly and provided specific maintenance window times.

This is part of our ongoing effort to scale the backend system and optimize the database. The changes we have made so far have worked as intended, and Canvas is running well. Our maintenance this weekend for the small percentage of our customers will keep us ahead of the curve.

Monday September 3, 2012 7:50 PM MDT

We'll be doing an hour of maintenance on Canvas tonight starting at 11 PM MT. No Canvas schools will be affected; only teachers who are evaluating Canvas for free on their own ("Free For Teacher" users) and their students.

Sunday August 26, 2012 7:35 AM MDT

We finished two rounds of optimization maintenance over the weekend: one Saturday morning and one just a few minutes ago. Both were successful. Institutions that were affected during maintenance were back to normal in less than an hour both days. Please contact Support if you see anything out of the ordinary.

Friday August 24, 2012 2:30 PM MDT

Just a quick update to recap the issues this week: what we’ve done so far and what we’re doing next.

We continue to make optimizations to our database cluster. We’ll be taking next steps Saturday and Sunday morning. A few institutions will experience brief outages as a result, and we’ll talk to them directly in advance. We’ll share another update here, via email to the Canvas Admin list, and on Twitter both days to let you know how things went.

Wednesday August 22, 2012 11:45 PM MDT

Canvas customers,

Our work this evening to increase database parallelism was successful. We will continue to make enhancements and closely monitor system performance. Please contact Support if you experience any issues.

Wednesday August 22, 2012 5:00 PM MDT

Earlier today, some institutions experienced intermittent slowness and time-outs while accessing Canvas. We’re monitoring the status of the system very closely and dealing with these issues as they appear.

Tonight, we're taking the next major step in the reconfiguration that Josh talked about in his blog post (http://voice.instructure.com/blog/bid/210688/A-Bad-Day-for-Canvas). We’ll directly notify institutions that will be affected by the work we're doing.

We will provide another update when the work is done.

Wednesday August 22, 2012 1:15 PM MDT

Our monitoring services show Canvas should be functioning normally for all users now.  Please let us know if you still experience any slowness or timeouts.

Wednesday August 22, 2012 12:40 PM MDT

Our operations team is working hard to address slowness and time-out issues some users have been experiencing throughout the past hour. A long term fix is being worked on and will be out by the end of the week - http://voice.instructure.com/blog/bid/210688/A-Bad-Day-for-Canvas …

We will address this intermittent slowness in our update at 5:00 PM MDT.

Wednesday August 22, 2012 11:40 AM MDT

We've received comments from some customers of slowness and not being able to connect to Canvas.  We're currently investigating and will post when we have more info.

Tuesday August 21, 2012 5:00 PM MDT

Progress has been made today but we're still monitoring things very closely.  We're continuing to see improvements and we've got some additional adjustments scheduled for tomorrow night.  For details about what went wrong and what we're doing to fix it please see our CEO's blog post here:http://voice.instructure.com/blog/bid/210688/A-Bad-Day-for-Canvas

We'll send an email to our LMS Admin distribution list and post an update here before 5:00 PM MDT tomorrow with a schedule of any activities and potential impact.

Tuesday August 21, 2012 12:15 PM MDT

Canvas customers,

Canvas page load times are in a normal range with Canvas traffic higher today than the peak level yesterday.  The overnight changes were successful and we continue to execute our plans for additional work on the database cluster.  Until the work is complete there may still be periods of degraded performance--we'll be monitoring it very closely.

We will provide another update around 5:00 PM MDT.

Tuesday August 21, 2012 8:00 AM MDT

Canvas customers,

We monitored the hardware changes throughout the night and system performance has improved.  We're optimistic and believe these changes were the right first steps to recovery and will continue executing the plan to distribute load across our database cluster.  We'll have more details on that progress later today.

We will continue to monitor the system and inform our users of any issues and mitigation steps along the way.  We will provide another update around noon MDT.

Monday August 20, 2012 10:05 PM MDT

Canvas customers,

The database hardware migration was a success.  The system is up and running and page load times have been reduced.  We will continue to monitor the system throughout the night and as usage increases tomorrow morning.

While the hardware switch was a success we don't expect all performance issues to be resolved and we're actively working toward distributing customers across our database cluster, which we believe will have the greatest impact.  In the meantime we will look for long-running queries and other opportunities to optimize the system for best performance.

We will provide another update at 8:00 AM MDT with any additional actions to be taken tomorrow and an update on our database load balancing efforts.  If there are any substantive changes we will provide an update earlier.  

Monday August 20, 2012 8:05 PM MDT

Canvas customers,

Summary: We're switching over to new database hardware at 8:15 PM MDT.  There may be a very small amount of downtime (about a minute).

Canvas customers,

Thank you for being patient with us throughout the day.  We are working hard to add more capacity to our database servers and will provide another update by 8:00 PM MDT.

An update will also be provided on Twitter @canvassupport.

Monday August 20, 2012 2:55 PM MDT

Canvas customers,

A portion of our users are experiencing slow loads or timeouts on certain pages.  This problem was caused during our recent database reconfiguration in anticipation of this upcoming semester. Unfortunately, this configuration has not performed as expected and we are currently working to resolve this as rapidly as possible. We're deeply sorry for the customer experience this has created and apologize to all of our customers who are experiencing issues.  

We are in current contact with all affected customers and are posting updates on Twitter @canvassupport.