DCC Maintenance Information
The DCC environment operates on a rolling maintenance procedure that occurs every three months. Previously, the DCC was taken completely offline for two maintenance periods annually. The rolling maintenance process allows the DCC to stay online for researchers while aiming to maintain at least fifty percent of researcher-owned hardware where possible.
Maintenance procedures
Communications
We will post notices in advance of an upcoming maintenance in order to prepare and give researchers a chance to reach out to us to adjust schedules in case the upcoming maintenance causes conflicts with any time-sensitive work. We will post notices in multiple locations, including:
- A notification banner at the top of the DCC and the Open OnDemand websites detailing the upcoming maintenance and where to look for additional information
- A message of the day on ssh login for the DCC
Reservations
To prepare for our rolling maintenance, we will create reservation times on the cluster. What this means it that we will reserve specific time blocks for nodes where no jobs can be submitted as we will be performing maintenance on those nodes in that given time block. We will do this well in advance of the actual maintenance times to make sure that long-running jobs are not affected.
Things to watch for:
- Some jobs may remain pending even when nodes appear idle. This most commonly affects jobs in lab-owned parititions that do not specify
--timeand rely instead on the 30-day partition limit. Jobs that cannot complete before the reservation window will show a state like this:
* Recommended action: Explicitly request a wall time that reflects actual runtime.
Example: batch job for 2 days
Example: interactive job for 4 hours
How to check progress and availability
If you would like to check on the availability of nodes within a specific partition, you can utilize the dcc-maint.sh script available to all users.
(base) rm145@dcc-login-04 ~ $ dcc-maint.sh -p common -t 7:00:00
common -- 86 nodes (MaxTime 30-00:00:00, DefaultTime 1-00:00:00)
9 node(s) are in maintenance right now and cannot run anything.
Requested walltime 7:00:00 (job would end 2026-09-05T00:28)
Eligible nodes 77 of 86
Unavailable 9 (maintenance starts before then)
It can start, on 77 of 86 nodes. Expect a longer queue wait than
usual, since the other 9 are off-limits for a job this long.
If you omit the -t flag, you would instead be given a table of time options for jobs to be submitted to that partition:
(base) rm145@dcc-login-04 ~ $ dcc-maint.sh -p common
common -- 86 nodes (MaxTime 30-00:00:00, DefaultTime 1-00:00:00)
9 node(s) are in maintenance right now and cannot run anything.
If your job ends Request at most Eligible nodes
before... (from now)
2026-09-08T00:00 3-06:30:25 77 of 86
(after all blocks) n/a 68 of 86
Each row: request no more than the middle column and your job can be
scheduled on that many nodes. Longer requests drop to the row below.
Emergency maintenance
With the increased usage of AI, we have seen more security vulnerabilities and exploits emerge. While we will do our best to make sure users of the DCC have advanced notice of any maintenance or planned downtime of resources, we must prioritize the safety and integrity of the cluster. As such, we may be forced to enter emergency maintenance for patching. If this occurs, we will try to update users as quickly as we can but we may need to take nodes offline for patching. Please see our Terms of Service for more details.