Maintenance Windows

This document defines optimal maintenance windows for system operations that require minimal interference from scheduled cron jobs.

Quick Reference

Window	Time (UTC)	Use Case	Cron Jobs Affected
Daily (night)	01:00-01:59	Low-traffic maintenance	Few daily jobs
Weekly (night)	Sunday 00:05-02:59	Major maintenance, deployments	Minimal (1 job on Sundays)
Daily (working hours)	16:00-16:59	Maintenance requiring dev availability	Zero daily/weekly jobs

Window Details

1. Daily Night Window (01:00-01:59 UTC)

Best for: Routine maintenance, database operations, cache clearing

Characteristics:

• 🟢 Lowest user activity period
• 🟨 Few scheduled cron jobs
• 🔻 1-hour duration

Jobs running in this window: See "Jobs by Hour" section below.

2. Weekly Night Window (Sunday 00:05-02:59 UTC)

Best for: Major deployments, infrastructure changes, long-running migrations

Characteristics:

• 🟢 Absolute minimum user activity (Sunday 01:00 UTC is the quietest hour)
• 🟢 Only 1 cron job scheduled on Sundays in this range
• 🟢 3-hour duration allows for complex operations

3. Working Hours Window (16:00-16:59 UTC)

Best for: Maintenance requiring immediate developer response

Characteristics:

• 🟢 Zero scheduled daily/weekly cron jobs
• 🟢 Developers available for monitoring and rollback
• 🔻 Higher user activity than night windows

Catch-Up Logic

Critical commands should implement catch-up logic to recover from missed runs during maintenance windows.

How it works: The command stores its last run timestamp in a non-volatile cache. On next run, it uses that timestamp as the starting point for querying records (with a max lookback limit to prevent huge queries). This ensures no records are missed if runs are skipped during maintenance.

Commands with Catch-Up Logic

• membership:individual:trial:capture-and-upgrade (306de0c)

Commands Needing Catch-Up Logic

Commands running during maintenance windows that process time-sensitive data should be reviewed:

• courses:grading:notify (01:47 daily) - description says "must be executed daily, otherwise we will miss some grades"
• notification:postcard:welcome (01:51 daily) - finds members signed up the last day

Before Scheduling Maintenance

1. Check current jobs: Review cron__jobs table for the specific hour
2. Identify critical jobs: Payment processing, time-sensitive notifications
3. Verify catch-up logic: Ensure critical commands can recover from missed runs
4. Communicate: Notify stakeholders in advance
5. Monitor: Watch for failed jobs after maintenance ends

Appendix: User Activity Data

Based on 4 years of course__quiz_answers.created_at data:

Quietest Hours (Total Quiz Answers):

1. Sunday 01:00 - 21,625 answers (absolute minimum)
2. Saturday 01:00 - 22,059 answers
3. Sunday 02:00 - 22,685 answers

Weekly Pattern:

• Sunday: Lowest activity (peaks at ~70k vs 147k on weekdays)
• Saturday: Second lowest (~62k peak)
• Weekdays: Highest activity, peak at 13:00-15:00 UTC

Daily Pattern:

• 00:00-05:00 UTC: Consistently lowest activity
• 13:00-15:00 UTC: Peak activity on weekdays

---

Last Updated: February 2026

Next Review: Annually or when significant user pattern changes are observed