<\!DOCTYPE html> Retention Scheduler
Back to Documentation

Retention Scheduler

The retention scheduler is the automated engine that executes your retention policies. This guide covers how to configure, monitor, and troubleshoot the scheduler to ensure reliable email lifecycle management.

Scheduler Overview

The retention scheduler:

  • Runs automatically at configured intervals
  • Executes all active policies for each account
  • Handles errors gracefully with retry logic
  • Logs all activity for monitoring and troubleshooting
  • Provides manual control when needed

Screenshot: Retention scheduler interface

Scheduler Configuration

Basic Settings

Access scheduler settings at Retention → Scheduler:

Execution Schedule

  • Frequency: How often to run (daily recommended)
  • Start Time: When to begin execution (2 AM default)
  • Time Zone: Local time zone for scheduling

Execution Limits

  • Timeout: Maximum time per execution (30 minutes default)
  • Batch Size: Emails to process per batch (1000 default)
  • Max Retries: Retry attempts for failed operations (3 default)

Screenshot: Scheduler configuration settings

Advanced Configuration

Performance Tuning

  • Concurrent Accounts: Process multiple accounts simultaneously
  • IMAP Connections: Connection pool size
  • Memory Limits: Prevent excessive memory usage

Error Handling

  • Retry Delays: Time between retry attempts
  • Failure Notifications: Alert on persistent failures
  • Partial Failures: Continue processing other policies if one fails

Starting and Stopping the Scheduler

Manual Control

Screenshot: Scheduler status and controls

Start Scheduler

  1. Click "Start Scheduler" button
  2. Scheduler status changes to "Running"
  3. Next execution time is displayed

Stop Scheduler

  1. Click "Stop Scheduler" button
  2. Current execution completes
  3. No new executions begin

Force Stop

  1. Use "Force Stop" for emergency situations
  2. Interrupts current execution immediately
  3. May leave operations incomplete

Automatic Startup

Configure scheduler to start automatically:

On Application Start

  • Enable "Auto-start with application"
  • Scheduler begins when Mail-Rulez starts
  • Useful for production deployments

After Restart

  • Enable "Resume after restart"
  • Scheduler remembers previous state
  • Continues scheduled executions

Scheduler Status and Monitoring

Real-Time Status

The scheduler dashboard shows:

Current State

  • Status: Running, Stopped, or Error
  • Current Activity: What's being processed
  • Progress: Completion percentage
  • ETA: Estimated time to completion

Recent Activity

  • Last Execution: When scheduler last ran
  • Duration: How long execution took
  • Emails Processed: Total count
  • Policies Executed: Number of policies run

Execution History

View detailed execution history:

Screenshot: Scheduler execution history

Execution Log

  • Start/End Times: Full execution window
  • Policies Executed: Which policies ran
  • Success/Failure Count: Per-policy results
  • Performance Metrics: Timing and throughput

Trend Analysis

  • Execution Time Trends: Performance over time
  • Email Volume Trends: Processing volume changes
  • Error Rate Trends: Reliability metrics

Manual Execution

When to Use Manual Execution

  • Testing new policies before automation
  • Recovering from errors in scheduled runs
  • Processing backlog after downtime
  • One-time cleanup operations

Running Manual Execution

  1. Select Scope

    • All accounts and policies
    • Specific account only
    • Individual policy only

  2. Choose Options

    • Preview Mode: See what would happen without changes
    • Force Execution: Ignore normal timing constraints
    • Skip Verification: Bypass safety checks (advanced)

Screenshot: Manual execution options

  1. Monitor Progress
    • Real-time progress display
    • Live log output
    • Cancel option if needed

Manual Execution Examples

Test New Policy

Scope: Single Policy "Junk Cleanup"
Mode: Preview
Options: Default safety checks

Emergency Cleanup

Scope: All Policies for Account
Mode: Execute
Options: Force execution, skip recent constraints

Performance Test

Scope: Single Account
Mode: Execute
Options: Reduced batch size, extended timeout

Scheduler Performance Optimization

Tuning Execution Settings

Batch Size Optimization

Small batches (100-500):

  • Lower memory usage
  • Better error isolation
  • Slower overall processing

Large batches (1000-5000):

  • Faster processing
  • Higher memory usage
  • Risk of timeout on slow connections

Concurrent Processing

Single-threaded:

  • Sequential policy execution
  • Lower resource usage
  • Simpler error handling

Multi-threaded:

  • Parallel account processing
  • Higher throughput
  • More complex coordination

IMAP Connection Optimization

Connection Pooling

  • Reuse connections across policies
  • Reduce authentication overhead
  • Improve overall performance

Connection Limits

  • Respect server connection limits
  • Avoid overwhelming IMAP servers
  • Balance throughput with stability

Timeout Configuration

  • Network timeout: 30 seconds
  • Operation timeout: 5 minutes
  • Total execution timeout: 30 minutes

Resource Management

Memory Usage

  • Monitor memory consumption
  • Limit email cache size
  • Clear caches between policies

Disk Usage

  • Log file rotation
  • Temporary file cleanup
  • Database maintenance

Error Handling and Recovery

Common Scheduler Errors

Authentication Failures

  • Expired passwords
  • Changed account settings
  • Two-factor authentication issues

Network Problems

  • IMAP server downtime
  • Network connectivity issues
  • Firewall restrictions

Resource Constraints

  • Insufficient memory
  • Disk space exhaustion
  • CPU resource limits

Error Recovery Strategies

Automatic Recovery

Retry Logic:

  1. Wait for configurable delay
  2. Retry failed operation
  3. Increase delay on subsequent failures
  4. Give up after max retries

Circuit Breaker:

  1. Detect persistent failures
  2. Stop retrying temporarily
  3. Resume after cool-down period
  4. Prevent resource exhaustion

Manual Recovery

When automatic recovery fails:

  1. Check error logs for root cause
  2. Resolve underlying issue
  3. Reset scheduler state if needed
  4. Resume normal operation

Failure Notifications

Configure alerts for:

  • Persistent Failures: Multiple retry failures
  • Timeout Errors: Executions taking too long
  • Resource Errors: Memory or disk issues
  • Authentication Errors: Account access problems

Scheduler Logs and Diagnostics

Log Levels

INFO Level (Default)

  • Execution start/stop
  • Policy results summary
  • Performance metrics

DEBUG Level (Troubleshooting)

  • Detailed operation steps
  • IMAP command/response
  • Internal state changes

ERROR Level (Problems Only)

  • Failed operations
  • Exception details
  • Recovery attempts

Log Analysis

Performance Analysis

2024-07-01 02:00:15 - Scheduler started
2024-07-01 02:00:16 - Processing account: [email protected]
2024-07-01 02:01:23 - Policy "Junk Cleanup": 127 emails processed in 67s
2024-07-01 02:02:45 - All policies completed in 150s

Error Diagnosis

2024-07-01 02:00:15 - Authentication failed for [email protected]
2024-07-01 02:00:45 - Retrying with fresh connection
2024-07-01 02:01:15 - Retry 1 failed: Connection timeout
2024-07-01 02:02:15 - Retry 2 failed: Server busy
2024-07-01 02:03:15 - Giving up after 3 attempts

Diagnostic Tools

Health Check

  • Verify scheduler components
  • Test account connections
  • Check policy configurations

Performance Profiler

  • Identify slow operations
  • Analyze resource usage
  • Optimize execution flow

Connection Tester

  • Test IMAP connectivity
  • Verify authentication
  • Check folder access

Best Practices

Scheduling Strategy

  1. Off-Peak Execution

    • Run during low-usage hours
    • Avoid peak email activity
    • Minimize user impact

  2. Staggered Execution

    • Don't process all accounts simultaneously
    • Spread load over time
    • Prevent server overload

  3. Regular Execution

    • Daily execution recommended
    • Consistent processing schedule
    • Predictable system behavior

Monitoring and Maintenance

  1. Daily Checks

    • Verify scheduler is running
    • Check for error alerts
    • Review processing volumes

  2. Weekly Reviews

    • Analyze execution trends
    • Check performance metrics
    • Update configurations as needed

  3. Monthly Audits

    • Review all policies
    • Optimize performance settings
    • Plan capacity upgrades

Troubleshooting Workflow

  1. Check Scheduler Status

    • Is scheduler running?
    • When did it last execute?
    • Are there active errors?

  2. Review Recent Logs

    • Look for error patterns
    • Check execution times
    • Verify policy results

  3. Test Components

    • Test account connections
    • Verify policy configurations
    • Run manual execution

  4. Apply Fixes

    • Update account credentials
    • Adjust timeout settings
    • Modify policy parameters

Integration with Monitoring

External Monitoring

Health Check Endpoint

GET /api/retention/scheduler/health

Response:

{
  "status": "healthy",
  "last_execution": "2024-07-01T02:00:00Z",
  "next_execution": "2024-07-02T02:00:00Z",
  "policies_active": 12,
  "errors": []
}

Metrics Export

  • Prometheus metrics available
  • Execution time histograms
  • Error rate counters
  • Active policy gauges

Alerting Integration

Alert Conditions

  • Scheduler stopped unexpectedly
  • Execution time exceeds threshold
  • Error rate above normal
  • No successful execution in 24 hours

Notification Channels

  • Email alerts to administrators
  • Slack/Discord notifications
  • PagerDuty for critical issues
  • Custom webhook integrations

Next Steps

With scheduler configured and running:

  1. Recovery Procedures - Learn to restore deleted emails from trash
  2. Advanced Retention - Complex retention scenarios and policy combinations
  3. Monitoring Guide - System-wide monitoring
  4. Troubleshooting - Solve retention problems