diff --git a/docs/admin/monitoring/connection-logs.md b/docs/admin/monitoring/connection-logs.md index b69bb2db186a8..210ca76d740cf 100644 --- a/docs/admin/monitoring/connection-logs.md +++ b/docs/admin/monitoring/connection-logs.md @@ -106,6 +106,14 @@ connection log entry, when `code-server` is opened: [API] 2025-07-03 06:57:16.157 [info] coderd: connection_log request_id=de3f6004-6cc1-4880-a296-d7c6ca1abf75 ID=f0249951-d454-48f6-9504-e73340fa07b7 Time="2025-07-03T06:57:16.144719Z" OrganizationID=0665a54f-0b77-4a58-94aa-59646fa38a74 WorkspaceOwnerID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa WorkspaceID=3c0b37c8-e58c-4980-b9a1-2732410480a5 WorkspaceName=dev AgentName=main Type=workspace_app Code=200 Ip=127.0.0.1 UserAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" UserID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa SlugOrPort=code-server ConnectionID= DisconnectReason="" ConnectionStatus=connected ``` +## Data Retention + +Coder supports configurable retention policies that automatically purge old +Connection Logs. To enable automated purging, configure the +`--connection-logs-retention` flag or `CODER_CONNECTION_LOGS_RETENTION` +environment variable. For comprehensive configuration options, see +[Data Retention](../setup/data-retention.md). + ## How to Enable Connection Logs This feature is only available with a [Premium license](../licensing/index.md). diff --git a/docs/admin/monitoring/index.md b/docs/admin/monitoring/index.md index 996d8040b0129..61e27e7930607 100644 --- a/docs/admin/monitoring/index.md +++ b/docs/admin/monitoring/index.md @@ -22,3 +22,4 @@ Learn how to install & read the docs on the Coder deployment, regardless of your monitoring stack. - [Health Check](./health-check.md): Learn about the periodic health check and error codes that run on Coder deployments. +- [Connection Logs](./connection-logs.md): Monitor connections to workspaces. diff --git a/docs/admin/security/audit-logs.md b/docs/admin/security/audit-logs.md index 29e9fe7b6f6c8..913611af283df 100644 --- a/docs/admin/security/audit-logs.md +++ b/docs/admin/security/audit-logs.md @@ -132,8 +132,21 @@ log entry: > Audit Logs provide critical security and compliance information. Purging Audit Logs may impact your organization's ability > to investigate security incidents or meet compliance requirements. Consult your security and compliance teams before purging any audit data. -Audit Logs are not automatically purged from the database, though they can account for a large amount of disk usage. -Use the following query to determine the amount of disk space used by the `audit_logs` table. +### Data Retention + +Coder supports configurable retention policies that automatically purge old +Audit Logs. To enable automated purging, configure the +`--audit-logs-retention` flag or `CODER_AUDIT_LOGS_RETENTION` environment +variable. For comprehensive configuration options, see +[Data Retention](../setup/data-retention.md). + +### Manual Purging + +Alternatively, you can purge Audit Logs manually by running SQL queries +directly against the database. + +Audit Logs can account for a large amount of disk usage. Use the following +query to determine the amount of disk space used by the `audit_logs` table. ```sql SELECT diff --git a/docs/admin/setup/data-retention.md b/docs/admin/setup/data-retention.md new file mode 100644 index 0000000000000..68409ad93f866 --- /dev/null +++ b/docs/admin/setup/data-retention.md @@ -0,0 +1,169 @@ +# Data Retention + +Coder supports configurable retention policies that automatically purge old +Audit Logs, Connection Logs, and API keys. These policies help manage database +growth by removing records older than a specified duration. + +## Overview + +Large deployments can accumulate significant amounts of data over time. +Retention policies help you: + +- **Reduce database size**: Automatically remove old records to free disk space. +- **Improve performance**: Smaller tables mean faster queries and backups. +- **Meet compliance requirements**: Configure retention periods that align with + your organization's data retention policies. + +> [!NOTE] +> Retention policies are disabled by default (set to `0`) to preserve existing +> behavior. The only exception is API keys, which defaults to 7 days. + +## Configuration + +You can configure retention policies using CLI flags, environment variables, or +a YAML configuration file. + +### Settings + +| Setting | CLI Flag | Environment Variable | Default | Description | +|-----------------|-------------------------------|-----------------------------------|----------------|--------------------------------------| +| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (disabled) | How long to retain Audit Log entries | +| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (disabled) | How long to retain Connection Logs | +| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys | + +### Duration Format + +Retention durations support days (`d`) and weeks (`w`) in addition to standard +Go duration units (`h`, `m`, `s`): + +- `7d` - 7 days +- `2w` - 2 weeks +- `30d` - 30 days +- `90d` - 90 days +- `365d` - 1 year + +### CLI Example + +```bash +coder server \ + --audit-logs-retention=365d \ + --connection-logs-retention=90d \ + --api-keys-retention=7d +``` + +### Environment Variables Example + +```bash +export CODER_AUDIT_LOGS_RETENTION=365d +export CODER_CONNECTION_LOGS_RETENTION=90d +export CODER_API_KEYS_RETENTION=7d +``` + +### YAML Configuration Example + +```yaml +retention: + audit_logs: 365d + connection_logs: 90d + api_keys: 7d +``` + +## How Retention Works + +### Background Purge Process + +Coder runs a background process that periodically deletes old records. The +purge process: + +1. Runs approximately every 10 minutes. +2. Processes records in batches to avoid database lock contention. +3. Deletes records older than the configured retention period. +4. Logs the number of deleted records for monitoring. + +### Effective Retention + +Each retention setting controls its data type independently: + +- When set to a non-zero duration, records older than that duration are deleted. +- When set to `0`, retention is disabled and data is kept indefinitely. + +### API Keys Special Behavior + +API key retention only affects **expired** keys. A key is deleted only when: + +1. The key has expired (past its `expires_at` timestamp). +2. The key has been expired for longer than the retention period. + +Setting `--api-keys-retention=7d` deletes keys that expired more than 7 days +ago. Active keys are never deleted by the retention policy. + +Keeping expired keys for a short period allows Coder to return a more helpful +error message when users attempt to use an expired key. + +## Best Practices + +### Recommended Starting Configuration + +For most deployments, we recommend: + +```yaml +retention: + audit_logs: 365d + connection_logs: 90d + api_keys: 7d +``` + +### Compliance Considerations + +> [!WARNING] +> Audit Logs provide critical security and compliance information. Purging +> Audit Logs may impact your organization's ability to investigate security +> incidents or meet compliance requirements. Consult your security and +> compliance teams before configuring Audit Log retention. + +Common compliance frameworks have varying retention requirements: + +- **SOC 2**: Typically requires 1 year of audit logs. +- **HIPAA**: Requires 6 years for certain records. +- **PCI DSS**: Requires 1 year of audit logs, with 3 months immediately + available. +- **GDPR**: Requires data minimization but does not specify maximum retention. + +### External Log Aggregation + +If you use an external log aggregation system (Splunk, Datadog, etc.), you can +configure shorter retention periods in Coder since logs are preserved +externally. See +[Capturing/Exporting Audit Logs](../security/audit-logs.md#capturingexporting-audit-logs) +for details on exporting logs. + +### Database Maintenance + +After enabling retention policies, you may want to run a `VACUUM` operation on +your PostgreSQL database to reclaim disk space. See +[Maintenance Procedures](../security/audit-logs.md#maintenance-procedures-for-the-audit-logs-table) +for guidance. + +## Keeping Data Indefinitely + +To keep data indefinitely for any data type, set its retention value to `0`: + +```yaml +retention: + audit_logs: 0s # Keep audit logs forever + connection_logs: 0s # Keep connection logs forever + api_keys: 0s # Keep expired API keys forever +``` + +## Monitoring + +The purge process logs deletion counts at the `DEBUG` level. To monitor +retention activity, enable debug logging or search your logs for entries +containing the table name (e.g., `audit_logs`, `connection_logs`, `api_keys`). + +## Related Documentation + +- [Audit Logs](../security/audit-logs.md): Learn about Audit Logs and manual + purge procedures. +- [Connection Logs](../monitoring/connection-logs.md): Learn about Connection + Logs and monitoring. diff --git a/docs/manifest.json b/docs/manifest.json index 8abfe48cf1c41..0ac633576f9c9 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -363,6 +363,11 @@ "title": "Telemetry", "description": "Learn what usage telemetry Coder collects", "path": "./admin/setup/telemetry.md" + }, + { + "title": "Data Retention", + "description": "Configure data retention policies for database tables", + "path": "./admin/setup/data-retention.md" } ] },