Add detailed specification for the EZ-API Control Plane Dashboard
frontend-backend integration. This document defines the data mapping,
API endpoints, and UI logic for global navigation, real-time metrics,
traffic analysis, and alert summaries.
Add TestTrafficChart_MinuteGranularityValidation to verify input
parameters including granularity, time range limits, and top_n
constraints. Include skipped placeholders for PostgreSQL-specific
aggregation tests.
Add new endpoint GET /admin/logs/stats/traffic-chart to provide
aggregated traffic metrics grouped by time and model. Features include:
- Time granularity selection (hour/minute)
- Top-N model breakdown with "other" aggregation
- Metrics for request counts and token usage
Includes generated Swagger documentation.
- Create application-level context with cancel function
- Pass base context to scheduler via WithBaseContext option
- Move scheduler.Stop() to explicit shutdown sequence after context cancellation
- Upgrade foundation dependency to v0.7.0 for new scheduler options
Replace internal goroutine-based timing loops with scheduler integration
for SyncOutboxService and ModelRegistryService. Both services now expose
RunOnce() methods called by the central scheduler instead of managing
their own background loops.
- Add Interval() and RunOnce() methods to SyncOutboxService
- Add RefreshEvery() and RunOnce() methods to ModelRegistryService
- Remove started flag from SyncOutboxService struct
- Move scheduler.Start() after all services are initialized
- Ensure initial model registry refresh before scheduler starts
Replace custom goroutine-based scheduling in cron jobs with centralized
foundation scheduler. Each cron job now exposes a RunOnce method called
by the scheduler instead of managing its own ticker loop.
Changes:
- Remove interval/enabled config from cron job structs
- Convert Start() methods to RunOnce() for all cron jobs
- Add scheduler setup in main.go with configurable intervals
- Update foundation dependency to v0.6.0 for scheduler support
- Update tests to validate RunOnce nil-safety
Add auto-generated Swagger/OpenAPI documentation files for the
EZ-API Control Plane. These files are generated by swaggo/swag
and provide comprehensive API documentation including:
- All admin endpoints (alerts, API keys, bindings, masters, etc.)
- Master/tenant endpoints for token management and stats
- Internal endpoints for DP-CP communication
- Authentication schemes (AdminAuth, MasterAuth)
- Request/response schema definitions
- Add MasterID field with index to LogRecord model for efficient queries
- Fix threshold config loading to use fixed ID=1 with FirstOrCreate
- Allow traffic spike detection to work without Redis for log-based checks
- Add traffic_spike to API documentation for alert type filter
- Add comprehensive tests for RPM/RPD/TPM spike detection scenarios
Add unit tests for alert-related functionality:
- alert_handler_test.go: tests for threshold CRUD operations,
alert creation with traffic_spike type, filtering, and stats
- alert_detector_test.go: tests for threshold config loading,
traffic spike severity calculation, deduplication logic,
error rate severity, and nil-safety checks
Also fix format string issues:
- Use %d instead of %.2f for integer QPS in alert messages
- Wrap error description with format directive to avoid linter warning
Add OpenAPI documentation for the new alert threshold management
endpoints used in traffic spike detection:
- GET /admin/alerts/thresholds - retrieve current threshold config
- PUT /admin/alerts/thresholds - update threshold configuration
Include AlertThresholdView and UpdateAlertThresholdsRequest schema
definitions with properties for QPS, RPM, TPM, RPD, TPD limits.
Introduce traffic_spike alert type for monitoring system and per-master
traffic levels with configurable thresholds stored in database.
- Add AlertThresholdConfig model for persistent threshold configuration
- Implement GET/PUT /admin/alerts/thresholds endpoints for threshold management
- Add traffic spike detection in alert detector cron job:
- Global QPS monitoring across all masters
- Per-master RPM/TPM checks with minimum sample thresholds
- Per-master RPD/TPD checks for daily limits
- Use warning severity at threshold, critical at 2x threshold
- Include metric metadata (value, threshold, window) in alert details
- Update API documentation with new endpoints and alert type
Implement AlertDetector background task that runs every minute to detect
and create alerts for various anomalies:
- Rate limit detection: monitors masters hitting rate limits
- Error spike detection: flags keys with >= 10% error rate
- Quota exceeded: warns when key quota usage >= 90%
- Provider down: alerts when API keys have >= 50% failure rate
Includes fingerprint-based deduplication with 5-minute cooldown to
prevent duplicate alerts for the same issue.
- Add dashboard alerts, realtime, and apikey-stats endpoints
- Document time range parameters for logs and apikey-stats
- Update daily operations workflow with new monitoring endpoints
- Clarify period parameter behavior (default returns all data)
Document two new API endpoints:
- GET /admin/apikey-stats/summary with optional since/until params
for querying statistics within a specific time range
- POST /internal/alerts/report for Data Plane to report alerts
with fingerprint-based deduplication mechanism
Add ReportAlerts endpoint for Data Plane to report alerts to Control Plane
with fingerprint-based deduplication using a 5-minute cooldown period.
Changes:
- Add POST /internal/alerts/report endpoint with validation
- Add Fingerprint field to Alert model for deduplication
- Extend GetAPIKeyStatsSummary with optional time range filtering
using since/until query parameters to query from log records
Add comprehensive API documentation for new admin endpoints:
- Dashboard summary endpoint with period/time range parameters
- System-level realtime statistics (QPS, RPM, rate limits)
- Log stats aggregation by hour/minute with time constraints
- API key status filtering (active/suspended/disabled)
- Complete alert management system documentation:
- Alert types, severity levels, and status definitions
- CRUD endpoints for alert lifecycle management
- Alert statistics endpoint
Add OpenAPI documentation for the new alerts management system:
- CRUD endpoints for system alerts (/admin/alerts)
- Alert acknowledgment and resolution endpoints
- Alert statistics endpoint
- Alert filtering by status, severity, and type
Also document minute-level aggregation support for log stats
with 6-hour time range limitation.
Introduce a comprehensive alert management system for monitoring
system events and notifications.
Changes include:
- Add Alert model with type, severity, status, and metadata fields
- Implement AlertHandler with full CRUD operations (create, list,
get, acknowledge, resolve, dismiss)
- Add alert statistics endpoint for counts by status and severity
- Register Alert model in database auto-migration
- Add minute-level aggregation to log stats (limited to 6-hour range)
Add new admin API endpoints for dashboard metrics and system-wide
realtime statistics:
- Add /admin/dashboard/summary endpoint with aggregated metrics
including requests, tokens, latency, masters, keys, and provider
keys statistics with time period filtering
- Add /admin/realtime endpoint for system-level realtime stats
aggregated across all masters
- Add status filter parameter to ListAPIKeys endpoint
- Add hour grouping option to log stats aggregation
- Update OpenAPI documentation with new endpoints and schemas
Introduce internal endpoint for flushing accumulated APIKey statistics
from data plane to control plane database, updating both individual
API keys and their parent provider groups with request counts and
success/failure rates.
Add admin endpoint to retrieve aggregated API key statistics summary
across all provider groups, including total requests, success/failure
counts, and calculated rates.
Implement automatic token refresh mechanism for CPA providers (Codex,
GeminiCLI, Antigravity, ClaudeCode) with the following features:
- Periodic refresh of expiring tokens based on configurable interval
- Redis event queue processing for on-demand token refresh
- Retry logic with exponential backoff for transient failures
- Automatic key deactivation on non-retryable errors
- Provider-specific OAuth token refresh implementations
- Sync service integration to update providers after refresh
Add support for OAuth-based authentication with access/refresh tokens
and expiration tracking for API keys. Extend provider groups with
static headers configuration and headers profile options.
Changes include:
- Add AccessToken, RefreshToken, ExpiresAt, AccountID, ProjectID to APIKey model
- Add StaticHeaders and HeadersProfile to ProviderGroup model
- Add TokenRefresh configuration for background token management
- Support new provider types: ClaudeCode, Codex, GeminiCLI, Antigravity
- Update sync service to include new fields in provider snapshots
Clarify that the dp_claude_cross_upstream feature flag controls routing
of Claude protocol requests to both OpenAI-compatible and Google-family
upstream providers.
Replace health_handler with status_handler providing public /status and
/about endpoints. Add build-time version injection via ldflags in
Makefile, and support --version/-v CLI flag.
- Add /status endpoint returning runtime status, uptime, and version
- Add /about endpoint with system metadata (name, description, repo)
- Configure VERSION variable with git describe fallback
- Update swagger docs and api.md for new public endpoints
- Remove deprecated /api/status/test endpoint
Use UnixNano for version field while keeping Unix seconds for updated_at
timestamp. This ensures version changes are detected even when multiple
syncs occur within the same second.
Add new response types and parameters for logs API:
- Add GroupedStatsItem and GroupedStatsResponse definitions
- Add ListLogsResponse and LogView definitions for detailed log records
- Add group_by enum parameter (model/day/month) to stats endpoint
- Update endpoint descriptions to clarify response types and request_body inclusion
- Update response schema references to use correct types
Add group_by parameter to /admin/logs/stats endpoint supporting:
- group_by=model: aggregate stats per model with avg latency
- group_by=day: daily aggregation with token counts
- group_by=month: monthly aggregation with token counts
Also include request_body field in admin ListLogs response for
full visibility into logged requests.
- Introduce `SyncOutboxService` and model to retry failed CP-to-Redis sync operations
- Update `SyncService` to handle sync failures by enqueuing tasks to the outbox
- Centralize provider group and API key validation logic into `ProviderGroupManager`
- Refactor API handlers to utilize the new manager and robust sync methods
- Add configuration options for sync outbox (interval, batch size, retries)
- Update TestBatchBindings_Status to use newTestHandlerWithRedis helper
- Remove unused jsonID helper function from sync_bindings_spec_test.go
- Add time import to models.go
Regenerate OpenAPI documentation to reflect the architectural split
of Provider into ProviderGroup and APIKey models:
- Replace /admin/providers endpoints with /admin/provider-groups
- Add new /admin/api-keys CRUD and batch endpoints
- Update BindingDTO to use group_id instead of route_group
- Remove provider-specific DTOs (ProviderCustomCreateDTO, ProviderPresetCreateDTO, ProviderGoogleCreateDTO)
- Add APIKeyDTO and ProviderGroupDTO definitions
- Update model definitions for APIKey, Binding, and ProviderGroup
Restructure the provider management system by separating the monolithic
Provider model into two distinct entities:
- ProviderGroup: defines shared upstream configuration (type, base_url,
google settings, models, status)
- APIKey: represents individual credentials within a group (api_key,
weight, status, auto_ban, ban settings)
This change also updates:
- Binding model to reference GroupID instead of RouteGroup string
- All CRUD handlers for the new provider-group and api-key endpoints
- Sync service to rebuild provider snapshots from joined tables
- Model registry to aggregate capabilities across group/key pairs
- Access handler to validate namespace existence and subset constraints
- Migration importer to handle the new schema structure
- All related tests to use the new model relationships
BREAKING CHANGE: Provider API endpoints replaced with /provider-groups
and /api-keys endpoints; Binding.RouteGroup replaced with Binding.GroupID
Introduce a new `import` subcommand to the server binary that reads
exported JSON files and imports masters, providers, keys, bindings,
and namespaces into the database.
Key features:
- Support for dry-run mode to validate without writing
- Conflict policies: skip existing or overwrite
- Optional binding import via --include-bindings flag
- Auto-generation of master keys with secure hashing
- Namespace auto-creation for referenced namespaces
- Detailed import summary with warnings and created credentials
