Commit Graph

65 Commits

Author SHA1 Message Date
zenfun
6170931454 feat(cron): add OAuth token refresh background job
Implement automatic token refresh mechanism for CPA providers (Codex,
GeminiCLI, Antigravity, ClaudeCode) with the following features:

- Periodic refresh of expiring tokens based on configurable interval
- Redis event queue processing for on-demand token refresh
- Retry logic with exponential backoff for transient failures
- Automatic key deactivation on non-retryable errors
- Provider-specific OAuth token refresh implementations
- Sync service integration to update providers after refresh
2025-12-28 03:03:19 +08:00
zenfun
f0fe9f0dad feat(api): add OAuth token fields and new provider types support
Add support for OAuth-based authentication with access/refresh tokens
and expiration tracking for API keys. Extend provider groups with
static headers configuration and headers profile options.

Changes include:
- Add AccessToken, RefreshToken, ExpiresAt, AccountID, ProjectID to APIKey model
- Add StaticHeaders and HeadersProfile to ProviderGroup model
- Add TokenRefresh configuration for background token management
- Support new provider types: ClaudeCode, Codex, GeminiCLI, Antigravity
- Update sync service to include new fields in provider snapshots
2025-12-28 02:49:54 +08:00
zenfun
f51cd63c82 test(api): remove health handler test file 2025-12-27 13:51:25 +08:00
zenfun
637bfa8210 feat(api): add public status endpoints with version injection
Replace health_handler with status_handler providing public /status and
/about endpoints. Add build-time version injection via ldflags in
Makefile, and support --version/-v CLI flag.

- Add /status endpoint returning runtime status, uptime, and version
- Add /about endpoint with system metadata (name, description, repo)
- Configure VERSION variable with git describe fallback
- Update swagger docs and api.md for new public endpoints
- Remove deprecated /api/status/test endpoint
2025-12-27 13:24:13 +08:00
zenfun
3d39c591fd fix(sync): use nanosecond precision for bindings version to ensure uniqueness
Use UnixNano for version field while keeping Unix seconds for updated_at
timestamp. This ensures version changes are detected even when multiple
syncs occur within the same second.
2025-12-26 11:20:53 +08:00
zenfun
c83fe03892 docs(swagger): update API documentation for log endpoints
Add new response types and parameters for logs API:
- Add GroupedStatsItem and GroupedStatsResponse definitions
- Add ListLogsResponse and LogView definitions for detailed log records
- Add group_by enum parameter (model/day/month) to stats endpoint
- Update endpoint descriptions to clarify response types and request_body inclusion
- Update response schema references to use correct types
2025-12-26 11:12:47 +08:00
zenfun
3f1d006006 feat(api): add grouped statistics and request_body to log endpoints
Add group_by parameter to /admin/logs/stats endpoint supporting:
- group_by=model: aggregate stats per model with avg latency
- group_by=day: daily aggregation with token counts
- group_by=month: monthly aggregation with token counts

Also include request_body field in admin ListLogs response for
full visibility into logged requests.
2025-12-26 11:06:39 +08:00
8e6d86edd7 feat(api): add /auth/whoami endpoint for identity detection 2025-12-25 14:41:38 +08:00
41998a3584 feat(swagger): support dynamic host via EZ_SWAGGER_HOST env var
- Add ServerConfig.SwaggerHost field
- Set docs.SwaggerInfo.Host dynamically at startup
- Update docker-compose, k8s manifests, and .env.example
2025-12-25 10:59:57 +08:00
zenfun
6a16712b9d feat(core): implement sync outbox mechanism and refactor provider validation
- Introduce `SyncOutboxService` and model to retry failed CP-to-Redis sync operations
- Update `SyncService` to handle sync failures by enqueuing tasks to the outbox
- Centralize provider group and API key validation logic into `ProviderGroupManager`
- Refactor API handlers to utilize the new manager and robust sync methods
- Add configuration options for sync outbox (interval, batch size, retries)
2025-12-25 01:24:19 +08:00
zenfun
dff2fc7562 chore(test): clean up unused code and fix test helper usage
- Update TestBatchBindings_Status to use newTestHandlerWithRedis helper
- Remove unused jsonID helper function from sync_bindings_spec_test.go
- Add time import to models.go
2025-12-24 02:27:11 +08:00
zenfun
dea8363e41 refactor(api): split Provider into ProviderGroup and APIKey models
Restructure the provider management system by separating the monolithic
Provider model into two distinct entities:

- ProviderGroup: defines shared upstream configuration (type, base_url,
  google settings, models, status)
- APIKey: represents individual credentials within a group (api_key,
  weight, status, auto_ban, ban settings)

This change also updates:
- Binding model to reference GroupID instead of RouteGroup string
- All CRUD handlers for the new provider-group and api-key endpoints
- Sync service to rebuild provider snapshots from joined tables
- Model registry to aggregate capabilities across group/key pairs
- Access handler to validate namespace existence and subset constraints
- Migration importer to handle the new schema structure
- All related tests to use the new model relationships

BREAKING CHANGE: Provider API endpoints replaced with /provider-groups
and /api-keys endpoints; Binding.RouteGroup replaced with Binding.GroupID
2025-12-24 02:15:52 +08:00
zenfun
cd5616dc26 feat(migrate): add import CLI command and importer for migration data
Introduce a new `import` subcommand to the server binary that reads
exported JSON files and imports masters, providers, keys, bindings,
and namespaces into the database.

Key features:
- Support for dry-run mode to validate without writing
- Conflict policies: skip existing or overwrite
- Optional binding import via --include-bindings flag
- Auto-generation of master keys with secure hashing
- Namespace auto-creation for referenced namespaces
- Detailed import summary with warnings and created credentials
2025-12-23 20:13:45 +08:00
zenfun
de1d23cce5 docs: regenerate swagger docs with new API endpoints
Add comprehensive API documentation for new endpoints including:
- Bindings CRUD and batch operations
- Namespaces management
- Master tenant CRUD, access settings, and realtime stats
- Provider CRUD with preset/custom/google variants
- Model registry status, refresh, check, and rollback
- Log management with stats and webhook config
- Key access settings endpoints
- Internal stats flush endpoint
- Health check endpoint

Also adds swagger annotations for FlushStats internal handler.
2025-12-22 13:03:52 +08:00
zenfun
2c5ccd56ee feat(api): add realtime stats endpoints for masters
Introduce StatsService integration to admin and master handlers,
exposing realtime metrics (requests, tokens, QPS, rate limit status)
via new endpoints:
- GET /admin/masters/:id/realtime
- GET /v1/realtime

Also embed realtime stats in the existing GET /admin/masters/:id
response and change GlobalQPS default to 0 with validation to
reject negative values.
2025-12-22 12:02:27 +08:00
zenfun
fa7f92c6e3 feat(api): allow batch status updates 2025-12-21 23:32:07 +08:00
zenfun
c2ed2f3f9e feat(api): add namespaces, batch ops, and admin logs 2025-12-21 23:16:27 +08:00
zenfun
73147fc55a feat(api): add model delete, pagination, and cors config 2025-12-21 23:03:12 +08:00
zenfun
816ea93339 feat(arch): add log partitioning and provider delete sync 2025-12-21 20:45:16 +08:00
zenfun
f819f89ba2 test(ops): cover status test and model fetch 2025-12-21 20:23:04 +08:00
zenfun
f7baa0f08f feat(ops): add status test and model fetch 2025-12-21 20:22:56 +08:00
zenfun
6e6c669ea0 test(log): expand handler, config, and metrics coverage 2025-12-21 16:18:38 +08:00
zenfun
c2c65e774b feat(log): wire log db, metrics, and body toggle 2025-12-21 16:18:22 +08:00
zenfun
4c1e03f83d feat(api): add log webhook notification service
Implement webhook notifications for log error threshold alerts with
configurable thresholds, time windows, and cooldown periods.

- Add LogWebhookService with Redis-backed configuration storage
- Add admin endpoints for webhook config management (GET/PUT)
- Trigger webhook notifications when error count exceeds threshold
- Support status code threshold and error message detection
- Include sample log record data in webhook payload
2025-12-21 14:13:35 +08:00
zenfun
00192f937e feat(api): add log_request_body_enabled feature flag support
Add runtime feature flag to control whether request bodies are stored
in logs. The Handler now accepts a Redis client to check the
log_request_body_enabled feature flag before persisting log records.

- Add logRequestBodyFeatureKey constant for feature flag
- Inject Redis client into Handler for feature flag lookups
- Strip request body from log records when feature is disabled
- Update tests to pass Redis client to NewHandler
2025-12-21 13:26:16 +08:00
zenfun
1089f9490e test(api): add feature handler tests for regular and mixed keys
Add tests for UpdateFeatures endpoint covering:
- Regular feature flag keys with various value types
- Mixed keys combining feature flags with log retention settings
- Redis storage verification for both scenarios
2025-12-21 13:07:00 +08:00
zenfun
2f58dd76a7 feat(api): add log retention and max records feature overrides
Add special handling for log_retention_days and log_max_records features
that are stored as separate Redis keys instead of in the features hash.

- Store log overrides in dedicated keys (meta:log:retention_days,
  meta:log:max_records) for runtime access by log cleanup cron
- Include log override values in ListFeatures response
- Support clearing overrides by setting value to 0
- Add comprehensive tests for log override CRUD operations
2025-12-21 12:26:48 +08:00
zenfun
25795a79d6 feat(cron): add automatic log cleanup with retention policy
Implement LogCleaner cron job to automatically clean up old log records
based on configurable retention period and maximum record count.

- Add LogCleaner with retention_days and max_records configuration
- Add EZ_LOG_RETENTION_DAYS and EZ_LOG_MAX_RECORDS environment variables
- Default to 30 days retention and 1,000,000 max records
- Include unit tests for log cleaner functionality
2025-12-21 12:01:52 +08:00
zenfun
369c204c16 feat: add admin log deletion 2025-12-21 00:53:52 +08:00
zenfun
88289015fc feat: add internal stats flush API 2025-12-19 23:26:33 +08:00
zenfun
ac9f0cd0a7 feat(stats): add usage stats and quota reset 2025-12-19 21:50:28 +08:00
zenfun
524f8c5a4e feat(key): extend key metadata and validation 2025-12-19 21:24:24 +08:00
zenfun
5e98368428 model-registry: default supports_stream true 2025-12-18 16:48:43 +08:00
zenfun
7dd3fac24e model-registry: add upstream check endpoint 2025-12-18 16:43:12 +08:00
zenfun
a61eff27e7 feat(admin/master): provider+master CRUD, token mgmt, logs APIs 2025-12-18 16:21:46 +08:00
zenfun
b2d2df18c5 feat(model-registry): models.dev updater + admin endpoints 2025-12-17 23:59:34 +08:00
zenfun
96e1fe41a5 feat(models): add kind and models_meta snapshot 2025-12-17 23:15:12 +08:00
zenfun
2b0ed3d3d5 feat(api): align provider creation with presets/custom/google sdk 2025-12-17 22:19:17 +08:00
zenfun
174735f152 feat(api): add provider creation endpoints and weight support
- Add `POST /admin/providers/preset` for streamlined creation of official providers (OpenAI, Anthropic, Gemini)
- Add `POST /admin/providers/custom` for generic OpenAI-compatible providers
- Add `weight` field to provider model and DTOs to enable weighted routing
- Update sync service to propagate provider weights
- Add unit tests for new creation handlers
2025-12-17 21:40:54 +08:00
zenfun
5a4e63b75d test(service): add specs for binding synchronization
Add unit tests for the SyncBindings service covering exact matching,
regex selectors, and normalized matching logic using in-memory
dependencies (SQLite and MiniRedis) to ensure correct upstream
resolution.
2025-12-17 01:18:02 +08:00
zenfun
6d8a12df34 feat(api): add access control and binding endpoints
Add endpoints for master and key access management to configure default
and allowed namespaces, including propagation options.
Implement GET and DELETE operations for individual bindings.
Update sync service to persist bindings snapshots even when no upstreams
are available.
2025-12-17 01:08:01 +08:00
zenfun
16fceec8e7 style(core): align struct tags and fields
Standardize alignment of struct tags in models and field assignments in
services for better readability. Additionally, include the Binding model
in the test database auto-migration.
2025-12-17 00:51:02 +08:00
zenfun
ed6446e586 feat(api): add admin endpoints for binding management
Implement handlers for creating, listing, and updating model bindings.
Register new routes in the admin server group and add DTO definitions.
Update provider handlers to trigger binding synchronization on changes
to ensure upstream mappings remain current.
2025-12-17 00:37:02 +08:00
zenfun
2e3b471533 feat(core): implement namespace support and routing bindings
Introduce namespace-aware routing capabilities through a new Binding
model and updates to Master/Key entities.

- Add Binding model for configuring model routes per namespace
- Add DefaultNamespace and Namespaces fields to Master and Key models
- Update auto-migration to include Binding table
- Implement Redis synchronization for binding configurations
- Propagate namespace settings during master creation and key issuance
2025-12-17 00:33:36 +08:00
zenfun
aa69ce3659 feat(api): add admin endpoint to issue keys for masters
Add `POST /admin/masters/{id}/keys` allowing admins to issue child keys
on behalf of a master. Introduce an `issued_by` field in the Key model
to audit whether a key was issued by the master or an admin.

Refactor master service to use typed errors for consistent HTTP status
mapping and ensure validation logic (active status, group check) is
shared.
2025-12-15 15:59:33 +08:00
zenfun
11f6e81798 feat(api): add feature flag management endpoints
Add FeatureHandler to manage lightweight runtime configuration toggles
stored in the Redis `meta:features` hash. This enables dynamic control
over system behavior (e.g., storage backends) via the admin API.

- Add `GET /admin/features` to list flags
- Add `PUT /admin/features` to update flags
- Update README with feature flag documentation
2025-12-15 15:01:01 +08:00
zenfun
d1d1b1c42a refactor(deps): use foundation shared utilities 2025-12-14 23:52:46 +08:00
zenfun
71c183a480 test(service): add golden file validation and master key tests
- Update TESTING.md to reflect current testing status and future plans
- Add golden file comparison for provider snapshot validation in sync_test.go
- Introduce master_test.go for testing Master/Key functionality
- Add testdata directory for contract testing snapshots
2025-12-14 23:37:16 +08:00
zenfun
d0011f3eb2 test: add comprehensive unit tests for provider, middleware, and sync service
- Add TESTING.md documentation explaining unit test conventions and integration testing setup
- Add miniredis and sqlite dependencies to go.mod for in-memory testing
- Add provider_handler_test.go ensuring Vertex providers default google_location to "global"
- Add request_id_test.go verifying request ID generation and header preservation
- Add sync_test.go validating Redis snapshot writes and routing key generation
- Update README.md with testing section referencing new documentation
2025-12-14 00:35:35 +08:00
zenfun
531ac6e5a8 feat(telemetry): implement request tracing identifier
Introduce a middleware layer to attach a unique identifier to each HTTP request for observability purposes. The identifier is propagated via the X-Request-ID header, allowing for correlation of logs and events across distributed system components.
2025-12-13 23:49:58 +08:00