feat: add server-side ETL pipeline, revert client to NocoDB reads

ETL Pipeline (server):
- POST /api/etl/sync?mode=full|incremental — fetches ERP, aggregates, writes NocoDB
- nocodbClient.ts: table discovery, paginated delete/insert
- etlSync.ts: orchestrates fetch → aggregate → upsert
- museumMapping.ts moved from client to server
- Auth via ETL_SECRET bearer token

Client:
- dataService.ts reverts to reading NocoDB DailySales table
- Paginated fetch via fetchNocoDBTable (handles >1000 rows)
- Suspicious data check: prefers cache if NocoDB returns <10 rows
- Deleted erpService.ts and client-side museumMapping.ts

First full sync: 391K transactions → 5,760 daily records in 108s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
fahed
2026-03-31 13:25:50 +03:00
parent 9c0ffa5721
commit 1f1e0756d0
12 changed files with 455 additions and 141 deletions

View File

@@ -23,3 +23,13 @@ export const erp = {
username: process.env.ERP_USERNAME || '',
password: process.env.ERP_PASSWORD || '',
};
export const nocodb = {
url: process.env.NOCODB_URL || '',
token: process.env.NOCODB_TOKEN || '',
baseId: process.env.NOCODB_BASE_ID || '',
};
export const etl = {
secret: process.env.ETL_SECRET || '',
};