feat: add server-side ETL pipeline, revert client to NocoDB reads

ETL Pipeline (server):
- POST /api/etl/sync?mode=full|incremental — fetches ERP, aggregates, writes NocoDB
- nocodbClient.ts: table discovery, paginated delete/insert
- etlSync.ts: orchestrates fetch → aggregate → upsert
- museumMapping.ts moved from client to server
- Auth via ETL_SECRET bearer token

Client:
- dataService.ts reverts to reading NocoDB DailySales table
- Paginated fetch via fetchNocoDBTable (handles >1000 rows)
- Suspicious data check: prefers cache if NocoDB returns <10 rows
- Deleted erpService.ts and client-side museumMapping.ts

First full sync: 391K transactions → 5,760 daily records in 108s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
fahed
2026-03-31 13:25:50 +03:00
parent 9c0ffa5721
commit 1f1e0756d0
12 changed files with 455 additions and 141 deletions

27
server/src/types.ts Normal file
View File

@@ -0,0 +1,27 @@
export interface ERPProduct {
ProductDescription: string;
SiteDescription: string | null;
UnitQuantity: number;
PeopleCount: number;
TaxAmount: number;
TotalPrice: number;
}
export interface ERPSaleRecord {
SaleId: number;
TransactionDate: string;
CustIdentification: string;
OperatingAreaName: string;
Payments: Array<{ PaymentMethodDescription: string }>;
Products: ERPProduct[];
}
export interface AggregatedRecord {
Date: string;
MuseumName: string;
Channel: string;
Visits: number;
Tickets: number;
GrossRevenue: number;
NetRevenue: number;
}