|
|
|
|
@@ -0,0 +1,669 @@
|
|
|
|
|
# DiunaBI Project Context
|
|
|
|
|
|
|
|
|
|
> This file is auto-generated for Claude Code to quickly understand the project structure.
|
|
|
|
|
> Last updated: 2025-12-05
|
|
|
|
|
|
|
|
|
|
## RECENT CHANGES (This Session)
|
|
|
|
|
|
|
|
|
|
**UI Reorganization (Dec 5, 2025):**
|
|
|
|
|
- ✅ Moved pages to feature-based folders: `Pages/Layers/`, `Pages/Jobs/`, `Pages/DataInbox/`
|
|
|
|
|
- ✅ Organized components: `Components/Layout/` (MainLayout, EmptyLayout, Routes), `Components/Auth/` (AuthGuard, LoginCard)
|
|
|
|
|
- ✅ Removed obsolete wrapper files (LayerListPage, JobListPage, DataInboxListPage, etc.)
|
|
|
|
|
- ✅ Removed duplicate component files (LayerListComponent, JobListComponent, DataInboxListComponent)
|
|
|
|
|
- ✅ Standardized code-behind: `.razor.cs` for complex logic, inline `@code` for simple pages
|
|
|
|
|
- ✅ Updated `_Imports.razor` with new namespaces: `DiunaBI.UI.Shared.Components.Layout`, `DiunaBI.UI.Shared.Components.Auth`
|
|
|
|
|
- ✅ All routes unchanged - backward compatible
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## PROJECT TYPE & TECH STACK
|
|
|
|
|
|
|
|
|
|
**Application Type:** Full-stack Business Intelligence (BI) platform with multi-tier architecture, real-time capabilities, and plugin system
|
|
|
|
|
|
|
|
|
|
**Core Stack:**
|
|
|
|
|
- Backend: ASP.NET Core 10.0 Web API
|
|
|
|
|
- Frontend: Blazor Server + MAUI Mobile
|
|
|
|
|
- Database: SQL Server + EF Core 10.0
|
|
|
|
|
- UI: MudBlazor 8.0
|
|
|
|
|
- Real-time: SignalR (EntityChangeHub)
|
|
|
|
|
- Google: Sheets API, Drive API, OAuth
|
|
|
|
|
- Logging: Serilog (Console, File, Seq)
|
|
|
|
|
- Auth: JWT Bearer + Google OAuth
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## SOLUTION STRUCTURE (10 Projects)
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
DiunaBI.API (Web API)
|
|
|
|
|
├── Controllers: Auth, Layers, Jobs, DataInbox
|
|
|
|
|
├── Hubs: EntityChangeHub (SignalR real-time updates)
|
|
|
|
|
└── Services: GoogleAuth, JwtToken
|
|
|
|
|
|
|
|
|
|
DiunaBI.Domain (Entities)
|
|
|
|
|
└── User, Layer, Record, RecordHistory, QueueJob, DataInbox, ProcessSource
|
|
|
|
|
|
|
|
|
|
DiunaBI.Application (DTOs)
|
|
|
|
|
└── LayerDto, RecordDto, UserDto, RecordHistoryDto, PagedResult, JobDto
|
|
|
|
|
|
|
|
|
|
DiunaBI.Infrastructure (Data + Services)
|
|
|
|
|
├── Data: AppDbContext, Migrations (47 total)
|
|
|
|
|
├── Interceptors: EntityChangeInterceptor (auto-broadcasts DB changes)
|
|
|
|
|
├── Services: PluginManager, JobScheduler, JobWorker, GoogleSheets/Drive
|
|
|
|
|
├── Plugins: BaseDataImporter, BaseDataProcessor, BaseDataExporter
|
|
|
|
|
└── Interfaces: IPlugin, IDataProcessor, IDataImporter, IDataExporter
|
|
|
|
|
|
|
|
|
|
DiunaBI.UI.Web (Blazor Server)
|
|
|
|
|
└── Server-side Blazor web application
|
|
|
|
|
|
|
|
|
|
DiunaBI.UI.Mobile (MAUI)
|
|
|
|
|
└── iOS, Android, Windows, macOS support
|
|
|
|
|
|
|
|
|
|
DiunaBI.UI.Shared (Blazor Component Library - Reorganized)
|
|
|
|
|
├── Pages/
|
|
|
|
|
│ ├── Layers/ (Index.razor, Details.razor)
|
|
|
|
|
│ ├── Jobs/ (Index.razor, Details.razor)
|
|
|
|
|
│ ├── DataInbox/ (Index.razor, Details.razor)
|
|
|
|
|
│ ├── Dashboard.razor, Login.razor, Index.razor
|
|
|
|
|
├── Components/
|
|
|
|
|
│ ├── Layout/ (MainLayout, EmptyLayout, Routes)
|
|
|
|
|
│ └── Auth/ (AuthGuard, LoginCard)
|
|
|
|
|
└── Services/
|
|
|
|
|
├── LayerService, JobService, DataInboxService
|
|
|
|
|
├── EntityChangeHubService (SignalR client)
|
|
|
|
|
├── FilterStateServices (remember filters)
|
|
|
|
|
└── AuthService, TokenProvider
|
|
|
|
|
|
|
|
|
|
DiunaBI.Plugins.Morska (Feature Plugin)
|
|
|
|
|
├── Importers: Standard, D1, D3, FK2 (4 total)
|
|
|
|
|
├── Processors: D6, T1, T3, T4, T5 variants (12 total)
|
|
|
|
|
└── Exporters: Google Sheets export (1)
|
|
|
|
|
|
|
|
|
|
DiunaBI.Plugins.PedrolloPL (Feature Plugin - NEW)
|
|
|
|
|
└── Importers: P2 (1 total)
|
|
|
|
|
|
|
|
|
|
DiunaBI.Tests (Testing)
|
|
|
|
|
└── Unit and integration tests
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## CORE FUNCTIONALITY
|
|
|
|
|
|
|
|
|
|
**Purpose:** BI platform for data import, processing, transformation via modular plugin architecture. Multi-layer workflows with audit trails, real-time notifications, scheduled job processing.
|
|
|
|
|
|
|
|
|
|
**Main Features:**
|
|
|
|
|
1. **Layer Management** - 4 types (Import/Processed/Admin/Dictionary), parent-child relationships, soft deletes
|
|
|
|
|
2. **Data Records** - 32 numeric columns (Value1-32) + description, hierarchical, full audit trail
|
|
|
|
|
3. **Plugin Architecture** - Dynamic assembly loading, base classes in Infrastructure, 3 types (Importers/Processors/Exporters)
|
|
|
|
|
4. **Job Queue System** - Background worker with retry logic (30s → 2m → 5m), priority-based, auto-scheduling
|
|
|
|
|
5. **External Data** - DataInbox API, Google Sheets read/write, Google Drive integration
|
|
|
|
|
6. **Real-time Updates** - SignalR broadcasts entity changes (create/update/delete) to all connected clients
|
|
|
|
|
7. **Audit Trail** - RecordHistory tracks all record changes with field-level diffs and JSON summaries
|
|
|
|
|
8. **Filter Persistence** - UI filter states saved across sessions (LayerFilterStateService, DataInboxFilterStateService)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## KEY ENTITIES
|
|
|
|
|
|
|
|
|
|
**Layer**
|
|
|
|
|
- Id, Number, Name, Type (Import/Processed/Administration/Dictionary)
|
|
|
|
|
- CreatedAt/ModifiedAt, CreatedBy/ModifiedBy (with user relations)
|
|
|
|
|
- IsDeleted (soft delete), IsCancelled (processing control), ParentId
|
|
|
|
|
- Relations: Records (1-to-many), ProcessSources (1-to-many)
|
|
|
|
|
|
|
|
|
|
**Record**
|
|
|
|
|
- Id, Code (unique identifier), LayerId
|
|
|
|
|
- Value1-Value32 (double?), Desc1 (string, max 10000 chars)
|
|
|
|
|
- CreatedAt/ModifiedAt, CreatedBy/ModifiedBy, IsDeleted
|
|
|
|
|
- Audit: Full history tracked in RecordHistory table
|
|
|
|
|
|
|
|
|
|
**RecordHistory** (NEW - Migration 47)
|
|
|
|
|
- RecordId, LayerId, ChangedAt, ChangedById
|
|
|
|
|
- ChangeType (Created/Updated/Deleted)
|
|
|
|
|
- Code, Desc1 (snapshot at time of change)
|
|
|
|
|
- ChangedFields (comma-separated field names)
|
|
|
|
|
- ChangesSummary (JSON with old/new values)
|
|
|
|
|
- Indexes: (RecordId, ChangedAt), (LayerId, ChangedAt) for performance
|
|
|
|
|
|
|
|
|
|
**QueueJob**
|
|
|
|
|
- LayerId, LayerName, PluginName
|
|
|
|
|
- JobType (Import/Process)
|
|
|
|
|
- Priority (0 = highest), Status (Pending/Running/Completed/Failed/Retrying)
|
|
|
|
|
- RetryCount, MaxRetries (default 5)
|
|
|
|
|
- CreatedAt, LastAttemptAt, CompletedAt
|
|
|
|
|
- LastError (detailed error message)
|
|
|
|
|
|
|
|
|
|
**DataInbox**
|
|
|
|
|
- Id, Name, Source (identifiers)
|
|
|
|
|
- Data (base64-encoded JSON array)
|
|
|
|
|
- CreatedAt
|
|
|
|
|
- Used by importers to stage incoming data
|
|
|
|
|
|
|
|
|
|
**User**
|
|
|
|
|
- Id (Guid), Email, UserName
|
|
|
|
|
- CreatedAt, LastLoginAt
|
|
|
|
|
- Google OAuth identity
|
|
|
|
|
|
|
|
|
|
**ProcessSource**
|
|
|
|
|
- Id, SourceLayerId, TargetLayerId
|
|
|
|
|
- Defines layer processing relationships
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## API ENDPOINTS
|
|
|
|
|
|
|
|
|
|
**Base:** `/` (ApiController routes)
|
|
|
|
|
|
|
|
|
|
### AuthController (/auth)
|
|
|
|
|
- `POST /auth/apiToken` - Exchange Google ID token for JWT (AllowAnonymous)
|
|
|
|
|
- `POST /auth/refresh` - Refresh expired JWT token
|
|
|
|
|
|
|
|
|
|
### LayersController (/layers)
|
|
|
|
|
- `GET /layers?page=1&pageSize=10&search=&type=` - List layers (paged, filterable)
|
|
|
|
|
- `GET /layers/{id}` - Get layer details with records
|
|
|
|
|
- `POST /layers` - Create new layer
|
|
|
|
|
- `PUT /layers/{id}` - Update layer
|
|
|
|
|
- `DELETE /layers/{id}` - Soft delete layer
|
|
|
|
|
- `POST /layers/{id}/records` - Add/update records
|
|
|
|
|
- `PUT /layers/{layerId}/records/{recordId}` - Update specific record
|
|
|
|
|
- `DELETE /layers/{layerId}/records/{recordId}` - Delete record
|
|
|
|
|
- `GET /layers/{layerId}/records/{recordId}/history` - Get record history
|
|
|
|
|
- `GET /layers/{layerId}/deleted-records` - Get deleted records with history
|
|
|
|
|
|
|
|
|
|
### JobsController (/jobs) - NEW
|
|
|
|
|
- `GET /jobs?page=1&pageSize=50&status=&jobType=` - List jobs (paged, filterable)
|
|
|
|
|
- `GET /jobs/{id}` - Get job details
|
|
|
|
|
- `GET /jobs/stats` - Get job statistics (counts by status)
|
|
|
|
|
- `POST /jobs/schedule/{apiKey}` - Schedule all jobs from layer configs
|
|
|
|
|
- `POST /jobs/schedule/imports/{apiKey}` - Schedule import jobs only
|
|
|
|
|
- `POST /jobs/schedule/processes/{apiKey}` - Schedule process jobs only
|
|
|
|
|
- `POST /jobs/create-for-layer/{layerId}` - Create job for specific layer (manual trigger)
|
|
|
|
|
- `POST /jobs/{id}/retry` - Retry failed job (resets to Pending)
|
|
|
|
|
- `DELETE /jobs/{id}` - Cancel pending/retrying job
|
|
|
|
|
|
|
|
|
|
### DataInboxController (/datainbox)
|
|
|
|
|
- `GET /datainbox?page=1&pageSize=10&search=` - List inbox items (paged, filterable)
|
|
|
|
|
- `GET /datainbox/{id}` - Get inbox item with decoded data
|
|
|
|
|
- `POST /datainbox` - Create inbox item
|
|
|
|
|
- `PUT /datainbox/Add/{apiKey}` - Add data (API key + Basic Auth)
|
|
|
|
|
- `DELETE /datainbox/{id}` - Delete inbox item
|
|
|
|
|
|
|
|
|
|
### SignalR Hub
|
|
|
|
|
- `/hubs/entitychanges` - SignalR hub for real-time entity change notifications
|
|
|
|
|
- Event: `EntityChanged(module, id, operation)` - broadcasts to all clients
|
|
|
|
|
- Modules: QueueJobs, Layers, Records, RecordHistory
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## AUTHENTICATION & SECURITY
|
|
|
|
|
|
|
|
|
|
**Flow:**
|
|
|
|
|
1. Client exchanges Google ID token → `/auth/apiToken`
|
|
|
|
|
2. GoogleAuthService validates token with Google, maps to internal User
|
|
|
|
|
3. Returns JWT (7-day expiration, HS256 signing)
|
|
|
|
|
4. JWT required on all protected endpoints (except /auth/apiToken, /health)
|
|
|
|
|
5. UserId extraction middleware sets X-UserId header for audit trails
|
|
|
|
|
|
|
|
|
|
**Security:**
|
|
|
|
|
- Google OAuth 2.0 for identity verification
|
|
|
|
|
- JWT Bearer tokens for API access
|
|
|
|
|
- API key + Basic Auth for DataInbox external endpoints
|
|
|
|
|
- CORS configured for:
|
|
|
|
|
- http://localhost:4200
|
|
|
|
|
- https://diuna.bim-it.pl
|
|
|
|
|
- https://morska.diunabi.com
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## KEY SERVICES
|
|
|
|
|
|
|
|
|
|
### Infrastructure Services
|
|
|
|
|
|
|
|
|
|
**PluginManager**
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Services/PluginManager.cs`
|
|
|
|
|
- Loads plugin assemblies from `bin/Plugins/` directory at startup
|
|
|
|
|
- Registers IDataProcessor, IDataImporter, IDataExporter implementations
|
|
|
|
|
- Provides plugin discovery and execution
|
|
|
|
|
|
|
|
|
|
**JobSchedulerService**
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Services/JobSchedulerService.cs`
|
|
|
|
|
- Creates QueueJob entries from Administration layer configs
|
|
|
|
|
- Reads layer.Records with Code="Plugin", Code="Priority", Code="MaxRetries"
|
|
|
|
|
- Methods: ScheduleImportJobsAsync, ScheduleProcessJobsAsync, ScheduleAllJobsAsync
|
|
|
|
|
|
|
|
|
|
**JobWorkerService** (BackgroundService)
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Services/JobWorkerService.cs`
|
|
|
|
|
- Polls QueueJobs table every 10 seconds
|
|
|
|
|
- Executes jobs via PluginManager (Import/Process)
|
|
|
|
|
- Retry logic with exponential backoff: 30s → 2m → 5m delays
|
|
|
|
|
- Rate limiting: 5-second delay after imports (Google Sheets API quota)
|
|
|
|
|
- Updates job status in real-time (triggers SignalR broadcasts)
|
|
|
|
|
|
|
|
|
|
**EntityChangeInterceptor**
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Interceptors/EntityChangeInterceptor.cs`
|
|
|
|
|
- EF Core SaveChangesInterceptor
|
|
|
|
|
- Captures entity changes: Added, Modified, Deleted
|
|
|
|
|
- Broadcasts changes via SignalR EntityChangeHub after successful save
|
|
|
|
|
- Uses reflection to avoid circular dependencies with IHubContext
|
|
|
|
|
|
|
|
|
|
**GoogleSheetsHelper**
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Helpers/GoogleSheetsHelper.cs`
|
|
|
|
|
- Google Sheets API v4 integration
|
|
|
|
|
- Methods: ReadRange, WriteRange, CreateSpreadsheet, UpdateSpreadsheet
|
|
|
|
|
|
|
|
|
|
**GoogleDriveHelper**
|
|
|
|
|
- Location: `DiunaBI.Infrastructure/Helpers/GoogleDriveHelper.cs`
|
|
|
|
|
- Google Drive API v3 integration
|
|
|
|
|
- Methods: UploadFile, ListFiles, MoveFile
|
|
|
|
|
|
|
|
|
|
**GoogleAuthService / JwtTokenService**
|
|
|
|
|
- Authentication and token management
|
|
|
|
|
- JWT generation and validation
|
|
|
|
|
|
|
|
|
|
### UI Services
|
|
|
|
|
|
|
|
|
|
**EntityChangeHubService**
|
|
|
|
|
- Location: `DiunaBI.UI.Shared/Services/EntityChangeHubService.cs`
|
|
|
|
|
- Singleton service for SignalR client connection
|
|
|
|
|
- Auto-reconnect enabled
|
|
|
|
|
- Event: `EntityChanged` - UI components subscribe for real-time updates
|
|
|
|
|
- Initialized in MainLayout.OnInitializedAsync
|
|
|
|
|
|
|
|
|
|
**LayerService / JobService / DataInboxService**
|
|
|
|
|
- HTTP clients for API communication
|
|
|
|
|
- DTOs serialization/deserialization
|
|
|
|
|
- Paged result handling
|
|
|
|
|
|
|
|
|
|
**LayerFilterStateService / DataInboxFilterStateService**
|
|
|
|
|
- Persist filter state across navigation
|
|
|
|
|
- Singleton services remember search, type, page selections
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## DATABASE SCHEMA
|
|
|
|
|
|
|
|
|
|
**Total Migrations:** 47
|
|
|
|
|
|
|
|
|
|
**Latest Migrations:**
|
|
|
|
|
|
|
|
|
|
**Migration 47: RecordHistory (Dec 1, 2025)**
|
|
|
|
|
- **NEW Table: RecordHistory**
|
|
|
|
|
- Tracks all record changes (Created, Updated, Deleted)
|
|
|
|
|
- Fields: Id, RecordId, LayerId, ChangedAt, ChangedById, ChangeType, Code, Desc1, ChangedFields, ChangesSummary
|
|
|
|
|
- Indexes: IX_RecordHistory_RecordId_ChangedAt, IX_RecordHistory_LayerId_ChangedAt
|
|
|
|
|
- Foreign key: RecordHistory.ChangedById → Users.Id
|
|
|
|
|
|
|
|
|
|
**Migration 46: FixLayerDefaultValues (Nov 20, 2025)**
|
|
|
|
|
- Set default value: Layers.IsDeleted = false
|
|
|
|
|
|
|
|
|
|
**Migration 45: UpdateModel (Nov 19, 2025)**
|
|
|
|
|
- Added GETUTCDATE() defaults for all timestamp fields
|
|
|
|
|
- Changed foreign key constraints from CASCADE to RESTRICT:
|
|
|
|
|
- Layers → Users (CreatedById, ModifiedById)
|
|
|
|
|
- Records → Users (CreatedById, ModifiedById)
|
|
|
|
|
- Added FK_ProcessSources_Layers_LayerId
|
|
|
|
|
|
|
|
|
|
**Core Tables:**
|
|
|
|
|
- Users (authentication, audit)
|
|
|
|
|
- Layers (4 types, soft deletes, parent-child)
|
|
|
|
|
- Records (32 Value fields + Desc1, audit, soft deletes)
|
|
|
|
|
- RecordHistory (change tracking, field diffs, JSON summaries)
|
|
|
|
|
- QueueJobs (job queue, retry logic, status tracking)
|
|
|
|
|
- DataInbox (incoming data staging, base64 encoded)
|
|
|
|
|
- ProcessSources (layer relationships)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## PLUGIN SYSTEM
|
|
|
|
|
|
|
|
|
|
### Base Classes (Infrastructure/Plugins/)
|
|
|
|
|
|
|
|
|
|
**BaseDataImporter** (`DiunaBI.Infrastructure/Plugins/BaseDataImporter.cs`)
|
|
|
|
|
- Abstract base for all importers
|
|
|
|
|
- Methods: ImportAsync(layerId, jobId), ValidateConfiguration()
|
|
|
|
|
- Access: AppDbContext, PluginManager, GoogleSheetsHelper, GoogleDriveHelper
|
|
|
|
|
|
|
|
|
|
**BaseDataProcessor** (`DiunaBI.Infrastructure/Plugins/BaseDataProcessor.cs`)
|
|
|
|
|
- Abstract base for all processors
|
|
|
|
|
- Methods: ProcessAsync(layerId, jobId), ValidateConfiguration()
|
|
|
|
|
- Access: AppDbContext, PluginManager
|
|
|
|
|
|
|
|
|
|
**BaseDataExporter** (`DiunaBI.Infrastructure/Plugins/BaseDataExporter.cs`)
|
|
|
|
|
- Abstract base for all exporters
|
|
|
|
|
- Methods: ExportAsync(layerId, jobId), ValidateConfiguration()
|
|
|
|
|
- Access: AppDbContext, GoogleSheetsHelper, GoogleDriveHelper
|
|
|
|
|
|
|
|
|
|
### Morska Plugin (DiunaBI.Plugins.Morska)
|
|
|
|
|
|
|
|
|
|
**Importers (4):**
|
|
|
|
|
- MorskaStandardImporter - Generic CSV/Excel import
|
|
|
|
|
- MorskaD1Importer - D1 data format
|
|
|
|
|
- MorskaD3Importer - D3 data format
|
|
|
|
|
- MorskaFK2Importer - FK2 data format
|
|
|
|
|
|
|
|
|
|
**Processors (12):**
|
|
|
|
|
- MorskaD6Processor
|
|
|
|
|
- MorskaT1R1Processor
|
|
|
|
|
- MorskaT1R3Processor
|
|
|
|
|
- MorskaT3SingleSourceProcessor
|
|
|
|
|
- MorskaT3SourceYearSummaryProcessor
|
|
|
|
|
- MorskaT3MultiSourceSummaryProcessor
|
|
|
|
|
- MorskaT3MultiSourceYearSummaryProcessor
|
|
|
|
|
- MorskaT4R2Processor
|
|
|
|
|
- MorskaT4SingleSourceProcessor
|
|
|
|
|
- MorskaT5LastValuesProcessor
|
|
|
|
|
- MorskaT3MultiSourceCopySelectedCodesProcessor-TO_REMOVE (deprecated)
|
|
|
|
|
- MorskaT3MultiSourceCopySelectedCodesYearSummaryProcessor-TO_REMOVE (deprecated)
|
|
|
|
|
|
|
|
|
|
**Exporters (1):**
|
|
|
|
|
- googleSheet.export.cs - Google Sheets export
|
|
|
|
|
|
|
|
|
|
**Total:** ~6,566 lines of code
|
|
|
|
|
|
|
|
|
|
### PedrolloPL Plugin (DiunaBI.Plugins.PedrolloPL) - NEW
|
|
|
|
|
|
|
|
|
|
**Importers (1):**
|
|
|
|
|
- **PedrolloPLImportP2** (`DiunaBI.Plugins.PedrolloPL/Importers/PedrolloPLImportP2.cs`)
|
|
|
|
|
- Imports P2 data from DataInbox
|
|
|
|
|
- Uses L1-D-P2-CODES dictionary layer for region code mapping
|
|
|
|
|
- Creates 12 monthly records per region (Value1-Value12)
|
|
|
|
|
- Generates Import layers: L{Number}-I-P2-{Year}-{Timestamp}
|
|
|
|
|
- Handles base64 JSON data decoding
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## UI STRUCTURE (DiunaBI.UI.Shared)
|
|
|
|
|
|
|
|
|
|
### Reorganized Structure (Dec 5, 2025)
|
|
|
|
|
|
|
|
|
|
**Pages/** (Routable pages with @page directive)
|
|
|
|
|
```
|
|
|
|
|
Pages/
|
|
|
|
|
├── Layers/
|
|
|
|
|
│ ├── Index.razor + Index.razor.cs - /layers (list with filters, pagination)
|
|
|
|
|
│ └── Details.razor + Details.razor.cs - /layers/{id} (detail, edit, history)
|
|
|
|
|
├── Jobs/
|
|
|
|
|
│ ├── Index.razor + Index.razor.cs - /jobs (list with filters, real-time updates)
|
|
|
|
|
│ └── Details.razor - /jobs/{id} (detail, retry, cancel, real-time)
|
|
|
|
|
├── DataInbox/
|
|
|
|
|
│ ├── Index.razor + Index.razor.cs - /datainbox (list with filters)
|
|
|
|
|
│ └── Details.razor + Details.razor.cs - /datainbox/{id} (detail, base64 decode)
|
|
|
|
|
├── Dashboard.razor - /dashboard (user info)
|
|
|
|
|
├── Login.razor - /login (Google OAuth)
|
|
|
|
|
└── Index.razor - / (redirects to /dashboard)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Components/** (Reusable components, no routes)
|
|
|
|
|
```
|
|
|
|
|
Components/
|
|
|
|
|
├── Layout/
|
|
|
|
|
│ ├── MainLayout.razor - Main app layout with drawer, nav menu
|
|
|
|
|
│ ├── EmptyLayout.razor - Minimal layout for login page
|
|
|
|
|
│ └── Routes.razor - Router configuration
|
|
|
|
|
└── Auth/
|
|
|
|
|
├── AuthGuard.razor - Authentication guard wrapper
|
|
|
|
|
└── LoginCard.razor - Google login button component
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Navigation Menu:**
|
|
|
|
|
- Dashboard (/dashboard) - User profile
|
|
|
|
|
- Layers (/layers) - Layer management
|
|
|
|
|
- Data Inbox (/datainbox) - Incoming data review
|
|
|
|
|
- Jobs (/jobs) - Job queue monitoring (with real-time status updates)
|
|
|
|
|
|
|
|
|
|
**Code-Behind Pattern:**
|
|
|
|
|
- Complex pages (50+ lines logic): Separate `.razor.cs` files
|
|
|
|
|
- Simple pages: Inline `@code` blocks
|
|
|
|
|
- Namespaces: `DiunaBI.UI.Shared.Pages.{Feature}`
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## REAL-TIME FEATURES (SignalR)
|
|
|
|
|
|
|
|
|
|
### Architecture
|
|
|
|
|
|
|
|
|
|
**Hub:** `DiunaBI.API/Hubs/EntityChangeHub.cs`
|
|
|
|
|
- Endpoint: `/hubs/entitychanges`
|
|
|
|
|
- Method: `SendEntityChange(string module, string id, string operation)`
|
|
|
|
|
- Broadcasts: `EntityChanged` event to all connected clients
|
|
|
|
|
|
|
|
|
|
**Interceptor:** `DiunaBI.Infrastructure/Interceptors/EntityChangeInterceptor.cs`
|
|
|
|
|
- EF Core SaveChangesInterceptor
|
|
|
|
|
- Detects: Added, Modified, Deleted entities
|
|
|
|
|
- Broadcasts: After successful SaveChanges
|
|
|
|
|
- Modules: QueueJobs, Layers, Records, RecordHistory
|
|
|
|
|
|
|
|
|
|
**UI Service:** `DiunaBI.UI.Shared/Services/EntityChangeHubService.cs`
|
|
|
|
|
- Singleton initialized in MainLayout
|
|
|
|
|
- Auto-reconnect enabled
|
|
|
|
|
- Components subscribe: `HubService.EntityChanged += OnEntityChanged`
|
|
|
|
|
|
|
|
|
|
### Real-time Update Flow
|
|
|
|
|
|
|
|
|
|
1. User action → API endpoint
|
|
|
|
|
2. DbContext.SaveChangesAsync()
|
|
|
|
|
3. EntityChangeInterceptor captures changes
|
|
|
|
|
4. SignalR broadcast to all clients: `EntityChanged(module, id, operation)`
|
|
|
|
|
5. UI components receive event and refresh data
|
|
|
|
|
6. StateHasChanged() updates UI
|
|
|
|
|
|
|
|
|
|
**Example:** Job status changes appear instantly on JobDetailPage and JobListPage
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## JOB QUEUE SYSTEM
|
|
|
|
|
|
|
|
|
|
### Components
|
|
|
|
|
|
|
|
|
|
**Entity:** `QueueJob` (DiunaBI.Domain/Entities/QueueJob.cs)
|
|
|
|
|
- JobType: Import, Process
|
|
|
|
|
- JobStatus: Pending, Running, Completed, Failed, Retrying
|
|
|
|
|
- Priority: 0 = highest priority
|
|
|
|
|
- Retry: 30s → 2m → 5m delays, max 5 attempts
|
|
|
|
|
|
|
|
|
|
**Scheduler:** `JobSchedulerService`
|
|
|
|
|
- Reads Administration layer configs (Type=ImportWorker/ProcessWorker)
|
|
|
|
|
- Auto-creates jobs based on layer.Records configuration
|
|
|
|
|
- API endpoints: `/jobs/schedule/{apiKey}`, `/jobs/schedule/imports/{apiKey}`, `/jobs/schedule/processes/{apiKey}`
|
|
|
|
|
|
|
|
|
|
**Worker:** `JobWorkerService` (BackgroundService)
|
|
|
|
|
- Polls every 10 seconds
|
|
|
|
|
- Executes via PluginManager
|
|
|
|
|
- Exponential backoff on failures
|
|
|
|
|
- Rate limiting for Google API quota
|
|
|
|
|
- Real-time status updates via SignalR
|
|
|
|
|
|
|
|
|
|
**UI:** `Pages/Jobs/`
|
|
|
|
|
- Index.razor - Job list with filters, real-time updates
|
|
|
|
|
- Details.razor - Job detail with retry/cancel, real-time status
|
|
|
|
|
|
|
|
|
|
### Job Lifecycle
|
|
|
|
|
|
|
|
|
|
1. **Creation** - JobSchedulerService or manual via API
|
|
|
|
|
2. **Queued** - Status: Pending, sorted by Priority
|
|
|
|
|
3. **Execution** - JobWorkerService picks up, Status: Running
|
|
|
|
|
4. **Completion** - Status: Completed or Failed
|
|
|
|
|
5. **Retry** - On failure, Status: Retrying with exponential backoff
|
|
|
|
|
6. **Real-time** - All status changes broadcast via SignalR
|
|
|
|
|
|
|
|
|
|
**Statistics Endpoint:** `GET /jobs/stats`
|
|
|
|
|
```json
|
|
|
|
|
{
|
|
|
|
|
"pending": 5,
|
|
|
|
|
"running": 2,
|
|
|
|
|
"completed": 150,
|
|
|
|
|
"failed": 3,
|
|
|
|
|
"retrying": 1,
|
|
|
|
|
"total": 161
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## RECENT DEVELOPMENT
|
|
|
|
|
|
|
|
|
|
**Recent Commits (Dec 2-5, 2025):**
|
|
|
|
|
- **193127b:** SignalR for realtime entitychanges (Dec 4)
|
|
|
|
|
- **bf2beda, 942da18:** Build fixes (Dec 4)
|
|
|
|
|
- **a3fa8f9:** P2 import is working (Dec 4)
|
|
|
|
|
- **0e3b393:** WIP: p2 plugin (Dec 3)
|
|
|
|
|
- **445c07a:** Morska plugins refactor (Dec 2)
|
|
|
|
|
- **3f8e62f:** WIP: queue engine (Dec 2)
|
|
|
|
|
- **248106a:** Plugins little refactor (Dec 2)
|
|
|
|
|
- **587d4d6:** Pedrollo plugins (Dec 2)
|
|
|
|
|
- **e70a8dd:** Remember list filters (Dec 2)
|
|
|
|
|
- **89859cd:** Record history is working (Dec 1)
|
|
|
|
|
|
|
|
|
|
**Development Focus (Last 30 Days):**
|
|
|
|
|
1. ✅ Real-time updates (SignalR integration)
|
|
|
|
|
2. ✅ Job queue system (background worker, retry logic)
|
|
|
|
|
3. ✅ PedrolloPL plugin (P2 importer)
|
|
|
|
|
4. ✅ Record history tracking (audit trail)
|
|
|
|
|
5. ✅ UI reorganization (feature-based folders)
|
|
|
|
|
6. ✅ Plugin refactoring (base classes in Infrastructure)
|
|
|
|
|
7. ✅ Filter persistence (UI state management)
|
|
|
|
|
|
|
|
|
|
**Major Features Added:**
|
|
|
|
|
- SignalR real-time entity change notifications
|
|
|
|
|
- Background job processing with retry logic
|
|
|
|
|
- Record history with field-level diffs
|
|
|
|
|
- PedrolloPL P2 data importer
|
|
|
|
|
- UI reorganization (Pages/Layers, Pages/Jobs, Pages/DataInbox)
|
|
|
|
|
- Filter state persistence across sessions
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## CONFIGURATION
|
|
|
|
|
|
|
|
|
|
**Key Settings (appsettings.Development.json):**
|
|
|
|
|
- ConnectionStrings:SQLDatabase - SQL Server (localhost:21433, DB: DiunaBI-PedrolloPL)
|
|
|
|
|
- JwtSettings:SecurityKey, ExpiryDays (7)
|
|
|
|
|
- GoogleAuth:ClientId, RedirectUri
|
|
|
|
|
- apiKey, apiUser, apiPass - DataInbox API security
|
|
|
|
|
- exportDirectory - Google Drive folder ID for exports
|
|
|
|
|
- apiLocalUrl - localhost:5400
|
|
|
|
|
- InstanceName - DEV/PROD environment identifier
|
|
|
|
|
|
|
|
|
|
**Logging Configuration:**
|
|
|
|
|
```json
|
|
|
|
|
"Serilog": {
|
|
|
|
|
"MinimumLevel": {
|
|
|
|
|
"Default": "Information",
|
|
|
|
|
"Override": {
|
|
|
|
|
"Microsoft.AspNetCore": "Warning",
|
|
|
|
|
"Microsoft.EntityFrameworkCore.Database.Command": "Warning",
|
|
|
|
|
"Microsoft.EntityFrameworkCore.Infrastructure": "Warning",
|
|
|
|
|
"System.Net.Http.HttpClient": "Warning",
|
|
|
|
|
"Google.Apis": "Warning",
|
|
|
|
|
"DiunaBI.Core.Services.PluginManager": "Information"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**CORS Origins:**
|
|
|
|
|
- http://localhost:4200 (development)
|
|
|
|
|
- https://diuna.bim-it.pl (production)
|
|
|
|
|
- https://morska.diunabi.com (production)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## PATTERNS & ARCHITECTURE
|
|
|
|
|
|
|
|
|
|
**Design Patterns:**
|
|
|
|
|
- Clean Architecture (Domain → Application → Infrastructure → API)
|
|
|
|
|
- Plugin Pattern (dynamic loading, base classes, interface contracts)
|
|
|
|
|
- Interceptor Pattern (EF Core SaveChangesInterceptor for change tracking)
|
|
|
|
|
- Hub Pattern (SignalR for real-time notifications)
|
|
|
|
|
- Service Pattern (dependency injection throughout)
|
|
|
|
|
- Repository Pattern (EF Core DbContext as repository)
|
|
|
|
|
- Background Service Pattern (JobWorkerService for async processing)
|
|
|
|
|
|
|
|
|
|
**Tech Versions:**
|
|
|
|
|
- .NET 10.0 (upgraded from .NET 8.0)
|
|
|
|
|
- EF Core 10.0
|
|
|
|
|
- C# 13.0
|
|
|
|
|
- Blazor Server (net10.0)
|
|
|
|
|
- MAUI (net10.0-ios/android/windows/macos)
|
|
|
|
|
- MudBlazor 8.0
|
|
|
|
|
|
|
|
|
|
**Architectural Decisions:**
|
|
|
|
|
- Plugin base classes in Infrastructure for reusability
|
|
|
|
|
- SignalR for real-time updates (no polling)
|
|
|
|
|
- Background service for job processing (no external scheduler)
|
|
|
|
|
- Soft deletes with audit trails
|
|
|
|
|
- Foreign key RESTRICT to prevent accidental cascades
|
|
|
|
|
- Feature-based folder structure in UI
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## QUICK REFERENCE
|
|
|
|
|
|
|
|
|
|
**Database:**
|
|
|
|
|
- SQL Server with 47 EF Core migrations
|
|
|
|
|
- Auto-timestamps via GETUTCDATE() defaults
|
|
|
|
|
- Soft deletes (IsDeleted flag)
|
|
|
|
|
- Audit trails (CreatedBy, ModifiedBy, RecordHistory table)
|
|
|
|
|
|
|
|
|
|
**Build Process:**
|
|
|
|
|
- MSBuild target copies plugin DLLs to `bin/Plugins/` after build
|
|
|
|
|
- Plugins: DiunaBI.Plugins.Morska.dll, DiunaBI.Plugins.PedrolloPL.dll
|
|
|
|
|
|
|
|
|
|
**SignalR:**
|
|
|
|
|
- Hub: `/hubs/entitychanges`
|
|
|
|
|
- Broadcasts: `EntityChanged(module, id, operation)`
|
|
|
|
|
- Auto-reconnect enabled in UI
|
|
|
|
|
- Real-time updates for QueueJobs, Layers, Records
|
|
|
|
|
|
|
|
|
|
**Job Queue:**
|
|
|
|
|
- Auto-scheduling from layer configs (Type=ImportWorker/ProcessWorker)
|
|
|
|
|
- Background processing every 10 seconds
|
|
|
|
|
- Retry logic: 30s → 2m → 5m (max 5 retries)
|
|
|
|
|
- Priority-based execution (0 = highest)
|
|
|
|
|
- Real-time status updates via SignalR
|
|
|
|
|
|
|
|
|
|
**Plugins:**
|
|
|
|
|
- **Morska:** 4 importers, 12 processors, 1 exporter (~6,566 LOC)
|
|
|
|
|
- **PedrolloPL:** 1 importer (P2 data)
|
|
|
|
|
- Base classes: BaseDataImporter, BaseDataProcessor, BaseDataExporter
|
|
|
|
|
- Dynamic loading from `bin/Plugins/` at startup
|
|
|
|
|
|
|
|
|
|
**UI Structure:**
|
|
|
|
|
- Feature-based folders: Pages/Layers, Pages/Jobs, Pages/DataInbox
|
|
|
|
|
- Separate code-behind for complex logic (.razor.cs files)
|
|
|
|
|
- Inline @code for simple pages
|
|
|
|
|
- Organized components: Layout/, Auth/
|
|
|
|
|
- Filter state persistence across navigation
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## FILE PATHS REFERENCE
|
|
|
|
|
|
|
|
|
|
**Key Configuration:**
|
|
|
|
|
- API: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.API/appsettings.json`
|
|
|
|
|
- API Startup: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.API/Program.cs`
|
|
|
|
|
|
|
|
|
|
**SignalR:**
|
|
|
|
|
- Hub: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.API/Hubs/EntityChangeHub.cs`
|
|
|
|
|
- Interceptor: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Infrastructure/Interceptors/EntityChangeInterceptor.cs`
|
|
|
|
|
- UI Service: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.UI.Shared/Services/EntityChangeHubService.cs`
|
|
|
|
|
|
|
|
|
|
**Job System:**
|
|
|
|
|
- Controller: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.API/Controllers/JobsController.cs`
|
|
|
|
|
- Scheduler: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Infrastructure/Services/JobSchedulerService.cs`
|
|
|
|
|
- Worker: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Infrastructure/Services/JobWorkerService.cs`
|
|
|
|
|
- UI Pages: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.UI.Shared/Pages/Jobs/`
|
|
|
|
|
|
|
|
|
|
**Plugins:**
|
|
|
|
|
- Base Classes: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Infrastructure/Plugins/`
|
|
|
|
|
- Morska: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Plugins.Morska/`
|
|
|
|
|
- PedrolloPL: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Plugins.PedrolloPL/`
|
|
|
|
|
|
|
|
|
|
**Migrations:**
|
|
|
|
|
- Latest: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.Infrastructure/Migrations/20251201165810_RecordHistory.cs`
|
|
|
|
|
|
|
|
|
|
**UI Components:**
|
|
|
|
|
- Pages: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.UI.Shared/Pages/`
|
|
|
|
|
- Components: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.UI.Shared/Components/`
|
|
|
|
|
- Services: `/Users/mz/Projects/Diuna/DiunaBI/DiunaBI.UI.Shared/Services/`
|
