DiunaBI/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

DiunaBI is a full-stack Business Intelligence platform built on .NET 10.0 with a multi-tier Clean Architecture. It provides a plugin-based system for importing, processing, and exporting business data with real-time updates, job queue management, and comprehensive audit trails.

**Tech Stack:**
- Backend: ASP.NET Core 10.0 Web API
- Frontend: Blazor Server + MAUI Mobile (iOS, Android, Windows, macOS)
- Database: SQL Server + EF Core 10.0
- UI Framework: MudBlazor 8.0
- Real-time: SignalR (EntityChangeHub)
- Authentication: JWT Bearer + Google OAuth
- External APIs: Google Sheets API, Google Drive API
- Logging: Serilog (Console, File)

## Common Commands

### Build and Run

```bash
# Build the entire solution
dotnet build DiunaBI.sln

# Build specific project
dotnet build DiunaBI.API/DiunaBI.API.csproj

# Run API (backend)
dotnet run --project DiunaBI.API/DiunaBI.API.csproj

# Run Web UI (Blazor Server)
dotnet run --project DiunaBI.UI.Web/DiunaBI.UI.Web.csproj

# Clean build artifacts
dotnet clean DiunaBI.sln
```

### Database Migrations

```bash
# Add new migration
dotnet ef migrations add <MigrationName> --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# Apply migrations to database
dotnet ef database update --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# Remove last migration (if not applied)
dotnet ef migrations remove --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# List all migrations
dotnet ef migrations list --project DiunaBI.Infrastructure --startup-project DiunaBI.API
```

### Testing

```bash
# Run all tests
dotnet test DiunaBI.Tests/DiunaBI.Tests.csproj

# Run tests with detailed output
dotnet test DiunaBI.Tests/DiunaBI.Tests.csproj --logger "console;verbosity=detailed"
```

### Plugin Development

```bash
# Build plugin project (automatically copies DLL to API bin/Plugins/)
dotnet build DiunaBI.Plugins.Morska/DiunaBI.Plugins.Morska.csproj
dotnet build DiunaBI.Plugins.PedrolloPL/DiunaBI.Plugins.PedrolloPL.csproj

# Plugins are auto-loaded from bin/Plugins/ at API startup
```

### Docker (Production)

```bash
# Build API container (with Morska plugin)
docker build -f DiunaBI.API/Dockerfile -t diunabi-api --build-arg PLUGIN_PROJECT=DiunaBI.Plugins.Morska .

# Build API container (with PedrolloPL plugin)
docker build -f DiunaBI.API/Dockerfile -t diunabi-api --build-arg PLUGIN_PROJECT=DiunaBI.Plugins.PedrolloPL .
```

## Architecture

### Solution Structure (10 Projects)

**DiunaBI.API** - ASP.NET Core Web API
- Controllers: Auth, Layers, Jobs, DataInbox
- Hubs: EntityChangeHub (SignalR)
- Services: GoogleAuthService, JwtTokenService
- API Key authorization via [ApiKeyAuth] attribute

**DiunaBI.Domain** - Domain entities (9 entities)
- User, Layer, Record, RecordHistory, QueueJob, DataInbox, ProcessSource

**DiunaBI.Application** - DTOs and application models
- LayerDto, RecordDto, UserDto, RecordHistoryDto, JobDto, PagedResult<T>

**DiunaBI.Infrastructure** - Data access and core services
- Data: AppDbContext, 47 EF Core migrations
- Interceptors: EntityChangeInterceptor (auto-broadcasts DB changes via SignalR)
- Services: PluginManager, JobSchedulerService, JobWorkerService
- Helpers: GoogleSheetsHelper, GoogleDriveHelper
- Plugin Base Classes: BaseDataImporter, BaseDataProcessor, BaseDataExporter

**DiunaBI.UI.Web** - Blazor Server web application

**DiunaBI.UI.Mobile** - MAUI mobile application (iOS, Android, Windows, macOS)

**DiunaBI.UI.Shared** - Shared Blazor component library
- Pages/: Feature-based folders (Layers/, Jobs/, DataInbox/)
- Components/: Layout/ (MainLayout, EmptyLayout, Routes), Auth/ (AuthGuard, LoginCard)
- Services/: LayerService, JobService, DataInboxService, EntityChangeHubService, AuthService

**DiunaBI.Plugins.Morska** - Production plugin (17 total)
- 4 Importers: Standard, D1, D3, FK2
- 12 Processors: D6, T1, T3, T4, T5 variants
- 1 Exporter: Google Sheets export

**DiunaBI.Plugins.PedrolloPL** - Production plugin (1 total)
- 1 Importer: B3 (DataInbox → Layer with dictionary mapping)

**DiunaBI.Tests** - Unit and integration tests

### Data Flow Architecture

1. **Import Flow**: External data → DataInbox API → Import Plugin → Import Layer
2. **Process Flow**: Import Layer → Process Plugin → Processed Layer
3. **Export Flow**: Processed Layer → Export Plugin → Google Sheets/Drive
4. **Real-time Updates**: DB Change → EntityChangeInterceptor → SignalR Hub → All Clients

### Plugin System

**Plugin Discovery:**
- Plugins loaded from `bin/Plugins/` directory at startup
- PluginManager scans assemblies for IDataImporter, IDataProcessor, IDataExporter implementations
- Plugins auto-registered in DI container

**Plugin Base Classes** (in DiunaBI.Infrastructure/Plugins/):
- `BaseDataImporter`: Abstract base for importers, access to AppDbContext, GoogleSheetsHelper, GoogleDriveHelper
- `BaseDataProcessor`: Abstract base for processors, access to AppDbContext, PluginManager
- `BaseDataExporter`: Abstract base for exporters, access to AppDbContext, GoogleSheetsHelper, GoogleDriveHelper

**Plugin Execution:**
- Importers: Read external data sources, create Layer + Records
- Processors: Read source Layers, apply transformations, create target Layers + Records
- Exporters: Read Layers, write to Google Sheets/Drive

**Plugin Configuration:**
- Stored in Administration layers (Type = ImportWorker/ProcessWorker)
- Records with Code = "Plugin", "Priority", "MaxRetries" define job configs
- JobSchedulerService reads configs and creates QueueJobs

### Job Queue System

**Components:**
- `QueueJob` entity: LayerId, PluginName, JobType (Import/Process), Status (Pending/Running/Completed/Failed/Retrying), Priority (0 = highest)
- `JobSchedulerService`: Creates jobs from Administration layer configs
- `JobWorkerService`: Background service polling every 5 seconds, executes jobs via PluginManager
- Retry logic: 30s → 2m → 5m delays, max 5 retries

**Job Lifecycle:**
1. Creation: JobSchedulerService or manual via `/jobs/create-for-layer/{layerId}`
2. Queued: Status = Pending, sorted by CreatedAt DESC then Priority ASC
3. Execution: JobWorkerService picks up, Status = Running
4. Completion: Status = Completed or Failed
5. Retry: On failure, Status = Retrying with exponential backoff
6. Real-time: All status changes broadcast via SignalR to UI

**Job Scheduling Endpoints:**
- `POST /jobs/ui/schedule` - Schedule all jobs (JWT auth for UI users)
- `POST /jobs/ui/schedule/imports` - Schedule import jobs only (JWT auth)
- `POST /jobs/ui/schedule/processes` - Schedule process jobs only (JWT auth)
- `POST /jobs/schedule/{apiKey}` - Schedule all jobs (API key auth for cron)
- `POST /jobs/schedule/imports/{apiKey}` - Schedule import jobs (API key auth)
- `POST /jobs/schedule/processes/{apiKey}` - Schedule process jobs (API key auth)

### Real-time Updates (SignalR)

**Architecture:**
- Hub: `/hubs/entitychanges` (requires JWT authentication)
- Interceptor: `EntityChangeInterceptor` captures EF Core changes (Added, Modified, Deleted)
- Broadcast: After SaveChanges, sends `EntityChanged(module, id, operation)` to all clients
- Modules: QueueJobs, Layers, Records, RecordHistory

**UI Integration:**
- `EntityChangeHubService`: Singleton service initialized after authentication in MainLayout
- Components subscribe: `HubService.EntityChanged += OnEntityChanged`
- Auto-reconnect enabled
- Pages with real-time updates: Jobs/Index, Jobs/Details, Layers/Index, Layers/Details, DataInbox/Index

**Authentication Flow:**
1. User logs in with Google OAuth → JWT token stored in localStorage
2. `TokenProvider.Token` populated in AuthService
3. `MainLayout` subscribes to `AuthenticationStateChanged` event
4. On auth success, SignalR connection initialized with JWT token
5. Token sent via `accessTokenProvider` in HubConnection options

### Authentication & Security

**Google OAuth Flow:**
1. Client exchanges Google ID token → `POST /auth/apiToken`
2. GoogleAuthService validates with Google, maps to internal User entity
3. Returns JWT (7-day expiration, HS256 signing)
4. JWT required on all protected endpoints except `/auth/apiToken`, `/health`
5. UserId extraction middleware sets X-UserId header for audit trails

**API Key Authentication:**
- Custom [ApiKeyAuth] attribute for cron job endpoints
- X-API-Key header with constant-time comparison
- Used for DataInbox external endpoints and job scheduling

**Security Features:**
- Rate limiting: 100 req/min general, 10 req/min auth
- Security headers: XSS, clickjacking, MIME sniffing protection
- Input validation: Pagination limits (1-1000), Base64 size limits (10MB)
- Stack trace hiding: Generic error messages in production
- CORS: Configured for localhost:4200, diuna.bim-it.pl, morska.diunabi.com

### Database Schema

**47 EF Core Migrations** - All in DiunaBI.Infrastructure/Migrations/

**Key Entities:**

**User**
- Id (Guid), Email, UserName
- Google OAuth identity
- Constant: `User.AutoImportUserId = "f392209e-123e-4651-a5a4-0b1d6cf9ff9d"` (system operations)

**Layer**
- Number, Name, Type (Import/Processed/Administration/Dictionary)
- ParentId (hierarchical relationships)
- IsDeleted (soft delete), IsCancelled (processing control)
- CreatedAt, ModifiedAt, CreatedById, ModifiedById (audit trail)

**Record**
- Code (unique identifier per layer), LayerId
- Value1-Value32 (double?), Desc1 (string, max 10000 chars)
- IsDeleted (soft delete)
- Full audit trail via RecordHistory

**RecordHistory** (Migration 47)
- RecordId, LayerId, ChangedAt, ChangedById
- ChangeType (Created/Updated/Deleted)
- Code, Desc1 (snapshot)
- ChangedFields (comma-separated), ChangesSummary (JSON old/new values)
- Indexes: (RecordId, ChangedAt), (LayerId, ChangedAt)

**QueueJob**
- LayerId, PluginName, JobType, Priority, Status
- RetryCount, MaxRetries (default 5)
- CreatedAt, ModifiedAt, LastAttemptAt, CompletedAt
- CreatedById, ModifiedById (uses User.AutoImportUserId for system jobs)

**DataInbox**
- Name, Source (identifiers), Data (base64-encoded JSON array)
- Used by importers to stage incoming data

**ProcessSource**
- SourceLayerId, TargetLayerId (defines layer processing relationships)

**Audit Trail Patterns:**
- All entities have CreatedAt, ModifiedAt with GETUTCDATE() defaults
- Foreign keys to Users: CreatedById, ModifiedById
- Soft deletes via IsDeleted flag
- RecordHistory tracks field-level changes with JSON diffs

### UI Organization

**Feature-Based Structure** (as of Dec 5, 2025):
- `Pages/Layers/` - Index.razor + Details.razor (list, detail, edit, history)
- `Pages/Jobs/` - Index.razor + Details.razor (list, detail, retry, cancel, scheduling)
- `Pages/DataInbox/` - Index.razor + Details.razor (list, detail, base64 decode)
- `Components/Layout/` - MainLayout, EmptyLayout, Routes
- `Components/Auth/` - AuthGuard, LoginCard

**Code-Behind Pattern:**
- Complex pages (50+ lines logic): Separate `.razor.cs` files
- Simple pages: Inline `@code` blocks
- Namespaces: `DiunaBI.UI.Shared.Pages.{Feature}`

**Filter State Persistence:**
- LayerFilterStateService, DataInboxFilterStateService
- Singleton services remember search, type, page selections across navigation

**Timezone Handling:**
- DateTimeHelper service detects browser timezone via JS Interop
- All dates stored as UTC in DB, converted to user's local timezone for display
- Format: "yyyy-MM-dd HH:mm:ss"

## Configuration

**Environment Variables** (appsettings.Development.json):
- `ConnectionStrings:SQLDatabase` - SQL Server connection (localhost:21433, DB: DiunaBI-PedrolloPL)
- `JwtSettings:SecurityKey`, `JwtSettings:ExpiryDays` (7)
- `GoogleAuth:ClientId`, `GoogleAuth:RedirectUri`
- `apiKey`, `apiUser`, `apiPass` - DataInbox API security
- `exportDirectory` - Google Drive folder ID for exports
- `InstanceName` - DEV/PROD environment identifier

**CORS Origins:**
- http://localhost:4200 (development)
- https://diuna.bim-it.pl (production)
- https://morska.diunabi.com (production)

**Logging:**
- Serilog with Console + File sinks
- Override levels: Microsoft.AspNetCore = Warning, EF Core = Warning, Google.Apis = Warning
- Sensitive data logging only in Development

## Development Patterns

### When Adding a New Plugin

1. Create new project: `DiunaBI.Plugins.{Name}`
2. Reference: DiunaBI.Domain, DiunaBI.Infrastructure
3. Inherit from: BaseDataImporter/BaseDataProcessor/BaseDataExporter
4. Implement abstract methods: ImportAsync/ProcessAsync/ExportAsync, ValidateConfiguration
5. Build triggers automatic copy to `bin/Plugins/` via MSBuild target
6. No registration needed - PluginManager auto-discovers at startup

### When Adding a New Entity

1. Create entity in DiunaBI.Domain/Entities/
2. Add DbSet to AppDbContext
3. Create migration: `dotnet ef migrations add {Name} --project DiunaBI.Infrastructure --startup-project DiunaBI.API`
4. If entity needs real-time updates, add module name to EntityChangeInterceptor
5. Create DTO in DiunaBI.Application/DTOs/
6. Add controller in DiunaBI.API/Controllers/ with [Authorize] and [RequireRateLimit("api")]

### When Adding Real-time Updates to a UI Page

1. Inject EntityChangeHubService
2. Subscribe to EntityChanged event in OnInitializedAsync
3. Filter by module name
4. Call StateHasChanged() in event handler
5. Implement IDisposable, unsubscribe in Dispose
6. Test both initial load and SignalR updates

### When Modifying Job System

- JobSchedulerService: Changes to job creation from layer configs
- JobWorkerService: Changes to job execution, retry logic, rate limiting
- QueueJob entity: Changes to job schema require migration
- Jobs UI: Real-time updates required, test SignalR broadcasts

### Security Considerations

- Never log sensitive data (tokens, passwords, API keys) except in Development
- Use generic error messages in production (no stack traces)
- All new endpoints require [Authorize] unless explicitly [AllowAnonymous]
- API key endpoints require [ApiKeyAuth] and [AllowAnonymous]
- Validate pagination parameters (1-1000 range)
- Use constant-time comparison for API keys
- Rate limit all public endpoints

## Known Limitations

- JobWorkerService polls every 5 seconds (not event-driven)
- Google Sheets API quota: 5-second delay after import jobs
- Retry delays fixed: 30s → 2m → 5m (not configurable per job)
- Plugin configuration stored in Records (not strongly typed)
- No plugin versioning or hot-reload support
- SignalR requires JWT authentication (no anonymous connections)

## Important Notes

- **DO NOT** commit `.env` file (contains secrets)
- **DO NOT** modify migration files after applying to production
- **ALWAYS** use User.AutoImportUserId for system-created entities (jobs, automated processes)
- **ALWAYS** implement IDisposable for pages subscribing to SignalR events
- **ALWAYS** test real-time updates when modifying entities with SignalR broadcasts
- **Plugin DLLs** are auto-copied to `bin/Plugins/` on build via MSBuild target
- **Database changes** require migration AND applying to production via deployment scripts
- **Foreign keys** use RESTRICT (not CASCADE) to prevent accidental data loss (Migration 45)
- **Soft deletes** via IsDeleted flag, not physical deletion
- **Timezone handling** - store UTC, display local (via DateTimeHelper)