mz/DiunaBI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f10dfe629e53a5e53440dbcf31869378cacf866f
DiunaBI/CLAUDE.md
Michał Zieliński f10dfe629e WIP: AI Validator
2025-12-15 20:05:26 +01:00

15 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

DiunaBI is a full-stack Business Intelligence platform built on .NET 10.0 with a multi-tier Clean Architecture. It provides a plugin-based system for importing, processing, and exporting business data with real-time updates, job queue management, and comprehensive audit trails.

Tech Stack:

  • Backend: ASP.NET Core 10.0 Web API
  • Frontend: Blazor Server + MAUI Mobile (iOS, Android, Windows, macOS)
  • Database: SQL Server + EF Core 10.0
  • UI Framework: MudBlazor 8.0
  • Real-time: SignalR (EntityChangeHub)
  • Authentication: JWT Bearer + Google OAuth
  • External APIs: Google Sheets API, Google Drive API
  • Logging: Serilog (Console, File)

Common Commands

Build and Run

# Build the entire solution
dotnet build DiunaBI.sln

# Build specific project
dotnet build DiunaBI.API/DiunaBI.API.csproj

# Run API (backend)
dotnet run --project DiunaBI.API/DiunaBI.API.csproj

# Run Web UI (Blazor Server)
dotnet run --project DiunaBI.UI.Web/DiunaBI.UI.Web.csproj

# Clean build artifacts
dotnet clean DiunaBI.sln

Database Migrations

# Add new migration
dotnet ef migrations add <MigrationName> --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# Apply migrations to database
dotnet ef database update --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# Remove last migration (if not applied)
dotnet ef migrations remove --project DiunaBI.Infrastructure --startup-project DiunaBI.API

# List all migrations
dotnet ef migrations list --project DiunaBI.Infrastructure --startup-project DiunaBI.API

Testing

# Run all tests
dotnet test DiunaBI.Tests/DiunaBI.Tests.csproj

# Run tests with detailed output
dotnet test DiunaBI.Tests/DiunaBI.Tests.csproj --logger "console;verbosity=detailed"

Plugin Development

# Build plugin project (automatically copies DLL to API bin/Plugins/)
dotnet build DiunaBI.Plugins.Morska/DiunaBI.Plugins.Morska.csproj
dotnet build DiunaBI.Plugins.PedrolloPL/DiunaBI.Plugins.PedrolloPL.csproj

# Plugins are auto-loaded from bin/Plugins/ at API startup

Docker (Production)

# Build API container (with Morska plugin)
docker build -f DiunaBI.API/Dockerfile -t diunabi-api --build-arg PLUGIN_PROJECT=DiunaBI.Plugins.Morska .

# Build API container (with PedrolloPL plugin)
docker build -f DiunaBI.API/Dockerfile -t diunabi-api --build-arg PLUGIN_PROJECT=DiunaBI.Plugins.PedrolloPL .

Architecture

Solution Structure (10 Projects)

DiunaBI.API - ASP.NET Core Web API

  • Controllers: Auth, Layers, Jobs, DataInbox
  • Hubs: EntityChangeHub (SignalR)
  • Services: GoogleAuthService, JwtTokenService
  • API Key authorization via [ApiKeyAuth] attribute

DiunaBI.Domain - Domain entities (9 entities)

  • User, Layer, Record, RecordHistory, QueueJob, DataInbox, ProcessSource

DiunaBI.Application - DTOs and application models

  • LayerDto, RecordDto, UserDto, RecordHistoryDto, JobDto, PagedResult

DiunaBI.Infrastructure - Data access and core services

  • Data: AppDbContext, 47 EF Core migrations
  • Interceptors: EntityChangeInterceptor (auto-broadcasts DB changes via SignalR)
  • Services: PluginManager, JobSchedulerService, JobWorkerService
  • Helpers: GoogleSheetsHelper, GoogleDriveHelper
  • Plugin Base Classes: BaseDataImporter, BaseDataProcessor, BaseDataExporter

DiunaBI.UI.Web - Blazor Server web application

DiunaBI.UI.Mobile - MAUI mobile application (iOS, Android, Windows, macOS)

DiunaBI.UI.Shared - Shared Blazor component library

  • Pages/: Feature-based folders (Layers/, Jobs/, DataInbox/)
  • Components/: Layout/ (MainLayout, EmptyLayout, Routes), Auth/ (AuthGuard, LoginCard)
  • Services/: LayerService, JobService, DataInboxService, EntityChangeHubService, AuthService

DiunaBI.Plugins.Morska - Production plugin (17 total)

  • 4 Importers: Standard, D1, D3, FK2
  • 12 Processors: D6, T1, T3, T4, T5 variants
  • 1 Exporter: Google Sheets export

DiunaBI.Plugins.PedrolloPL - Production plugin (1 total)

  • 1 Importer: B3 (DataInbox → Layer with dictionary mapping)

DiunaBI.Tests - Unit and integration tests

Data Flow Architecture

  1. Import Flow: External data → DataInbox API → Import Plugin → Import Layer
  2. Process Flow: Import Layer → Process Plugin → Processed Layer
  3. Export Flow: Processed Layer → Export Plugin → Google Sheets/Drive
  4. Real-time Updates: DB Change → EntityChangeInterceptor → SignalR Hub → All Clients

Plugin System

Plugin Discovery:

  • Plugins loaded from bin/Plugins/ directory at startup
  • PluginManager scans assemblies for IDataImporter, IDataProcessor, IDataExporter implementations
  • Plugins auto-registered in DI container

Plugin Base Classes (in DiunaBI.Infrastructure/Plugins/):

  • BaseDataImporter: Abstract base for importers, access to AppDbContext, GoogleSheetsHelper, GoogleDriveHelper
  • BaseDataProcessor: Abstract base for processors, access to AppDbContext, PluginManager
  • BaseDataExporter: Abstract base for exporters, access to AppDbContext, GoogleSheetsHelper, GoogleDriveHelper

Plugin Execution:

  • Importers: Read external data sources, create Layer + Records
  • Processors: Read source Layers, apply transformations, create target Layers + Records
  • Exporters: Read Layers, write to Google Sheets/Drive

Plugin Configuration:

  • Stored in Administration layers (Type = ImportWorker/ProcessWorker)
  • Records with Code = "Plugin", "Priority", "MaxRetries" define job configs
  • JobSchedulerService reads configs and creates QueueJobs

Job Queue System

Components:

  • QueueJob entity: LayerId, PluginName, JobType (Import/Process), Status (Pending/Running/Completed/Failed/Retrying), Priority (0 = highest)
  • JobSchedulerService: Creates jobs from Administration layer configs
  • JobWorkerService: Background service polling every 5 seconds, executes jobs via PluginManager
  • Retry logic: 30s → 2m → 5m delays, max 5 retries

Job Lifecycle:

  1. Creation: JobSchedulerService or manual via /jobs/create-for-layer/{layerId}
  2. Queued: Status = Pending, sorted by CreatedAt DESC then Priority ASC
  3. Execution: JobWorkerService picks up, Status = Running
  4. Completion: Status = Completed or Failed
  5. Retry: On failure, Status = Retrying with exponential backoff
  6. Real-time: All status changes broadcast via SignalR to UI

Job Scheduling Endpoints:

  • POST /jobs/ui/schedule - Schedule all jobs (JWT auth for UI users)
  • POST /jobs/ui/schedule/imports - Schedule import jobs only (JWT auth)
  • POST /jobs/ui/schedule/processes - Schedule process jobs only (JWT auth)
  • POST /jobs/schedule/{apiKey} - Schedule all jobs (API key auth for cron)
  • POST /jobs/schedule/imports/{apiKey} - Schedule import jobs (API key auth)
  • POST /jobs/schedule/processes/{apiKey} - Schedule process jobs (API key auth)

Real-time Updates (SignalR)

Architecture:

  • Hub: /hubs/entitychanges (requires JWT authentication)
  • Interceptor: EntityChangeInterceptor captures EF Core changes (Added, Modified, Deleted)
  • Broadcast: After SaveChanges, sends EntityChanged(module, id, operation) to all clients
  • Modules: QueueJobs, Layers, Records, RecordHistory

UI Integration:

  • EntityChangeHubService: Singleton service initialized after authentication in MainLayout
  • Components subscribe: HubService.EntityChanged += OnEntityChanged
  • Auto-reconnect enabled
  • Pages with real-time updates: Jobs/Index, Jobs/Details, Layers/Index, Layers/Details, DataInbox/Index

Authentication Flow:

  1. User logs in with Google OAuth → JWT token stored in localStorage
  2. TokenProvider.Token populated in AuthService
  3. MainLayout subscribes to AuthenticationStateChanged event
  4. On auth success, SignalR connection initialized with JWT token
  5. Token sent via accessTokenProvider in HubConnection options

Authentication & Security

Google OAuth Flow:

  1. Client exchanges Google ID token → POST /auth/apiToken
  2. GoogleAuthService validates with Google, maps to internal User entity
  3. Returns JWT (7-day expiration, HS256 signing)
  4. JWT required on all protected endpoints except /auth/apiToken, /health
  5. UserId extraction middleware sets X-UserId header for audit trails

API Key Authentication:

  • Custom [ApiKeyAuth] attribute for cron job endpoints
  • X-API-Key header with constant-time comparison
  • Used for DataInbox external endpoints and job scheduling

Security Features:

  • Rate limiting: 100 req/min general, 10 req/min auth
  • Security headers: XSS, clickjacking, MIME sniffing protection
  • Input validation: Pagination limits (1-1000), Base64 size limits (10MB)
  • Stack trace hiding: Generic error messages in production
  • CORS: Configured for localhost:4200, diuna.bim-it.pl, morska.diunabi.com

Database Schema

47 EF Core Migrations - All in DiunaBI.Infrastructure/Migrations/

Key Entities:

User

  • Id (Guid), Email, UserName
  • Google OAuth identity
  • Constant: User.AutoImportUserId = "f392209e-123e-4651-a5a4-0b1d6cf9ff9d" (system operations)

Layer

  • Number, Name, Type (Import/Processed/Administration/Dictionary)
  • ParentId (hierarchical relationships)
  • IsDeleted (soft delete), IsCancelled (processing control)
  • CreatedAt, ModifiedAt, CreatedById, ModifiedById (audit trail)

Record

  • Code (unique identifier per layer), LayerId
  • Value1-Value32 (double?), Desc1 (string, max 10000 chars)
  • IsDeleted (soft delete)
  • Full audit trail via RecordHistory

RecordHistory (Migration 47)

  • RecordId, LayerId, ChangedAt, ChangedById
  • ChangeType (Created/Updated/Deleted)
  • Code, Desc1 (snapshot)
  • ChangedFields (comma-separated), ChangesSummary (JSON old/new values)
  • Indexes: (RecordId, ChangedAt), (LayerId, ChangedAt)

QueueJob

  • LayerId, PluginName, JobType, Priority, Status
  • RetryCount, MaxRetries (default 5)
  • CreatedAt, ModifiedAt, LastAttemptAt, CompletedAt
  • CreatedById, ModifiedById (uses User.AutoImportUserId for system jobs)

DataInbox

  • Name, Source (identifiers), Data (base64-encoded JSON array)
  • Used by importers to stage incoming data

ProcessSource

  • SourceLayerId, TargetLayerId (defines layer processing relationships)

Audit Trail Patterns:

  • All entities have CreatedAt, ModifiedAt with GETUTCDATE() defaults
  • Foreign keys to Users: CreatedById, ModifiedById
  • Soft deletes via IsDeleted flag
  • RecordHistory tracks field-level changes with JSON diffs

UI Organization

Feature-Based Structure (as of Dec 5, 2025):

  • Pages/Layers/ - Index.razor + Details.razor (list, detail, edit, history)
  • Pages/Jobs/ - Index.razor + Details.razor (list, detail, retry, cancel, scheduling)
  • Pages/DataInbox/ - Index.razor + Details.razor (list, detail, base64 decode)
  • Components/Layout/ - MainLayout, EmptyLayout, Routes
  • Components/Auth/ - AuthGuard, LoginCard

Code-Behind Pattern:

  • Complex pages (50+ lines logic): Separate .razor.cs files
  • Simple pages: Inline @code blocks
  • Namespaces: DiunaBI.UI.Shared.Pages.{Feature}

Filter State Persistence:

  • LayerFilterStateService, DataInboxFilterStateService
  • Singleton services remember search, type, page selections across navigation

Timezone Handling:

  • DateTimeHelper service detects browser timezone via JS Interop
  • All dates stored as UTC in DB, converted to user's local timezone for display
  • Format: "yyyy-MM-dd HH:mm:ss"

Configuration

Environment Variables (appsettings.Development.json):

  • ConnectionStrings:SQLDatabase - SQL Server connection (localhost:21433, DB: DiunaBI-PedrolloPL)
  • JwtSettings:SecurityKey, JwtSettings:ExpiryDays (7)
  • GoogleAuth:ClientId, GoogleAuth:RedirectUri
  • apiKey, apiUser, apiPass - DataInbox API security
  • exportDirectory - Google Drive folder ID for exports
  • InstanceName - DEV/PROD environment identifier

CORS Origins:

Logging:

  • Serilog with Console + File sinks
  • Override levels: Microsoft.AspNetCore = Warning, EF Core = Warning, Google.Apis = Warning
  • Sensitive data logging only in Development

Development Patterns

When Adding a New Plugin

  1. Create new project: DiunaBI.Plugins.{Name}
  2. Reference: DiunaBI.Domain, DiunaBI.Infrastructure
  3. Inherit from: BaseDataImporter/BaseDataProcessor/BaseDataExporter
  4. Implement abstract methods: ImportAsync/ProcessAsync/ExportAsync, ValidateConfiguration
  5. Build triggers automatic copy to bin/Plugins/ via MSBuild target
  6. No registration needed - PluginManager auto-discovers at startup

When Adding a New Entity

  1. Create entity in DiunaBI.Domain/Entities/
  2. Add DbSet to AppDbContext
  3. Create migration: dotnet ef migrations add {Name} --project DiunaBI.Infrastructure --startup-project DiunaBI.API
  4. If entity needs real-time updates, add module name to EntityChangeInterceptor
  5. Create DTO in DiunaBI.Application/DTOs/
  6. Add controller in DiunaBI.API/Controllers/ with [Authorize] and [RequireRateLimit("api")]

When Adding Real-time Updates to a UI Page

  1. Inject EntityChangeHubService
  2. Subscribe to EntityChanged event in OnInitializedAsync
  3. Filter by module name
  4. Call StateHasChanged() in event handler
  5. Implement IDisposable, unsubscribe in Dispose
  6. Test both initial load and SignalR updates

When Modifying Job System

  • JobSchedulerService: Changes to job creation from layer configs
  • JobWorkerService: Changes to job execution, retry logic, rate limiting
  • QueueJob entity: Changes to job schema require migration
  • Jobs UI: Real-time updates required, test SignalR broadcasts

Security Considerations

  • Never log sensitive data (tokens, passwords, API keys) except in Development
  • Use generic error messages in production (no stack traces)
  • All new endpoints require [Authorize] unless explicitly [AllowAnonymous]
  • API key endpoints require [ApiKeyAuth] and [AllowAnonymous]
  • Validate pagination parameters (1-1000 range)
  • Use constant-time comparison for API keys
  • Rate limit all public endpoints

Known Limitations

  • JobWorkerService polls every 5 seconds (not event-driven)
  • Google Sheets API quota: 5-second delay after import jobs
  • Retry delays fixed: 30s → 2m → 5m (not configurable per job)
  • Plugin configuration stored in Records (not strongly typed)
  • No plugin versioning or hot-reload support
  • SignalR requires JWT authentication (no anonymous connections)

Important Notes

  • DO NOT commit .env file (contains secrets)
  • DO NOT modify migration files after applying to production
  • ALWAYS use User.AutoImportUserId for system-created entities (jobs, automated processes)
  • ALWAYS implement IDisposable for pages subscribing to SignalR events
  • ALWAYS test real-time updates when modifying entities with SignalR broadcasts
  • Plugin DLLs are auto-copied to bin/Plugins/ on build via MSBuild target
  • Database changes require migration AND applying to production via deployment scripts
  • Foreign keys use RESTRICT (not CASCADE) to prevent accidental data loss (Migration 45)
  • Soft deletes via IsDeleted flag, not physical deletion
  • Timezone handling - store UTC, display local (via DateTimeHelper)