Invite de système pour spécialiste en optimisation des performances

# TODO_perf-tuning.md ## Context ### Current Performance Profile - **System Overview**: Monolithic Node.js/TypeScript backend with React frontend, PostgreSQL database, Redis cache layer - **Identified Bottlenecks** (ranked by impact): 1. N+1 query pattern in user profile aggregation endpoint (3x database round-trips per request) 2. Unoptimized React list rendering causing excessive re-renders (layout thrashing) 3. Synchronous JSON serialization blocking event loop under high concurrency 4. Missing composite indexes on frequently queried timestamp+status columns 5. Memory leak in WebSocket connection handler (growing heap under sustained connections) ### Baseline Metrics | Metric | Current Value | Target Value | |--------|--------------|--------------| | Response time p50 | 245ms | <150ms | | Response time p95 | 890ms | <300ms | | Response time p99 | 1.8s | <500ms | | Throughput | 850 RPS | >1500 RPS | | Memory heap size | 2.4GB peak | <1.5GB peak | | GC pause time | 180ms max | <50ms max | | Database query time | 120ms avg | <50ms avg | | Frontend LCP | 3.2s | <2.5s | | Frontend CLS | 0.15 | <0.1 | | Frontend FID | 120ms | <100ms | ### Target SLAs - API response time: <200ms (p95) for standard endpoints - Database queries: <50ms (p95) for OLTP operations - Frontend Core Web Vitals: LCP <2.5s, FID <100ms, CLS <0.1 - System availability: 99.9% with no memory leaks under sustained load ### Optimization Priorities 1. Eliminate N+1 query patterns (highest impact, low complexity) 2. Optimize React rendering with virtualization and memoization 3. Implement async JSON serialization with worker threads 4. Add composite database indexes for common query patterns 5. Fix WebSocket memory leak and implement connection pooling --- ## Performance Optimization Plan - [ ] **PERF-PLAN-1.1 [Database Query Optimization]**: - **Bottleneck**: N+1 query pattern in user profile aggregation causing 3x database round-trips per request - **Technique**: Implement batch loading with `Promise.all()` and `IN` clauses; use DataLoader pattern for nested resources - **Expected Impact**: 60-70% reduction in database round-trips; 40% reduction in p95 response time - **Trade-offs**: Slightly more complex query logic; requires careful handling of partial failures - [ ] **PERF-PLAN-1.2 [Frontend Rendering Optimization]**: - **Bottleneck**: Unoptimized list rendering causing layout thrashing and excessive re-renders - **Technique**: Implement virtual scrolling with `react-window`, add `React.memo()` for expensive components, use `useMemo`/`useCallback` for stable references - **Expected Impact**: 70% reduction in render time; 50% reduction in main thread blocking - **Trade-offs**: Increased component complexity; requires careful dependency management - [ ] **PERF-PLAN-1.3 [Async Serialization Optimization]**: - **Bottleneck**: Synchronous JSON serialization blocking event loop under high concurrency - **Technique**: Move serialization to worker threads using `worker_threads` module; implement streaming serialization for large payloads - **Expected Impact**: 45% improvement in throughput; 60% reduction in event loop blocking - **Trade-offs**: Additional memory overhead for worker threads; complexity in error handling - [ ] **PERF-PLAN-1.4 [Database Indexing Strategy]**: - **Bottleneck**: Missing composite indexes causing full table scans on timestamp+status queries - **Technique**: Add composite indexes on `(created_at, status)` and `(user_id, created_at)` columns; implement partial indexes for active records - **Expected Impact**: 80% reduction in query execution time for filtered queries - **Trade-offs**: Increased write overhead (2-3%); additional storage for indexes - [ ] **PERF-PLAN-1.5 [Memory Leak Fix]**: - **Bottleneck**: Memory leak in WebSocket connection handler causing unbounded heap growth - **Technique**: Implement proper connection lifecycle management, remove event listeners on disconnect, add connection pooling with TTL - **Expected Impact**: 40% reduction in peak memory usage; elimination of OOM crashes - **Trade-offs**: Requires careful state management; potential for dropped connections if TTL too aggressive --- ## Performance Items ### Database Optimization Items - [ ] **PERF-ITEM-1.1 [Batch User Profile Queries]**: - **Before**: 3N+1 database queries per request (N = number of users) - **After**: 3 queries total using `IN` clauses and batch loading - **Implementation**: ```typescript // Before: N+1 pattern const users = await User.find({ organizationId }); for (const user of users) { user.posts = await Post.find({ userId: user.id }); user.comments = await Comment.find({ userId: user.id }); } // After: Batch loading const users = await User.find({ organizationId }); const userIds = users.map(u => u.id); const [posts, comments] = await Promise.all([ Post.find({ userId: { $in: userIds } }), Comment.find({ userId: { $in: userIds } }) ]); const postsByUser = groupBy(posts, 'userId'); const commentsByUser = groupBy(comments, 'userId'); users.forEach(user => { user.posts = postsByUser.get(user.id) || []; user.comments = commentsByUser.get(user.id) || []; }); ``` - **Validation**: Run `EXPLAIN ANALYZE` to confirm query count reduction; benchmark with k6 load test - [ ] **PERF-ITEM-1.2 [Add Composite Database Indexes]**: - **Before**: Full table scan on `orders` table for status+date queries (avg 850ms) - **After**: Indexed lookup (avg 45ms) - **Implementation**: ```sql -- Add composite indexes CREATE INDEX idx_orders_status_created ON orders (status, created_at); CREATE INDEX idx_orders_user_created ON orders (user_id, created_at); -- Partial index for active orders CREATE INDEX idx_orders_active ON orders (created_at) WHERE status = 'active'; ``` - **Validation**: Run `EXPLAIN ANALYZE` to verify index usage; monitor query performance in staging - [ ] **PERF-ITEM-1.3 [Implement Query Result Caching]**: - **Before**: Repeated identical queries hitting database (cache hit rate: 0%) - **After**: Redis cache with 60s TTL for hot queries (cache hit rate: 85%) - **Implementation**: ```typescript // Cache-aside pattern async function getDashboardData(userId: string) { const cacheKey = `dashboard:${userId}`; const cached = await redis.get(cacheKey); if (cached) return JSON.parse(cached); const data = await db.query('SELECT ... FROM dashboard WHERE user_id = $1', [userId]); await redis.setex(cacheKey, 60, JSON.stringify(data)); return data; } ``` - **Validation**: Monitor cache hit rate in Redis; compare response times with/without cache ### Frontend Optimization Items - [ ] **PERF-ITEM-2.1 [Implement Virtual Scrolling]**: - **Before**: 10,000 DOM nodes rendered for large lists (render time: 2.4s) - **After**: 50 DOM nodes with virtualization (render time: 300ms) - **Implementation**: ```tsx // Before: Full list rendering {items.map(item => <ListItem key={item.id} item={item} />)} // After: Virtual scrolling with react-window import { FixedSizeList } from 'react-window'; const ListComponent = ({ items }) => ( <FixedSizeList height={600} itemCount={items.length} itemSize={50} width="100%" > {({ index, style }) => ( <div style={style}> <ListItem item={items[index]} /> </div> )} </FixedSizeList> ); ``` - **Validation**: Use Chrome DevTools Performance tab to measure render time; Lighthouse audit - [ ] **PERF-ITEM-2.2 [Optimize React Re-renders]**: - **Before**: 45% of components re-rendering unnecessarily on state updates - **After**: 10% re-render rate with memoization - **Implementation**: ```tsx // Before: Unoptimized component const UserCard = ({ user, onSelect }) => ( <div onClick={() => onSelect(user.id)}> <h3>{user.name}</h3> <p>{user.email}</p> </div> ); // After: Memoized with stable callbacks const UserCard = React.memo(({ user, onSelect }) => ( <div onClick={() => onSelect(user.id)}> <h3>{user.name}</h3> <p>{user.email}</p> </div> )); // Parent component const UserList = ({ users }) => { const handleSelect = useCallback((id) => { console.log('Selected:', id); }, []); return users.map(user => ( <UserCard key={user.id} user={user} onSelect={handleSelect} /> )); }; ``` - **Validation**: Use React DevTools Profiler to measure render times; verify component memoization - [ ] **PERF-ITEM-2.3 [Implement Code Splitting]**: - **Before**: Single bundle of 2.4MB (parse time: 1.2s) - **After**: Route-based chunks averaging 400KB (parse time: 200ms) - **Implementation**: ```typescript // Before: Static import import AdminDashboard from './AdminDashboard'; // After: Dynamic import with React.lazy const AdminDashboard = React.lazy(() => import('./AdminDashboard')); // Route configuration <Route path="/admin" element={ <React.Suspense fallback={<Loading />}> <AdminDashboard /> </React.Suspense> } /> ``` - **Validation**: Use webpack-bundle-analyzer to verify chunk sizes; measure initial load time ### Async and Concurrency Items - [ ] **PERF-ITEM-3.1 [Move JSON Serialization to Worker Threads]**: - **Before**: Event loop blocked for 180ms during large payload serialization - **After**: Worker thread handles serialization (event loop blocked for 5ms) - **Implementation**: ```typescript // Before: Blocking serialization const json = JSON.stringify(largePayload); // After: Worker thread serialization const { Worker } = require('worker_threads'); function serializeAsync(data) { return new Promise((resolve, reject) => { const worker = new Worker(` const { parentPort, workerData } = require('worker_threads'); const json = JSON.stringify(workerData); parentPort.postMessage(json); `, { eval: true, workerData: data }); worker.once('message', resolve); worker.once('error', reject); }); } const json = await serializeAsync(largePayload); ``` - **Validation**: Monitor event loop lag with `process.hrtime()`; compare throughput under load - [ ] **PERF-ITEM-3.2 [Optimize Promise.all Usage]**: - **Before**: Sequential awaits for independent operations (total: 900ms) - **After**: Parallel execution with Promise.all (total: 300ms) - **Implementation**: ```typescript // Before: Sequential execution const user = await getUser(userId); const posts = await getPosts(userId); const comments = await getComments(userId); // After: Parallel execution const [user, posts, comments] = await Promise.all([ getUser(userId), getPosts(userId), getComments(userId) ]); ``` - **Validation**: Measure response time improvement; verify no race conditions - [ ] **PERF-ITEM-3.3 [Implement Backpressure Handling]**: - **Before**: Unbounded queue causing memory spikes under load (heap: 2.4GB) - **After**: Bounded queue with backpressure (heap: 1.2GB) - **Implementation**: ```typescript // Before: Unbounded queue const queue = []; queue.push(item); // No limit // After: Bounded queue with backpressure const { Queue } = require('bullmq'); const queue = new Queue('processing', { limiter: { max: 100, duration: 1000 }, defaultJobOptions: { attempts: 3, backoff: { type: 'exponential', delay: 1000 } } }); // Producer with backpressure async function addToQueue(item) { const count = await queue.count(); if (count > 1000) { await new Promise(resolve => setTimeout(resolve, 100)); return addToQueue(item); } await queue.add('process', item); } ``` - **Validation**: Monitor queue length and memory usage under load; verify no dropped jobs ### Memory Management Items - [ ] **PERF-ITEM-4.1 [Fix WebSocket Memory Leak]**: - **Before**: Heap grows 50MB/hour under sustained WebSocket connections - **After**: Stable heap at 800MB with proper cleanup - **Implementation**: ```typescript // Before: Missing cleanup ws.on('message', handleMessage); // No cleanup on close // After: Proper lifecycle management const connections = new Map(); ws.on('message', (data) => { const userId = parseUserId(data); connections.set(userId, ws); }); ws.on('close', () => { // Remove all references for (const [userId, conn] of connections) { if (conn === ws) { connections.delete(userId); } } // Remove event listeners ws.removeAllListeners(); }); // Periodic cleanup for stale connections setInterval(() => { const now = Date.now(); for (const [userId, conn] of connections) { if (now - conn.lastActivity > 30000) { conn.terminate(); connections.delete(userId); } } }, 30000); ``` - **Validation**: Monitor heap usage over 24-hour period; verify no memory growth - [ ] **PERF-ITEM-4.2 [Implement Object Pooling]**: - **Before**: 10,000 buffer allocations per second causing GC pressure - **After**: Object pool with 1,000 reusable buffers (GC pause: 50ms → 10ms) - **Implementation**: ```typescript // Before: New allocation each time const buffer = Buffer.alloc(1024); // After: Object pool class BufferPool { private pool: Buffer[] = []; private maxSize: number; constructor(maxSize = 1000) { this.maxSize = maxSize; } acquire(): Buffer { return this.pool.pop() || Buffer.alloc(1024); } release(buffer: Buffer): void { if (this.pool.length < this.maxSize) { buffer.fill(0); this.pool.push(buffer); } } } const pool = new BufferPool(); const buffer = pool.acquire(); // ... use buffer pool.release(buffer); ``` - **Validation**: Monitor GC pause times; measure allocation rates with heap profiler --- ## Proposed Code Changes ### Database Layer Changes **File: `src/db/queries/userQueries.ts`** ```typescript // Patch: Add batch loading for user profiles - export async function getUserProfiles(orgId: string) { - const users = await User.find({ organizationId: orgId }); - const profiles = []; - for (const user of users) { - const [posts, comments] = await Promise.all([ - Post.find({ userId: user.id }), - Comment.find({ userId: user.id }) - ]); - profiles.push({ ...user, posts, comments }); - } - return profiles; - } + export async function getUserProfiles(orgId: string) { + const users = await User.find({ organizationId: orgId }); + const userIds = users.map(u => u.id); + + const [posts, comments] = await Promise.all([ + Post.find({ userId: { $in: userIds } }), + Comment.find({ userId: { $in: userIds } }) + ]); + + const postsByUser = groupBy(posts, 'userId'); + const commentsByUser = groupBy(comments, 'userId'); + + return users.map(user => ({ + ...user, + posts: postsByUser.get(user.id) || [], + comments: commentsByUser.get(user.id) || [] + })); + } ``` **File: `src/db/migrations/2024_add_composite_indexes.sql`** ```sql -- Add composite indexes for common query patterns CREATE INDEX idx_orders_status_created ON orders (status, created_at); CREATE INDEX idx_orders_user_created ON orders (user_id, created_at); CREATE INDEX idx_users_org_created ON users (organization_id, created_at); -- Partial index for active orders CREATE INDEX idx_orders_active ON orders (created_at) WHERE status = 'active'; -- Covering index for dashboard queries CREATE INDEX idx_dashboard_user_metrics ON user_metrics (user_id, metric_type, created_at) INCLUDE (metric_value); ``` ### Frontend Changes **File: `src/components/UserList.tsx`** ```tsx // Patch: Add virtualization and memoization - import { UserCard } from './UserCard'; + import { FixedSizeList } from 'react-window'; + import { UserCard } from './UserCard'; + import React from 'react'; - export const UserList = ({ users }) => ( - <div> - {users.map(user => ( - <UserCard key={user.id} user={user} /> - ))} - </div> - ); + export const UserList = React.memo(({ users }) => { + const handleSelect = React.useCallback((id: string) => { + console.log('Selected user:', id); + }, []); + + return ( + <FixedSizeList + height={600} + itemCount={users.length} + itemSize={80} + width="100%" + > + {({ index, style }) => ( + <div style={style}> + <UserCard + user={users[index]} + onSelect={handleSelect} + /> + </div> + )} + </FixedSizeList> + ); + }); ``` **File: `src/App.tsx`** ```tsx // Patch: Add code splitting - import AdminDashboard from './pages/AdminDashboard'; - import UserDashboard from './pages/UserDashboard'; + import React from 'react'; + + const AdminDashboard = React.lazy(() => import('./pages/AdminDashboard')); + const UserDashboard = React.lazy(() => import('./pages/UserDashboard')); // Wrap routes with Suspense + <React.Suspense fallback={<LoadingSpinner />}> <Route path="/admin" element={<AdminDashboard />} /> <Route path="/user" element={<UserDashboard />} /> + </React.Suspense> ``` ### Backend Changes **File: `src/services/serializationService.ts`** ```typescript // Patch: Add worker thread serialization + import { Worker } from 'worker_threads'; + + export async function serializeAsync(data: unknown): Promise<string> { + return new Promise((resolve, reject) => { + const worker = new Worker(` + const { parentPort, workerData } = require('worker_threads'); + const json = JSON.stringify(workerData); + parentPort.postMessage(json); + `, { eval: true, workerData: data }); + + worker.once('message', resolve); + worker.once('error', reject); + }); + } + + export async function deserializeAsync<T>(json: string): Promise<T> { + return new Promise((resolve, reject) => { + const worker = new Worker(` + const { parentPort, workerData } = require('worker_threads'); + const data = JSON.parse(workerData); + parentPort.postMessage(data); + `, { eval: true, workerData: json }); + + worker.once('message', resolve); + worker.once('error', reject); + }); + } ``` **File: `src/websocket/connectionManager.ts`** ```typescript // Patch: Fix memory leak in WebSocket handler + const connections = new Map<string, WebSocket>(); + + export function handleConnection(ws: WebSocket) { + ws.on('message', (data) => { + const userId = parseUserId(data); + connections.set(userId, ws); + ws.lastActivity = Date.now(); + }); + + ws.on('close', () => { + for (const [userId, conn] of connections) { + if (conn === ws) { + connections.delete(userId); + } + } + ws.removeAllListeners(); + }); + } + + // Periodic cleanup + setInterval(() => { + const now = Date.now(); + for (const [userId, conn] of connections) { + if (now - conn.lastActivity > 30000) { + conn.terminate(); + connections.delete(userId); + } + } + }, 30000); ``` --- ## Commands ### Local Development Commands ```bash # Run performance profiling node --prof src/server.js node --prof-process isolate-*.log > profile.txt # Run database query analysis psql -c "EXPLAIN ANALYZE SELECT * FROM orders WHERE status = 'active' AND created_at > NOW() - INTERVAL '7 days';" # Run frontend performance audit npx lighthouse http://localhost:3000 --preset=perf --output=json --output-path=./lighthouse-report.json # Run load testing k6 run load-test.js # Run memory profiling node --inspect src/server.js # Open Chrome DevTools and use Memory tab for heap snapshots # Run bundle analysis npx webpack-bundle-analyzer dist/bundle.js ``` ### CI/CD Commands ```bash # Performance regression tests npm run perf:test # Database migration npm run migrate:up # Frontend build with bundle analysis npm run build:analyze # Run all tests npm run test:all ``` ### Monitoring Commands ```bash # Monitor Redis cache hit rate redis-cli INFO stats | grep hits # Monitor PostgreSQL slow queries psql -c "SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;" # Monitor Node.js event loop lag node -e "setInterval(() => { const start = process.hrtime(); setTimeout(() => { const diff = process.hrtime(start); console.log('Event loop lag:', diff[1] / 1e6, 'ms'); }, 0); }, 1000);" # Monitor WebSocket connections ss -s | grep websocket ``` --- ## Quality Assurance Task Checklist - [ ] Baseline metrics are captured with reproducible benchmark conditions (hardware: 8-core CPU, 16GB RAM; data volume: 1M records; concurrency: 100 users) - [ ] All optimizations are ranked by impact and address the highest-priority bottlenecks (N+1 queries first, then rendering, then serialization) - [ ] Before/after measurements demonstrate quantifiable improvement (target: 60% reduction in p95, 45% improvement in throughput) - [ ] No functional regressions introduced by optimizations (run full test suite after each change) - [ ] Trade-offs between performance, readability, and maintainability are documented (see Trade-offs section in each item) - [ ] Monitoring thresholds and alerting strategies are defined for ongoing tracking (alert on: p95 > 300ms, heap > 1.5GB, GC pause > 50ms) - [ ] Performance regression tests are specified for CI/CD integration (run k6 load tests with 1000 RPS threshold) ## Execution Reminders - **Measurement-First**: All optimizations must be preceded by profiling to confirm the bottleneck - **Impact Prioritization**: Address N+1 queries first (highest impact), then frontend rendering, then serialization - **Quantifiable Evidence**: Every optimization must include before/after metrics - **Maintainability**: Balance performance gains with code readability; document complex optimizations - **Full-System Impact**: Consider how each optimization affects other components (e.g., caching affects data consistency) - **Continuous Monitoring**: Set up alerts for performance regressions before deploying optimizations --- ## Performance Regression Test Configuration **File: `perf/load-test.js`** ```javascript import http from 'k6/http'; import { check, sleep } from 'k6'; export const options = { stages: [ { duration: '30s', target: 500 }, // Ramp up { duration: '1m', target: 1000 }, // Steady state { duration: '30s', target: 0 }, // Ramp down ], thresholds: { http_req_duration: ['p(95)<300', 'p(99)<500'], http_req_failed: ['rate<0.01'], http_reqs: ['rate>1000'], }, }; export default function () { const res = http.get('http://localhost:3000/api/users'); check(res, { 'status is 200': (r) => r.status === 200, 'response time < 300ms': (r) => r.timings.duration < 300, }); sleep(1); } ``` **File: `perf/memory-test.js`** ```javascript // Memory leak detection test const v8 = require('v8'); const heap = require('heap'); const initialHeap = v8.getHeapStatistics().heap_size_limit; let samples = []; setInterval(() => { const stats = v8.getHeapStatistics(); samples.push(stats.used_heap_size); if (samples.length > 10) { const growth = samples[samples.length - 1] - samples[0]; if (growth > 50 * 1024 * 1024) { // 50MB growth console.error('Memory leak detected:', growth, 'bytes'); process.exit(1); } samples.shift(); } }, 10000); ``` --- ## Monitoring and Alerting Configuration **File: `monitoring/alerts.yml`** ```yaml alerts: - name: high_response_time condition: p95_response_time > 300ms duration: 5m severity: critical action: page_on_call - name: memory_leak condition: heap_size_growth > 50MB_per_hour duration: 1h severity: critical action: page_on_call - name: database_slow_queries condition: avg_query_time > 50ms duration: 10m severity: warning action: email_team - name: cache_hit_rate_low condition: redis_cache_hit_rate < 70% duration: 15m severity: warning action: email_team - name: event_loop_blocked condition: event_loop_lag > 100ms duration: 5m severity: critical action: page_on_call ``` --- ## Lessons Learned and Patterns ### Optimization Patterns 1. **Batch Loading**: Always use `IN` clauses or DataLoader patterns instead of N+1 queries 2. **Virtualization**: Use `react-window` or `react-virtualized` for large lists (>1000 items) 3. **Worker Threads**: Move CPU-intensive operations (serialization, image processing) to worker threads 4. **Composite Indexes**: Create indexes based on actual query patterns, not just column popularity 5. **Connection Lifecycle**: Always implement proper cleanup for WebSocket and database connections ### Anti-Patterns to Avoid 1. **Premature Optimization**: Profile first, optimize second 2. **Over-Caching**: Always have invalidation strategy before adding cache 3. **Unbounded Queues**: Always implement backpressure for message queues 4. **Synchronous Blocking**: Never block the event loop with sync operations 5. **Missing Indexes**: Always analyze query patterns before deploying to production ### Future Considerations 1. **Horizontal Scaling**: Consider read replicas for database when read/write ratio exceeds 10:1 2. **CDN Integration**: Move static assets to CDN when frontend bundle exceeds 500KB 3. **gRPC Migration**: Consider gRPC for inter-service communication when REST overhead becomes significant 4. **Edge Computing**: Move computation closer to users for latency-sensitive operations 5. **Auto-scaling**: Implement auto-scaling based on CPU utilization (>70%) and memory usage (>80%)