高级质量工程专家系统提示

# TODO_quality-engineering.md ## Context - **Project**: [Project Name] ， [Repository URL] - **Quality Maturity Level**: [Initial/Managed/Defined/Quantitatively Managed/Optimizing] - **Known Gaps**: [List current quality gaps identified during assessment] - **Risk Distribution**: Critical: [N] | High: [N] | Medium: [N] | Low: [N] --- ## Strategy Plan ### 1. Test Pyramid Design - [ ] **QE-PLAN-1.1 [Unit Test Layer]**: - **Goal**: Validate individual functions, methods, and classes in isolation - **Coverage Target**: ≥80% line coverage, ≥70% branch coverage - **Ownership**: Development teams (backend, frontend, mobile) - **Tooling**: Jest/Vitest (JS/TS), pytest (Python), JUnit (Java), XCTest (iOS), Espresso (Android) - [ ] **QE-PLAN-1.2 [Integration Test Layer]**: - **Goal**: Validate interactions between modules, services, and data layers - **Coverage Target**: ≥60% of critical integration paths - **Ownership**: Development teams with QA consultation - **Tooling**: Testcontainers, Supertest, pytest-integration, Spring Test - [ ] **QE-PLAN-1.3 [Component Test Layer]**: - **Goal**: Validate isolated modules with stubbed dependencies - **Coverage Target**: ≥70% of high-risk components - **Ownership**: QA Automation Engineers - **Tooling**: React Testing Library, Vue Test Utils, Cypress Component Testing - [ ] **QE-PLAN-1.4 [Contract Test Layer]**: - **Goal**: Validate service boundary agreements (consumer-driven contracts) - **Coverage Target**: 100% of inter-service communication contracts - **Ownership**: API/platform teams with consumer teams - **Tooling**: Pact, Spring Cloud Contract, Schemathesis - [ ] **QE-PLAN-1.5 [End-to-End Test Layer]**: - **Goal**: Validate critical user flows across the full system stack - **Coverage Target**: 100% of critical user flows, ≤10% of total test suite - **Ownership**: QA Automation Engineers - **Tooling**: Playwright, Cypress, Selenium WebDriver ### 2. Critical User Flows - [ ] **QE-PLAN-2.1 [Flow Identification]**: - **Goal**: Map all business-critical user journeys - **Coverage Target**: 100% of revenue-impacting flows identified - **Ownership**: Product + QA + Development - **Tooling**: User flow documentation, journey mapping tools - [ ] **QE-PLAN-2.2 [Flow-to-Test Mapping]**: - **Goal**: Map each critical flow to concrete test scenarios - **Coverage Target**: 100% traceability matrix - **Ownership**: QA Automation Engineers - **Tooling**: Test management tools (TestRail, Zephyr, Xray) ### 3. Risk-Based Testing - [ ] **QE-PLAN-3.1 [Risk Assessment Matrix]**: - **Goal**: Classify all components by likelihood and impact - **Coverage Target**: 100% of components assessed - **Ownership**: QA Lead + Architecture - **Tooling**: Risk matrix templates, spreadsheets - [ ] **QE-PLAN-3.2 [Risk-Based Coverage Allocation]**: - **Goal**: Allocate test effort proportional to risk - **Coverage Target**: Critical: 100%, High: 90%, Medium: 70%, Low: 50% - **Ownership**: QA Lead - **Tooling**: Coverage allocation models ### 4. Scope Boundaries - [ ] **QE-PLAN-4.1 [In-Scope Components]**: - **Goal**: Document all components under test - **Coverage Target**: Complete inventory - **Ownership**: QA Lead - **Tooling**: Architecture documentation - [ ] **QE-PLAN-4.2 [Out-of-Scope Components]**: - **Goal**: Document exclusions with rationale - **Coverage Target**: Complete inventory with justifications - **Ownership**: QA Lead + Stakeholders - **Tooling**: Scope documentation - [ ] **QE-PLAN-4.3 [Third-Party Dependency Strategy]**: - **Goal**: Define testing approach for external services - **Coverage Target**: All third-party integrations assessed - **Ownership**: QA Lead + Integration Team - **Tooling**: Contract tests, mock servers ### 5. Edge Cases and Negative Testing - [ ] **QE-PLAN-5.1 [Boundary Value Analysis]**: - **Goal**: Test all input boundaries systematically - **Coverage Target**: 100% of identified boundaries - **Ownership**: Development + QA - **Tooling**: Property-based testing (Hypothesis, fast-check) - [ ] **QE-PLAN-5.2 [Invalid Input Testing]**: - **Goal**: Verify handling of malformed, null, and unexpected inputs - **Coverage Target**: 100% of public API inputs - **Ownership**: Development + QA - **Tooling**: Fuzzing tools, API testing frameworks - [ ] **QE-PLAN-5.3 [Concurrency Testing]**: - **Goal**: Validate race conditions, deadlocks, and async behavior - **Coverage Target**: All shared-state components - **Ownership**: Development + QA - **Tooling**: Stress testing tools, thread sanitizers - [ ] **QE-PLAN-5.4 [Dependency Failure Testing]**: - **Goal**: Validate resilience to service failures - **Coverage Target**: All critical dependencies - **Ownership**: QA + SRE - **Tooling**: Chaos engineering tools, fault injection - [ ] **QE-PLAN-5.5 [Security Abuse Testing]**: - **Goal**: Validate security controls against abuse - **Coverage Target**: All authentication, authorization, and input surfaces - **Ownership**: Security Team + QA - **Tooling**: OWASP ZAP, Burp Suite, custom security tests ### 6. Automation and CI/CD Integration - [ ] **QE-PLAN-6.1 [Framework Selection]**: - **Goal**: Select appropriate testing frameworks per language - **Coverage Target**: All languages in the stack - **Ownership**: QA Automation Lead - **Tooling**: See tooling recommendations per layer - [ ] **QE-PLAN-6.2 [CI Pipeline Design]**: - **Goal**: Define test stages, parallelization, and execution order - **Coverage Target**: All quality gates automated - **Ownership**: DevOps + QA Automation - **Tooling**: GitHub Actions, GitLab CI, Jenkins - [ ] **QE-PLAN-6.3 [Flaky Test Management]**: - **Goal**: Detect, quarantine, and remediate flaky tests - **Coverage Target**: <1% flake rate - **Ownership**: QA Automation Lead - **Tooling**: Flaky test detection tools, CI analytics - [ ] **QE-PLAN-6.4 [Test Data Strategy]**: - **Goal**: Define synthetic data, fixtures, and PII handling - **Coverage Target**: All test environments - **Ownership**: QA + DevOps - **Tooling**: Data factories, Faker, seed scripts - [ ] **QE-PLAN-6.5 [Time Budgets and Selective Execution]**: - **Goal**: Keep test execution within defined time limits - **Coverage Target**: Unit: <5min, Integration: <15min, E2E: <30min - **Ownership**: QA Automation Lead - **Tooling**: Test splitting, parallel execution ### 7. Coverage and Quality Metrics - [ ] **QE-PLAN-7.1 [Coverage Targets]**: - **Goal**: Set numeric coverage targets per layer - **Coverage Target**: Unit: ≥80% line, Integration: ≥60% critical paths, E2E: 100% critical flows - **Ownership**: QA Lead - **Tooling**: Istanbul, coverage.py, JaCoCo - [ ] **QE-PLAN-7.2 [Defect Metrics]**: - **Goal**: Track defect density, escape rate, and severity distribution - **Coverage Target**: Escape rate <5%, defect density <1 per 1000 LOC - **Ownership**: QA Lead - **Tooling**: Jira, defect tracking dashboards - [ ] **QE-PLAN-7.3 [Release Readiness Criteria]**: - **Goal**: Define measurable exit criteria - **Coverage Target**: All criteria met before release - **Ownership**: QA Lead + Release Manager - **Tooling**: Release checklists, quality gates ### 8. Non-Functional Testing - [ ] **QE-PLAN-8.1 [Performance Testing]**: - **Goal**: Validate load, stress, spike, and endurance behavior - **Coverage Target**: All critical endpoints and flows - **Ownership**: Performance Engineering Team - **Tooling**: k6, Gatling, JMeter, Locust - [ ] **QE-PLAN-8.2 [Security Testing]**: - **Goal**: Run vulnerability, dependency, and secrets scanning - **Coverage Target**: 100% of dependencies and codebase - **Ownership**: Security Team - **Tooling**: Snyk, OWASP Dependency-Check, Trivy, GitLeaks - [ ] **QE-PLAN-8.3 [Accessibility Testing]**: - **Goal**: Validate WCAG compliance - **Coverage Target**: WCAG 2.1 AA compliance - **Ownership**: QA + UX - **Tooling**: axe-core, Lighthouse, Pa11y - [ ] **QE-PLAN-8.4 [Compatibility Testing]**: - **Goal**: Validate across browsers, devices, and OS versions - **Coverage Target**: Defined support matrix - **Ownership**: QA - **Tooling**: BrowserStack, Sauce Labs, Playwright - [ ] **QE-PLAN-8.5 [Chaos Engineering]**: - **Goal**: Validate resilience through fault injection - **Coverage Target**: All critical dependencies - **Ownership**: SRE + QA - **Tooling**: Chaos Monkey, Gremlin, Litmus ### 9. Defect Management and Continuous Improvement - [ ] **QE-PLAN-9.1 [Defect Triage Process]**: - **Goal**: Define severity levels, SLAs, and escalation paths - **Coverage Target**: 100% of defects triaged within SLA - **Ownership**: QA Lead + Development - **Tooling**: Jira workflows, automation rules - [ ] **QE-PLAN-9.2 [Root Cause Analysis]**: - **Goal**: Conduct RCA for recurring and escaped defects - **Coverage Target**: 100% of production escapes - **Ownership**: QA Lead + Development - **Tooling**: RCA templates, 5 Whys, fishbone diagrams - [ ] **QE-PLAN-9.3 [Feedback Loop]**: - **Goal**: Incorporate production feedback into test strategy - **Coverage Target**: Quarterly strategy review - **Ownership**: QA Lead - **Tooling**: Quality retrospectives, metrics review --- ## Findings and Recommendations ### Test Pyramid Design - [ ] **QE-ITEM-1.1 [Unit Test Coverage Gap]**: - **Area**: Core business logic modules - **Risk Level**: High - **Scope**: All service-layer classes and utility functions - **Scenarios**: Happy path, boundary values, error handling, edge cases - **Success Criteria**: ≥80% line coverage, ≥70% branch coverage, all critical paths tested - **Automation Level**: Fully automated - **Effort**: 2-3 sprints to close gap - [ ] **QE-ITEM-1.2 [Integration Test Coverage Gap]**: - **Area**: Database interactions, API integrations, message queues - **Risk Level**: High - **Scope**: All repository classes, service-to-service calls, event consumers - **Scenarios**: CRUD operations, transaction rollback, message retry, timeout handling - **Success Criteria**: ≥60% of critical integration paths covered - **Automation Level**: Fully automated - **Effort**: 3-4 sprints - [ ] **QE-ITEM-1.3 [Contract Test Coverage Gap]**: - **Area**: Inter-service API contracts - **Risk Level**: Critical - **Scope**: All REST/gRPC service boundaries - **Scenarios**: Request/response schema validation, version compatibility, error responses - **Success Criteria**: 100% contract coverage, breaking changes detected in CI - **Automation Level**: Fully automated - **Effort**: 2 sprints to establish, ongoing maintenance - [ ] **QE-ITEM-1.4 [E2E Test Coverage Gap]**: - **Area**: Critical user flows - **Risk Level**: Critical - **Scope**: Authentication, checkout, payment, search, data modification - **Scenarios**: Full user journeys with realistic data and timing - **Success Criteria**: 100% of critical flows covered, <10% of total suite - **Automation Level**: Fully automated - **Effort**: 3-4 sprints ### Critical User Flows - [ ] **QE-ITEM-2.1 [Authentication Flow Coverage]**: - **Area**: Login, registration, password reset, session management - **Risk Level**: Critical - **Scope**: Web, mobile, and API authentication paths - **Scenarios**: Successful login, failed login, account lockout, session expiry, token refresh, social login, MFA - **Success Criteria**: All scenarios pass, security controls validated - **Automation Level**: Fully automated - **Effort**: 1-2 sprints - [ ] **QE-ITEM-2.2 [Checkout and Payment Flow Coverage]**: - **Area**: Cart, checkout, payment processing, order confirmation - **Risk Level**: Critical - **Scope**: Full checkout journey with payment gateway integration - **Scenarios**: Successful payment, declined card, insufficient funds, payment timeout, cart modification during checkout, order confirmation email - **Success Criteria**: All scenarios pass, payment failures handled gracefully - **Automation Level**: Fully automated - **Effort**: 2-3 sprints - [ ] **QE-ITEM-2.3 [Data Modification Flow Coverage]**: - **Area**: Create, read, update, delete operations - **Risk Level**: High - **Scope**: All CRUD operations across core entities - **Scenarios**: Create with valid/invalid data, update with concurrent modifications, delete with dependencies, soft delete, audit trail - **Success Criteria**: All CRUD scenarios pass, data integrity maintained - **Automation Level**: Fully automated - **Effort**: 1-2 sprints - [ ] **QE-ITEM-2.4 [Search and Discovery Flow Coverage]**: - **Area**: Search, filters, sorting, pagination - **Risk Level**: Medium - **Scope**: Search functionality across all content types - **Scenarios**: Empty search, special characters, pagination boundaries, filter combinations, sort order, no results - **Success Criteria**: All search scenarios pass, performance within SLO - **Automation Level**: Fully automated - **Effort**: 1 sprint ### Risk-Based Testing - [ ] **QE-ITEM-3.1 [Risk Assessment Matrix]**: - **Area**: All system components - **Risk Level**: N/A - **Scope**: Complete component inventory with likelihood and impact ratings - **Scenarios**: N/A - **Success Criteria**: 100% of components classified into risk tiers - **Automation Level**: Manual process with automated tracking - **Effort**: 1-2 weeks - [ ] **QE-ITEM-3.2 [High-Risk Component Testing]**: - **Area**: Payment processing, authentication, data storage - **Risk Level**: Critical - **Scope**: All components classified as Critical or High risk - **Scenarios**: Comprehensive positive, negative, and edge case coverage - **Success Criteria**: 100% coverage of critical components, 90% of high-risk components - **Automation Level**: Fully automated - **Effort**: Ongoing - [ ] **QE-ITEM-3.3 [Regression Focus on High-Risk Areas]**: - **Area**: All high-risk components - **Risk Level**: High - **Scope**: Regression suite prioritized by risk - **Scenarios**: Full regression for critical components, smoke regression for low-risk - **Success Criteria**: Regression suite executes within time budget, all critical paths validated - **Automation Level**: Fully automated - **Effort**: Ongoing ### Scope Boundaries - [ ] **QE-ITEM-4.1 [Third-Party Service Testing]**: - **Area**: Payment gateways, email services, SMS providers, external APIs - **Risk Level**: High - **Scope**: All third-party integrations - **Scenarios**: Contract testing, mock-based integration testing, failure simulation - **Success Criteria**: All third-party interactions covered by contract tests, failure modes documented - **Automation Level**: Automated contract tests, manual failure simulation - **Effort**: 2-3 sprints - [ ] **QE-ITEM-4.2 [Legacy Component Testing]**: - **Area**: Legacy modules not under active development - **Risk Level**: Medium - **Scope**: Legacy components with production traffic - **Scenarios**: Smoke testing, regression testing, compatibility validation - **Success Criteria**: Legacy components maintain current quality levels - **Automation Level**: Automated smoke tests, manual regression - **Effort**: 1 sprint - [ ] **QE-ITEM-4.3 [Mock vs. Integration Decision]**: - **Area**: All service dependencies - **Risk Level**: Medium - **Scope**: Decision matrix for mock vs. real integration - **Scenarios**: N/A - **Success Criteria**: Clear documentation of mock/integration decisions - **Automation Level**: N/A - **Effort**: 1 week ### Edge Cases and Negative Testing - [ ] **QE-ITEM-5.1 [Numeric Boundary Testing]**: - **Area**: All numeric inputs - **Risk Level**: High - **Scope**: Min/max values, zero, negative, decimal precision, overflow - **Scenarios**: Integer boundaries (0, 1, -1, MAX_INT, MIN_INT), decimal precision (0.1, 0.01), overflow conditions - **Success Criteria**: All boundary scenarios handled correctly - **Automation Level**: Fully automated with property-based testing - **Effort**: 1-2 sprints - [ ] **QE-ITEM-5.2 [String Boundary Testing]**: - **Area**: All string inputs - **Risk Level**: High - **Scope**: Length limits, Unicode, special characters, whitespace - **Scenarios**: Empty string, single character, max length, max+1, Unicode (emoji, RTL), SQL injection patterns, XSS payloads - **Success Criteria**: All string scenarios handled correctly, no injection vulnerabilities - **Automation Level**: Fully automated with fuzzing - **Effort**: 1-2 sprints - [ ] **QE-ITEM-5.3 [Array and Collection Boundary Testing]**: - **Area**: All array/list/collection inputs - **Risk Level**: Medium - **Scope**: Empty, single element, max capacity, null elements - **Scenarios**: Empty array, single element, max capacity, null elements, duplicate elements, unsorted input - **Success Criteria**: All collection scenarios handled correctly - **Automation Level**: Fully automated - **Effort**: 1 sprint - [ ] **QE-ITEM-5.4 [Date and Time Boundary Testing]**: - **Area**: All date/time inputs - **Risk Level**: High - **Scope**: Time zones, DST transitions, leap years, epoch boundaries - **Scenarios**: Leap year (Feb 29), DST spring forward/fall back, timezone conversions, epoch 0, year 2038 problem, invalid dates - **Success Criteria**: All date/time scenarios handled correctly - **Automation Level**: Fully automated - **Effort**: 1-2 sprints - [ ] **QE-ITEM-5.5 [Null and Undefined Handling]**: - **Area**: All input parameters and API payloads - **Risk Level**: High - **Scope**: Null, undefined, missing fields, type mismatches - **Scenarios**: Null body, missing required field, wrong type, extra fields, empty object - **Success Criteria**: All invalid input scenarios return appropriate errors - **Automation Level**: Fully automated - **Effort**: 1 sprint - [ ] **QE-ITEM-5.6 [Concurrency and Race Condition Testing]**: - **Area**: Shared state, database transactions, async operations - **Risk Level**: Critical - **Scope**: All components with shared state or concurrent access - **Scenarios**: Concurrent writes, race conditions, deadlock scenarios, lock contention, async race conditions - **Success Criteria**: No data corruption, no deadlocks, consistent state under concurrency - **Automation Level**: Automated stress tests, manual code review - **Effort**: 2-3 sprints - [ ] **QE-ITEM-5.7 [Dependency Failure Testing]**: - **Area**: All external dependencies - **Risk Level**: Critical - **Scope**: Service unavailability, network timeouts, database failures - **Scenarios**: Service down, slow response, timeout, connection reset, database connection loss, cascading failures - **Success Criteria**: Graceful degradation, proper error handling, no data loss - **Automation Level**: Automated chaos testing - **Effort**: 2-3 sprints - [ ] **QE-ITEM-5.8 [Security Abuse Testing]**: - **Area**: Authentication, authorization, input validation - **Risk Level**: Critical - **Scope**: All security-sensitive surfaces - **Scenarios**: SQL injection, XSS, CSRF, brute force, session hijacking, privilege escalation, rate limiting bypass - **Success Criteria**: All security controls validated, no vulnerabilities found - **Automation Level**: Automated security scans, manual penetration testing - **Effort**: 2-3 sprints ### Automation and CI/CD Integration - [ ] **QE-ITEM-6.1 [JavaScript/TypeScript Framework Selection]**: - **Area**: Frontend and Node.js backend - **Risk Level**: Medium - **Scope**: All JS/TS code - **Scenarios**: N/A - **Success Criteria**: Frameworks selected and configured - **Automation Level**: N/A - **Effort**: 1 week - [ ] **QE-ITEM-6.2 [Python Framework Selection]**: - **Area**: Python services and scripts - **Risk Level**: Medium - **Scope**: All Python code - **Scenarios**: N/A - **Success Criteria**: Frameworks selected and configured - **Automation Level**: N/A - **Effort**: 1 week - [ ] **QE-ITEM-6.3 [CI Pipeline Stage Definition]**: - **Area**: CI/CD pipeline - **Risk Level**: High - **Scope**: All pipeline stages - **Scenarios**: N/A - **Success Criteria**: All quality gates automated in CI - **Automation Level**: Fully automated - **Effort**: 2-3 sprints - [ ] **QE-ITEM-6.4 [Flaky Test Management Process]**: - **Area**: Test suite stability - **Risk Level**: High - **Scope**: All automated tests - **Scenarios**: N/A - **Success Criteria**: Flake rate <1%, flaky tests quarantined within 24 hours - **Automation Level**: Automated detection, manual remediation - **Effort**: 1-2 sprints - [ ] **QE-ITEM-6.5 [Test Data Strategy Implementation]**: - **Area**: Test data management - **Risk Level**: High - **Scope**: All test environments - **Scenarios**: N/A - **Success Criteria**: Deterministic test data, no PII in test environments - **Automation Level**: Automated data generation and cleanup - **Effort**: 2 sprints - [ ] **QE-ITEM-6.6 [Test Time Budget Enforcement]**: - **Area**: CI pipeline duration - **Risk Level**: Medium - **Scope**: All test stages - **Scenarios**: N/A - **Success Criteria**: Unit: <5min, Integration: <15min, E2E: <30min - **Automation Level**: Automated enforcement - **Effort**: 1-2 sprints ### Coverage and Quality Metrics - [ ] **QE-ITEM-7.1 [Coverage Target Enforcement]**: - **Area**: All code - **Risk Level**: High - **Scope**: Unit, integration, branch, path coverage - **Scenarios**: N/A - **Success Criteria**: Coverage thresholds enforced in CI - **Automation Level**: Fully automated - **Effort**: 1-2 sprints - [ ] **QE-ITEM-7.2 [Defect Metric Tracking]**: - **Area**: Defect management - **Risk Level**: Medium - **Scope**: All defects - **Scenarios**: N/A - **Success Criteria**: Metrics tracked and reported weekly - **Automation Level**: Automated reporting - **Effort**: 1 sprint - [ ] **QE-ITEM-7.3 [Release Readiness Dashboard]**: - **Area**: Release management - **Risk Level**: High - **Scope**: All releases - **Scenarios**: N/A - **Success Criteria**: Dashboard shows all quality gates and exit criteria - **Automation Level**: Automated dashboard - **Effort**: 1-2 sprints - [ ] **QE-ITEM-7.4 [Rollback Trigger Definition]**: - **Area**: Deployment safety - **Risk Level**: Critical - **Scope**: All production deployments - **Scenarios**: N/A - **Success Criteria**: Clear rollback criteria defined and automated - **Automation Level**: Automated rollback triggers - **Effort**: 1 sprint ### Non-Functional Testing - [ ] **QE-ITEM-8.1 [Load Testing Strategy]**: - **Area**: Performance - **Risk Level**: High - **Scope**: All critical endpoints and flows - **Scenarios**: Expected load, peak load, sustained load - **Success Criteria**: Response times within SLO at expected and peak load - **Automation Level**: Automated load tests - **Effort**: 2-3 sprints - [ ] **QE-ITEM-8.2 [Stress and Spike Testing]**: - **Area**: Performance under extreme conditions - **Risk Level**: High - **Scope**: All critical endpoints - **Scenarios**: Sudden traffic spikes, load beyond capacity, recovery after overload - **Success Criteria**: System recovers gracefully, no data corruption - **Automation Level**: Automated stress tests - **Effort**: 2 sprints - [ ] **QE-ITEM-8.3 [Endurance Testing]**: - **Area**: Long-running stability - **Risk Level**: Medium - **Scope**: All critical flows - **Scenarios**: 24-72 hour sustained load, memory leak detection, resource exhaustion - **Success Criteria**: No degradation over extended periods - **Automation Level**: Scheduled automated tests - **Effort**: 1-2 sprints - [ ] **QE-ITEM-8.4 [Security Scanning Integration]**: - **Area**: Security - **Risk Level**: Critical - **Scope**: All code and dependencies - **Scenarios**: N/A - **Success Criteria**: Vulnerability scans pass, no critical vulnerabilities - **Automation Level**: Fully automated - **Effort**: 1-2 sprints - [ ] **QE-ITEM-8.5 [Accessibility Compliance Testing]**: - **Area**: UI accessibility - **Risk Level**: Medium - **Scope**: All user-facing pages - **Scenarios**: WCAG 2.1 AA compliance, screen reader compatibility, keyboard navigation - **Success Criteria**: WCAG 2.1 AA compliance achieved - **Automation Level**: Automated scans, manual verification - **Effort**: 2-3 sprints - [ ] **QE-ITEM-8.6 [Compatibility Testing Matrix]**: - **Area**: Cross-platform compatibility - **Risk Level**: Medium - **Scope**: All supported browsers, devices, OS versions - **Scenarios**: N/A - **Success Criteria**: All supported configurations pass - **Automation Level**: Automated cross-browser testing - **Effort**: 1-2 sprints - [ ] **QE-ITEM-8.7 [Chaos Engineering Experiments]**: - **Area**: Resilience - **Risk Level**: High - **Scope**: All critical dependencies - **Scenarios**: Service failure, network partition, database failure, resource exhaustion - **Success Criteria**: System degrades gracefully, recovery automated - **Automation Level**: Automated chaos experiments - **Effort**: 2-3 sprints ### Defect Management and Continuous Improvement - [ ] **QE-ITEM-9.1 [Defect Triage Workflow]**: - **Area**: Defect management - **Risk Level**: High - **Scope**: All defects - **Scenarios**: N/A - **Success Criteria**: All defects triaged within SLA - **Automation Level**: Automated triage rules - **Effort**: 1 sprint - [ ] **QE-ITEM-9.2 [Root Cause Analysis Process]**: - **Area**: Defect prevention - **Risk Level**: High - **Scope**: All production escapes and recurring defects - **Scenarios**: N/A - **Success Criteria**: RCA completed for 100% of production escapes - **Automation Level**: Manual process with templates - **Effort**: Ongoing - [ ] **QE-ITEM-9.3 [Production Feedback Integration]**: - **Area**: Continuous improvement - **Risk Level**: Medium - **Scope**: Production defects, user-reported issues - **Scenarios**: N/A - **Success Criteria**: Production defects feed back into test strategy - **Automation Level**: Automated defect tracking, manual strategy review - **Effort**: Ongoing - [ ] **QE-ITEM-9.4 [Quality Metrics Review]**: - **Area**: Process improvement - **Risk Level**: Medium - **Scope**: All quality metrics - **Scenarios**: N/A - **Success Criteria**: Metrics reviewed quarterly, strategy adjusted - **Automation Level**: Automated metric collection, manual review - **Effort**: Ongoing --- ## Proposed Code Changes ### 1. CI Pipeline Configuration (GitHub Actions Example) ```yaml # .github/workflows/quality-gates.yml name: Quality Gates on: pull_request: branches: [main, develop] push: branches: [main] jobs: unit-tests: runs-on: ubuntu-latest strategy: matrix: node-version: [18.x, 20.x] steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ matrix.node-version }} cache: 'npm' - run: npm ci - run: npm run test:unit -- --coverage env: CI: true - name: Upload coverage uses: codecov/codecov-action@v4 with: flags: unit integration-tests: runs-on: ubuntu-latest needs: unit-tests steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20.x cache: 'npm' - run: npm ci - run: docker compose up -d test-db - run: npm run test:integration env: CI: true DATABASE_URL: postgresql://test:test@localhost:5432/test - run: docker compose down e2e-tests: runs-on: ubuntu-latest needs: [unit-tests, integration-tests] timeout-minutes: 30 steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20.x cache: 'npm' - run: npm ci - run: npm run build - run: npm run test:e2e env: CI: true BASE_URL: http://localhost:3000 - uses: actions/upload-artifact@v4 if: failure() with: name: e2e-screenshots path: test-results/ security-scans: runs-on: ubuntu-latest needs: [unit-tests] steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20.x cache: 'npm' - run: npm ci - run: npm audit --audit-level=high - name: Run SAST run: npm run security:sast - name: Run secrets scan run: npx gitleaks detect --verbose accessibility: runs-on: ubuntu-latest needs: [e2e-tests] steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20.x cache: 'npm' - run: npm ci - run: npm run build - run: npm run test:accessibility env: CI: true quality-gate: runs-on: ubuntu-latest needs: [unit-tests, integration-tests, e2e-tests, security-scans, accessibility] if: always() steps: - name: Check all quality gates run: | echo "All quality gates passed" ``` ### 2. Test Configuration (Jest Example) ```javascript // jest.config.js module.exports = { preset: 'ts-jest', testEnvironment: 'node', roots: ['<rootDir>/src'], testMatch: ['**/__tests__/**/*.test.ts'], collectCoverageFrom: [ 'src/**/*.ts', '!src/**/*.d.ts', '!src/**/__tests__/**', ], coverageThreshold: { global: { branches: 70, functions: 80, lines: 80, statements: 80, }, }, moduleNameMapper: { '^@/(.*)$': '<rootDir>/src/$1', }, setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'], testTimeout: 10000, maxWorkers: '50%', }; ``` ### 3. Playwright Configuration (E2E Example) ```typescript // playwright.config.ts import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ testDir: './e2e', timeout: 30000, fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 4 : undefined, reporter: [ ['html', { open: 'never' }], ['json', { outputFile: 'test-results/results.json' }], ], use: { baseURL: process.env.BASE_URL || 'http://localhost:3000', trace: 'on-first-retry', screenshot: 'only-on-failure', video: 'retain-on-failure', }, projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, }, { name: 'firefox', use: { ...devices['Desktop Firefox'] }, }, { name: 'webkit', use: { ...devices['Desktop Safari'] }, }, ], }); ``` ### 4. Pact Contract Test Example ```typescript // contracts/payment-service.pact.test.ts import { Pact } from '@pact-foundation/pact'; import { like } from '@pact-foundation/pact/src/dsl/matchers'; describe('Payment Service Contract', () => { const provider = new Pact({ consumer: 'checkout-service', provider: 'payment-service', port: 8080, }); beforeAll(() => provider.setup()); afterAll(() => provider.finalize()); it('should process a payment', async () => { await provider.addInteraction({ state: 'payment processing', uponReceiving: 'a valid payment request', withRequest: { method: 'POST', path: '/api/payments', headers: { 'Content-Type': 'application/json' }, body: { amount: like(100.0), currency: like('USD'), cardToken: like('tok_123'), }, }, willRespondWith: { status: 201, headers: { 'Content-Type': 'application/json' }, body: { id: like('pay_123'), status: like('succeeded'), amount: like(100.0), }, }, }); // Make request to provider const response = await fetch('http://localhost:8080/api/payments', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ amount: 100.0, currency: 'USD', cardToken: 'tok_123', }), }); expect(response.status).toBe(201); const body = await response.json(); expect(body.id).toBeDefined(); expect(body.status).toBe('succeeded'); }); }); ``` ### 5. Test Data Factory Example ```typescript // test/factories/user.factory.ts import { faker } from '@faker-js/faker'; import { User, UserRole } from '../../src/models/user'; export class UserFactory { static create(overrides: Partial<User> = {}): User { return { id: faker.string.uuid(), email: faker.internet.email(), password: faker.internet.password(), firstName: faker.person.firstName(), lastName: faker.person.lastName(), role: UserRole.USER, createdAt: faker.date.past(), updatedAt: faker.date.recent(), ...overrides, }; } static createAdmin(overrides: Partial<User> = {}): User { return this.create({ role: UserRole.ADMIN, ...overrides }); } static createMany(count: number, overrides: Partial<User> = {}): User[] { return Array.from({ length: count }, () => this.create(overrides)); } } ``` ### 6. Flaky Test Detection Script ```python #!/usr/bin/env python3 """Flaky test detection and quarantine script.""" import json import subprocess from datetime import datetime, timedelta from pathlib import Path class FlakyTestDetector: def __init__(self, results_dir: str = "test-results"): self.results_dir = Path(results_dir) self.flaky_threshold = 3 # failures in last 10 runs def analyze_results(self) -> list: """Analyze test results for flakiness patterns.""" flaky_tests = [] results = self._load_recent_results(days=7) for test_name, runs in results.items(): failures = sum(1 for r in runs if r["status"] == "failed") if failures >= self.flaky_threshold: flaky_tests.append({ "test": test_name, "failure_rate": failures / len(runs), "runs": len(runs), "last_failure": max(r["timestamp"] for r in runs if r["status"] == "failed") }) return flaky_tests def quarantine(self, test_name: str) -> None: """Add test to quarantine list.""" quarantine_file = self.results_dir / "quarantine.json" quarantined = json.loads(quarantine_file.read_text()) if quarantine_file.exists() else [] quarantined.append({ "test": test_name, "quarantined_at": datetime.now().isoformat(), "reason": "flaky" }) quarantine_file.write_text(json.dumps(quarantined, indent=2)) def _load_recent_results(self, days: int) -> dict: """Load test results from the last N days.""" results = {} cutoff = datetime.now() - timedelta(days=days) for result_file in self.results_dir.glob("results-*.json"): data = json.loads(result_file.read_text()) for test in data.get("tests", []): timestamp = datetime.fromisoformat(test["timestamp"]) if timestamp >= cutoff: results.setdefault(test["name"], []).append(test) return results if __name__ == "__main__": detector = FlakyTestDetector() flaky = detector.analyze_results() for test in flaky: print(f"Flaky test detected: {test['test']} (failure rate: {test['failure_rate']:.1%})") detector.quarantine(test["test"]) ``` ### 7. Performance Test Example (k6) ```javascript // performance/load-test.js import http from 'k6/http'; import { check, sleep } from 'k6'; import { Rate, Trend } from 'k6/metrics'; const errorRate = new Rate('error_rate'); const responseTime = new Trend('response_time'); export const options = { stages: [ { duration: '2m', target: 50 }, // ramp up { duration: '5m', target: 50 }, // steady state { duration: '2m', target: 100 }, // ramp up { duration: '5m', target: 100 }, // peak load { duration: '2m', target: 0 }, // ramp down ], thresholds: { http_req_duration: ['p(95)<500', 'p(99)<1000'], error_rate: ['rate<0.01'], }, }; export default function () { const baseUrl = __ENV.BASE_URL || 'http://localhost:3000'; // Critical flow: user login const loginRes = http.post(`${baseUrl}/api/auth/login`, { email: `user${__VU}@example.com`, password: 'password123', }); check(loginRes, { 'login status is 200': (r) => r.status === 200, }); const token = loginRes.json('token'); const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', }; // Critical flow: search products const searchRes = http.get( `${baseUrl}/api/products?search=laptop&page=1&limit=20`, { headers } ); check(searchRes, { 'search status is 200': (r) => r.status === 200, 'search returns results': (r) => r.json('items').length > 0, }); responseTime.add(searchRes.timings.duration); errorRate.add(searchRes.status !== 200); sleep(1); } ``` --- ## Commands ### Local Development ```bash # Install dependencies npm ci # Run unit tests with coverage npm run test:unit -- --coverage # Run integration tests npm run test:integration # Run E2E tests npm run test:e2e # Run all tests npm test # Run security scans npm audit --audit-level=high npm run security:sast npx gitleaks detect --verbose # Run accessibility tests npm run test:accessibility # Run performance tests k6 run performance/load-test.js # Run contract tests npm run test:contracts ``` ### CI Pipeline ```bash # All quality gates run automatically on PR and push to main # Quality gates include: unit, integration, e2e, security, accessibility # Coverage thresholds enforced: 80% lines, 70% branches # Test time budgets: unit <5min, integration <15min, e2e <30min ``` --- ## Quality Assurance Task Checklist - [ ] Every recommendation maps to a requirement or risk statement - [ ] Coverage references cite relevant code areas, services, or critical paths - [ ] Recommendations reference current test and defect data where available - [ ] All findings are based on identified risks, not assumptions - [ ] Test descriptions provide concrete scenarios, not vague summaries - [ ] Automated vs manual tests are clearly distinguished - [ ] Quality gate verification steps are actionable and measurable --- ## Additional Task Focus Areas ### Stability and Regression - [ ] **QE-ITEM-10.1 [Regression Risk Assessment]**: - **Area**: All critical flows - **Risk Level**: High - **Scope**: Regression suite for critical user flows - **Scenarios**: Full regression for critical flows, smoke regression for low-risk areas - **Success Criteria**: Regression suite executes within time budget, all critical paths validated - **Automation Level**: Fully automated - **Effort**: Ongoing - [ ] **QE-ITEM-10.2 [Flakiness Prevention]**: - **Area**: Test suite stability - **Risk Level**: High - **Scope**: All automated tests - **Scenarios**: N/A - **Success Criteria**: Flake rate <1%, flaky tests quarantined within 24 hours - **Automation Level**: Automated detection, manual remediation - **Effort**: Ongoing - [ ] **QE-ITEM-10.3 [Test Stability Monitoring]**: - **Area**: Test infrastructure - **Risk Level**: Medium - **Scope**: All test environments and infrastructure - **Scenarios**: N/A - **Success Criteria**: Test infrastructure uptime ≥99.9% - **Automation Level**: Automated monitoring - **Effort**: 1 sprint - [ ] **QE-ITEM-10.4 [Release Confidence Indicators]**: - **Area**: Release management - **Risk Level**: High - **Scope**: All releases - **Scenarios**: N/A - **Success Criteria**: Clear indicators for release confidence - **Automation Level**: Automated dashboard - **Effort**: 1-2 sprints ### Non-Functional Coverage - [ ] **QE-ITEM-11.1 [Reliability Targets]**: - **Area**: System reliability - **Risk Level**: Critical - **Scope**: All critical services - **Scenarios**: N/A - **Success Criteria**: 99.9% uptime for critical services - **Automation Level**: Automated monitoring - **Effort**: Ongoing - [ ] **QE-ITEM-11.2 [Performance Baselines]**: - **Area**: Performance - **Risk Level**: High - **Scope**: All critical endpoints - **Scenarios**: N/A - **Success Criteria**: Baselines established and monitored - **Automation Level**: Automated baseline collection - **Effort**: 1-2 sprints - [ ] **QE-ITEM-11.3 [Security Baseline in CI]**: - **Area**: Security - **Risk Level**: Critical - **Scope**: All code and dependencies - **Scenarios**: N/A - **Success Criteria**: Security scans pass in CI - **Automation Level**: Fully automated - **Effort**: 1 sprint - [ ] **QE-ITEM-11.4 [Compliance Coverage]**: - **Area**: Regulatory compliance - **Risk Level**: High - **Scope**: All compliance requirements - **Scenarios**: N/A - **Success Criteria**: All compliance requirements tested - **Automation Level**: Automated where possible, manual for regulatory - **Effort**: 2-3 sprints --- ## Execution Reminders - [ ] Prioritize coverage by risk so that the highest-impact areas receive the most rigorous testing - [ ] Provide concrete, measurable targets rather than aspirational statements - [ ] Balance automation investment against the defect categories that cause the most production pain - [ ] Treat test infrastructure as a first-class engineering concern with versioning, review, and monitoring - [ ] Close the feedback loop by routing production defects back into strategy refinement - [ ] Evolve continuously; a strategy that never changes is a strategy that has already drifted from reality