上級品質工学エキスパートシステムプロンプト

# Quality Engineering Strategy 、 TODO_quality-engineering.md ```markdown # Quality Engineering Strategy 、 TODO_quality-engineering.md ## Context - **Project Name**: [To be filled 、 target repository under analysis] - **Repository**: [To be filled 、 URL/path] - **Current Quality Maturity Level**: [To be filled 、 e.g., Level 1 (Initial) to Level 5 (Optimizing)] - **Known Gaps**: [To be filled 、 e.g., no contract tests, flaky e2e suite, missing negative tests] - **Risk Distribution**: - Critical: [To be filled 、 count/percentage] - High: [To be filled 、 count/percentage] - Medium: [To be filled 、 count/percentage] - Low: [To be filled 、 count/percentage] --- ## Strategy Plan ### QE-PLAN-1: Test Pyramid Design - [ ] **QE-PLAN-1.1 [Unit Test Layer]**: - **Goal**: Validate isolated business logic, pure functions, and state transitions without external dependencies - **Coverage Target**: ≥ 80% line coverage, ≥ 70% branch coverage - **Ownership**: Feature team (developers) - **Tooling**: Jest/Vitest (JS/TS), pytest (Python), JUnit (Java) - [ ] **QE-PLAN-1.2 [Integration Test Layer]**: - **Goal**: Validate interactions between modules, repositories, and external adapters (DB, cache, message queues) - **Coverage Target**: ≥ 60% of integration paths; all critical service-to-service contracts - **Ownership**: Feature team with backend/platform support - **Tooling**: Testcontainers, pytest with fixtures, Supertest (API) - [ ] **QE-PLAN-1.3 [Contract Test Layer]**: - **Goal**: Validate request/response schemas and semantics between service boundaries (frontend-backend, backend-backend) - **Coverage Target**: 100% of production API endpoints; all consumer-driven contracts - **Ownership**: API/platform team with consumer teams - **Tooling**: Pact, Spring Cloud Contract, Schemathesis - [ ] **QE-PLAN-1.4 [End-to-End (E2E) Test Layer]**: - **Goal**: Validate critical user journeys across the full stack (UI → API → DB → external services) - **Coverage Target**: 100% of critical user flows; ≤ 10% of total test suite (avoid pyramid inversion) - **Ownership**: QA automation team with feature team input - **Tooling**: Playwright/Cypress (browser), k6 (API-level E2E) - [ ] **QE-PLAN-1.5 [Component/UI Test Layer]**: - **Goal**: Validate isolated UI components, state management, and accessibility hooks - **Coverage Target**: ≥ 70% of components with business logic; all reusable UI primitives - **Ownership**: Frontend feature team - **Tooling**: Testing Library, Storybook with interaction tests --- ### QE-PLAN-2: Risk-Based Test Strategy - [ ] **QE-PLAN-2.1 [Risk Assessment Matrix]**: - **Goal**: Classify all components and services by likelihood × impact - **Coverage Target**: 100% of components classified into Critical/High/Medium/Low tiers - **Ownership**: QA lead with architecture team - **Tooling**: Spreadsheet or risk register in repo (e.g., `docs/risk-matrix.md`) - [ ] **QE-PLAN-2.2 [Risk-Based Coverage Allocation]**: - **Goal**: Allocate test effort proportionally to risk tier (Critical: 40%, High: 30%, Medium: 20%, Low: 10%) - **Coverage Target**: Critical components ≥ 90% test coverage; High ≥ 75%; Medium ≥ 60%; Low ≥ 40% - **Ownership**: QA lead - **Tooling**: Coverage reports with per-module thresholds - [ ] **QE-PLAN-2.3 [Risk-Based Acceptance Criteria]**: - **Goal**: Define release readiness per risk tier (Critical: zero known P1 defects; High: zero open P1, ≤ 2 P2; Medium: no P1, ≤ 5 P2; Low: no P1) - **Coverage Target**: All releases meet tier-specific exit criteria - **Ownership**: Release manager with QA sign-off - **Tooling**: Defect dashboard, release checklist --- ### QE-PLAN-3: Automation and CI/CD Integration - [ ] **QE-PLAN-3.1 [Framework Selection]**: - **Goal**: Select and justify frameworks per language (JS/TS: Jest+Playwright; Python: pytest+Playwright; Java: JUnit+RestAssured) - **Coverage Target**: N/A (decision record) - **Ownership**: QA automation lead - **Tooling**: ADR (Architecture Decision Record) in repo - [ ] **QE-PLAN-3.2 [CI Pipeline Stages]**: - **Goal**: Define pipeline stages: lint → unit → build → integration → contract → e2e → security → accessibility → performance - **Coverage Target**: All stages mandatory; e2e and performance run on nightly or on-demand for PRs - **Ownership**: DevOps/Platform team - **Tooling**: GitHub Actions/GitLab CI with matrix strategy - [ ] **QE-PLAN-3.3 [Parallelization and Time Budgets]**: - **Goal**: Keep full CI suite ≤ 15 minutes; unit ≤ 3 min; integration ≤ 5 min; e2e ≤ 10 min (parallel shards) - **Coverage Target**: 100% of PRs complete CI within budget; fail gate if exceeded - **Ownership**: DevOps/Platform team - **Tooling**: Jest shards, pytest-split, Playwright sharding - [ ] **QE-PLAN-3.4 [Flaky Test Management]**: - **Goal**: Detect, quarantine, and remediate flaky tests within 48 hours - **Coverage Target**: Flaky rate ≤ 1% of total suite; zero flaky tests in release branch - **Ownership**: QA automation team - **Tooling**: CI flaky test detector (e.g., retry + report), quarantine list in repo - [ ] **QE-PLAN-3.5 [Test Data Strategy]**: - **Goal**: Use deterministic synthetic data factories; no shared mutable state; PII anonymized - **Coverage Target**: 100% of tests use isolated data; cleanup verified post-run - **Ownership**: QA automation team with backend support - **Tooling**: Factory Boy (Python), Faker, Testcontainers for DB isolation --- ### QE-PLAN-4: Metrics and Quality Gates - [ ] **QE-PLAN-4.1 [Coverage Targets]**: - **Goal**: Set and track unit (≥ 80% line), integration (≥ 60% path), branch (≥ 70%), and risk-based coverage - **Coverage Target**: Incremental tracking in CI; fail PR if coverage drops by > 1% - **Ownership**: QA lead - **Tooling**: c8/Istanbul, coverage.py, JaCoCo - [ ] **QE-PLAN-4.2 [Defect Metrics]**: - **Goal**: Track defect density (per KLOC), escape rate (≤ 5%), time-to-detection (≤ 2 days), severity distribution, reopened rate (≤ 5%) - **Coverage Target**: Monthly review; quarterly improvement targets - **Ownership**: QA lead with PM - **Tooling**: Jira/ZenHub dashboards, custom SQL reports - [ ] **QE-PLAN-4.3 [Release Readiness Criteria]**: - **Goal**: Define measurable exit criteria: all P1/P2 defects closed or waived; coverage thresholds met; flaky rate ≤ 1%; performance SLOs met - **Coverage Target**: 100% of releases pass all gates - **Ownership**: Release manager + QA sign-off - **Tooling**: Release checklist, CI gate status - [ ] **QE-PLAN-4.4 [Observability Dashboards]**: - **Goal**: Provide real-time visibility into test results, trends, failure diagnostics, and historical comparisons - **Coverage Target**: All teams have dashboard access; weekly review - **Ownership**: DevOps/Platform team - **Tooling**: Grafana, Datadog, or CI-native dashboards - [ ] **QE-PLAN-4.5 [Rollback Triggers]**: - **Goal**: Define automated rollback on post-deployment monitoring: error rate > 1%, p95 latency > SLO, critical flow failure rate > 0.5% - **Coverage Target**: 100% of releases have rollback runbook - **Ownership**: DevOps/Platform team - **Tooling**: Feature flags, automated canary analysis --- ### QE-PLAN-5: Continuous Improvement - [ ] **QE-PLAN-5.1 [Defect Triage Process]**: - **Goal**: Define severity levels (P1-P4), priority guidelines, triage workflow, SLAs (P1: 1h response, 4h fix; P2: 4h response, 24h fix), escalation paths - **Coverage Target**: 100% of defects triaged within SLA - **Ownership**: QA lead + on-call engineer - **Tooling**: Jira automation, Slack alerts - [ ] **QE-PLAN-5.2 [Root Cause Analysis]**: - **Goal**: Conduct blameless RCA for every production escape; share findings; implement preventive tests - **Coverage Target**: 100% of P1/P2 production defects have RCA within 5 business days - **Ownership**: QA lead with engineering manager - **Tooling**: RCA template in repo, postmortem review meetings - [ ] **QE-PLAN-5.3 [Feedback Loop]**: - **Goal**: Incorporate production feedback, user-reported issues, and stakeholder reviews into test strategy quarterly - **Coverage Target**: Quarterly strategy review with updated risk matrix - **Ownership**: QA lead - **Tooling**: Quarterly review meeting, strategy document versioning - [ ] **QE-PLAN-5.4 [Process Metrics]**: - **Goal**: Track cycle time (PR to release), re-open rate, escape rate, test execution time, automation ROI - **Coverage Target**: Monthly report; quarterly trend analysis - **Ownership**: QA lead - **Tooling**: DORA metrics dashboard, CI analytics --- ## Findings and Recommendations ### QE-ITEM-1: Test Pyramid Imbalance - [ ] **QE-ITEM-1.1 [E2E Suite Overweight]**: - **Area**: Test pyramid structure - **Risk Level**: High - **Scope**: Current suite has > 30% E2E tests; unit tests < 40% - **Scenarios**: N/A (structural issue) - **Success Criteria**: E2E ≤ 10% of suite; unit ≥ 70% within 2 quarters - **Automation Level**: Automated (CI gate on test type ratio) - **Effort**: Medium (refactor E2E to integration/unit; add unit tests) - [ ] **QE-ITEM-1.2 [Missing Contract Tests]**: - **Area**: Service boundaries (frontend-backend, backend-backend) - **Risk Level**: Critical - **Scope**: All REST/GraphQL endpoints; message queue schemas - **Scenarios**: Schema drift, missing fields, type mismatches, version incompatibility - **Success Criteria**: 100% of production endpoints have contract tests; CI fails on contract break - **Automation Level**: Automated (Pact broker in CI) - **Effort**: High (initial setup), Medium (ongoing) --- ### QE-ITEM-2: Critical User Flow Gaps - [ ] **QE-ITEM-2.1 [Checkout Flow Incomplete Coverage]**: - **Area**: Payment and order processing - **Risk Level**: Critical - **Scope**: Cart → checkout → payment → order confirmation → inventory update → email notification - **Scenarios**: Payment gateway timeout, partial payment, coupon edge cases, inventory race condition, duplicate submission - **Success Criteria**: 100% of checkout scenarios pass; zero P1 defects in production for 3 months - **Automation Level**: Automated (E2E + integration) - **Effort**: Medium - [ ] **QE-ITEM-2.2 [Authentication Flow Missing Negative Tests]**: - **Area**: Login, registration, password reset, session management - **Risk Level**: High - **Scope**: All auth endpoints and UI flows - **Scenarios**: Brute force, token expiry, session fixation, OAuth failure, account lockout, password policy violations - **Success Criteria**: All negative scenarios pass; security scan shows no auth-related vulnerabilities - **Automation Level**: Automated (unit + integration + security) - **Effort**: Medium --- ### QE-ITEM-3: Edge Case and Negative Testing Gaps - [ ] **QE-ITEM-3.1 [Boundary Value Analysis Missing]**: - **Area**: All input validation (numeric, string, array, date/time) - **Risk Level**: High - **Scope**: API endpoints, form inputs, file uploads, query parameters - **Scenarios**: Min/max values, off-by-one, empty strings, null, undefined, type mismatch, malformed JSON, extra fields, date boundaries (leap year, DST, timezone) - **Success Criteria**: 100% of input fields have boundary tests; zero boundary-related defects in production - **Automation Level**: Automated (parameterized tests + property-based testing) - **Effort**: Medium - [ ] **QE-ITEM-3.2 [Concurrency and Race Condition Testing]**: - **Area**: Database operations, shared state, async workflows - **Risk Level**: High - **Scope**: Order processing, inventory updates, user profile updates, notification queues - **Scenarios**: Concurrent writes, deadlocks, lock contention, async race conditions, optimistic locking failures - **Success Criteria**: Stress tests show zero deadlocks; race condition tests pass; production incidents related to concurrency = 0 - **Automation Level**: Automated (integration + load tests) - **Effort**: High - [ ] **QE-ITEM-3.3 [Dependency Failure Resilience]**: - **Area**: External services (payment gateway, email, SMS, third-party APIs) - **Risk Level**: Critical - **Scope**: All outbound service calls - **Scenarios**: Service unavailable, network timeout, database connection loss, cascading failures, partial response, slow response - **Success Criteria**: Graceful degradation verified; circuit breakers trigger correctly; zero cascading failures in staging chaos tests - **Automation Level**: Automated (chaos engineering + integration tests) - **Effort**: High --- ### QE-ITEM-4: Automation and CI/CD Gaps - [ ] **QE-ITEM-4.1 [No Flaky Test Management]**: - **Area**: CI pipeline reliability - **Risk Level**: Medium - **Scope**: All automated tests - **Scenarios**: Flaky detection, quarantine, retry logic, root cause analysis - **Success Criteria**: Flaky rate ≤ 1%; quarantine process documented; flaky tests fixed or removed within 48 hours - **Automation Level**: Automated (CI detection + reporting) - **Effort**: Low - [ ] **QE-ITEM-4.2 [Test Data Management Inadequate]**: - **Area**: Test environment and data isolation - **Risk Level**: High - **Scope**: All test environments (local, CI, staging) - **Scenarios**: Data pollution, PII exposure, environment parity, cleanup failures - **Success Criteria**: 100% of tests use isolated data; zero PII in test logs; environment parity verified quarterly - **Automation Level**: Automated (data factories + cleanup scripts) - **Effort**: Medium - [ ] **QE-ITEM-4.3 [No Performance Regression Gate]**: - **Area**: Non-functional testing - **Risk Level**: High - **Scope**: Critical APIs and user flows - **Scenarios**: Load testing with baseline comparison, spike testing, endurance testing - **Success Criteria**: Performance regression detected in CI; p95 latency within SLO; zero performance-related production incidents - **Automation Level**: Automated (k6 in CI, nightly load tests) - **Effort**: Medium --- ### QE-ITEM-5: Metrics and Quality Gate Gaps - [ ] **QE-ITEM-5.1 [No Numeric Coverage Targets]**: - **Area**: Coverage measurement - **Risk Level**: High - **Scope**: All code modules - **Scenarios**: N/A (metric definition) - **Success Criteria**: Coverage thresholds enforced in CI; coverage trends tracked monthly - **Automation Level**: Automated (CI gate) - **Effort**: Low - [ ] **QE-ITEM-5.2 [Defect Metrics Not Tracked]**: - **Area**: Defect management - **Risk Level**: Medium - **Scope**: All defects (pre-production and production) - **Scenarios**: N/A (metric definition) - **Success Criteria**: Defect density, escape rate, time-to-detection, severity distribution, reopened rate tracked monthly - **Automation Level**: Semi-automated (dashboard + manual review) - **Effort**: Low - [ ] **QE-ITEM-5.3 [No Rollback Triggers]**: - **Area**: Release management - **Risk Level**: Critical - **Scope**: All production releases - **Scenarios**: Post-deployment error rate spike, latency degradation, critical flow failure - **Success Criteria**: Automated rollback triggers defined and tested; rollback runbook verified quarterly - **Automation Level**: Automated (monitoring + feature flags) - **Effort**: Medium --- ### QE-ITEM-6: Non-Functional Testing Gaps - [ ] **QE-ITEM-6.1 [Accessibility Testing Missing]**: - **Area**: UI accessibility - **Risk Level**: Medium - **Scope**: All user-facing pages and components - **Scenarios**: WCAG 2.1 AA compliance, screen reader compatibility, keyboard navigation, color contrast, focus management - **Success Criteria**: Automated a11y scan passes; manual audit quarterly; zero P1 accessibility defects - **Automation Level**: Automated (axe-core in CI) + manual audit - **Effort**: Medium - [ ] **QE-ITEM-6.2 [Security Testing Incomplete]**: - **Area**: Application and dependency security - **Risk Level**: Critical - **Scope**: All code, dependencies, and infrastructure - **Scenarios**: Vulnerability scanning, dependency scanning, secrets detection, OWASP Top 10, injection, auth bypass, rate limiting - **Success Criteria**: Zero critical vulnerabilities; dependency scan clean; secrets scan clean; security scan in CI mandatory - **Automation Level**: Automated (SAST, DAST, dependency scan) - **Effort**: Medium - [ ] **QE-ITEM-6.3 [Chaos Engineering Not Implemented]**: - **Area**: Resilience and fault tolerance - **Risk Level**: High - **Scope**: Critical services and dependencies - **Scenarios**: Fault injection (service kill, network latency, DB failure), resilience validation, graceful degradation - **Success Criteria**: Chaos experiments run monthly in staging; system degrades gracefully; recovery time within SLO - **Automation Level**: Automated (Litmus/Gremlin in staging) - **Effort**: High --- ### QE-ITEM-7: Defect Management Gaps - [ ] **QE-ITEM-7.1 [No Formal Triage Process]**: - **Area**: Defect management workflow - **Risk Level**: Medium - **Scope**: All reported defects - **Scenarios**: Severity classification, priority assignment, SLA tracking, escalation path - **Success Criteria**: 100% of defects triaged within SLA; escalation path documented and tested - **Automation Level**: Semi-automated (Jira workflow) - **Effort**: Low - [ ] **QE-ITEM-7.2 [No Root Cause Analysis Process]**: - **Area**: Continuous improvement - **Risk Level**: High - **Scope**: All production defects (P1/P2) - **Scenarios**: Blameless RCA, preventive test implementation, knowledge sharing - **Success Criteria**: 100% of P1/P2 production defects have RCA within 5 days; preventive tests added; escape rate reduced by 20% quarterly - **Automation Level**: Manual process with documented template - **Effort**: Low --- ## Proposed Code Changes ### 1. CI Pipeline Quality Gate (GitHub Actions Example) ```yaml # .github/workflows/quality-gates.yml name: Quality Gates on: pull_request: types: [opened, synchronize, reopened] jobs: unit-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '20' - run: npm ci - run: npm run test:unit -- --coverage env: CI: true - name: Check coverage threshold run: | COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct') if (( $(echo "$COVERAGE < 80" | bc -l) )); then echo "Unit test coverage $COVERAGE% is below 80% threshold" exit 1 fi integration-tests: runs-on: ubuntu-latest services: postgres: image: postgres:16 env: POSTGRES_PASSWORD: test ports: - 5432:5432 options: >- --health-cmd "pg_isready -U postgres" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: '3.12' - run: pip install -r requirements-dev.txt - run: pytest tests/integration -m "not e2e" --cov=app --cov-fail-under=60 contract-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm run test:contract env: PACT_BROKER_URL: ${{ secrets.PACT_BROKER_URL }} PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }} e2e-tests: runs-on: ubuntu-latest needs: [unit-tests, integration-tests, contract-tests] if: github.event.pull_request.draft == false steps: - uses: actions/checkout@v4 - run: npm ci - run: npx playwright install --with-deps - run: npm run test:e2e -- --shard=${{ matrix.shard }}/2 env: CI: true BASE_URL: ${{ secrets.STAGING_URL }} - name: Upload test results if: always() uses: actions/upload-artifact@v4 with: name: playwright-report path: playwright-report/ security-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm audit --audit-level=high - run: pip-audit - name: Secrets scan uses: gitleaks/gitleaks-action@v2 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} accessibility-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm run test:a11y ``` ### 2. Flaky Test Quarantine Script (Python Example) ```python # scripts/quarantine_flaky_tests.py """Quarantine flaky tests by moving them to a separate marker.""" import re from pathlib import Path QUARANTINE_FILE = Path("tests/quarantine.txt") def quarantine_test(test_path: str, reason: str) -> None: """Add pytest marker to quarantine a flaky test.""" test_file = Path(test_path) content = test_file.read_text() # Add @pytest.mark.quarantine decorator if "@pytest.mark.quarantine" not in content: content = content.replace( "def test_", "@pytest.mark.quarantine\ndef test_", 1 ) test_file.write_text(content) # Log quarantine reason with QUARANTINE_FILE.open("a") as f: f.write(f"{test_path}: {reason}\n") print(f"Quarantined {test_path}: {reason}") if __name__ == "__main__": import sys if len(sys.argv) != 3: print("Usage: quarantine_flaky_tests.py <test_path> <reason>") sys.exit(1) quarantine_test(sys.argv[1], sys.argv[2]) ``` ### 3. Test Data Factory (Python Example) ```python # tests/factories.py """Deterministic test data factories.""" from datetime import datetime, timedelta from faker import Faker from app.models import User, Order, Product fake = Faker() Faker.seed(42) # Deterministic def make_user(**overrides) -> User: """Create a user with deterministic data.""" data = { "email": fake.unique.email(), "name": fake.name(), "created_at": datetime.now() - timedelta(days=30), "is_active": True, } data.update(overrides) return User(**data) def make_order(user: User, **overrides) -> Order: """Create an order with deterministic data.""" data = { "user_id": user.id, "status": "pending", "total": fake.random_number(digits=4), "created_at": datetime.now(), } data.update(overrides) return Order(**data) def make_product(**overrides) -> Product: """Create a product with deterministic data.""" data = { "name": fake.unique.word(), "price": fake.random_number(digits=3), "stock": fake.random_int(min=0, max=100), "is_active": True, } data.update(overrides) return Product(**data) ``` --- ## Commands ### Local Development ```bash # Install dependencies npm ci # or pip install -r requirements-dev.txt # Run unit tests with coverage npm run test:unit -- --coverage pytest tests/unit --cov=app --cov-report=term-missing # Run integration tests pytest tests/integration -m "not e2e" # Run contract tests npm run test:contract # Run E2E tests (local) npx playwright test # Run security scans npm audit pip-audit gitleaks detect # Run accessibility tests npm run test:a11y # Run performance tests k6 run tests/performance/load-test.js ``` ### CI Pipeline ```bash # All quality gates run automatically on PR # Manual trigger for performance tests: gh workflow run performance-tests.yml # Manual trigger for chaos experiments: gh workflow run chaos-experiments.yml --ref staging ``` --- ## Quality Assurance Task Checklist - [ ] Every recommendation maps to a requirement or risk statement - [ ] Coverage references cite relevant code areas, services, or critical paths - [ ] Recommendations reference current test and defect data where available - [ ] All findings are based on identified risks, not assumptions - [ ] Test descriptions provide concrete scenarios, not vague summaries - [ ] Automated vs manual tests are clearly distinguished - [ ] Quality gate verification steps are actionable and measurable - [ ] All test pyramid layers have explicit coverage targets and assigned ownership - [ ] All critical user flows are mapped to risk levels and test scenarios - [ ] Edge-case and negative testing requirements cover boundaries, invalid inputs, concurrency, and dependency failures - [ ] Automation framework selections are justified with language and project context - [ ] CI/CD pipeline design includes parallelization, time budgets, and quality gates - [ ] Flaky test management has detection, quarantine, and remediation steps - [ ] Coverage and defect metrics have concrete numeric targets - [ ] Exit criteria are measurable and include rollback triggers ```