Purpose
To evaluate vendors based on verifiable outcomes, risk posture, and long-term value, using independent telemetry and observability rather than self-reported metrics.
Scoring model overview
- Scale: 1–5 (1 = unacceptable, 5 = exceeds requirements).
- Weighting: adjust by criticality (example weights below).
- Evidence: scores must be supported by telemetry, logs, or audit artifacts.
- Review cadence: monthly (operational), quarterly (governance), annually (renewal).
Category A — Service performance and reliability (25%)
What this measures
Whether the vendor delivers what was contracted, consistently.
Evidence required: independent telemetry feeds, incident timelines, post-incident reports with timestamps.
- Uptime and availability against SLA.
- Latency and throughput vs agreed thresholds.
- Error rates and failure frequency.
- Mean time to detect (MTTD).
- Mean time to recover (MTTR).
Scoring guidance: 5 = exceeds SLAs with early warning; 3 = meets SLAs with reactive recovery; 1 = repeated SLA breaches or unverifiable performance.
Category B — Observability and transparency (15%)
How visible and understandable the vendor’s system behavior is.
Evidence required: telemetry schemas, access controls, incident root-cause analyses.
- Access to raw telemetry (not dashboards only).
- Traceability across components.
- Root cause clarity during incidents.
- Data retention and auditability.
Scoring guidance: 5 = full transparency; 3 = partial visibility; 1 = black-box systems with limited insight.
Category C — Governance and control (15%)
Whether the organization retains authority and control.
Evidence required: change logs, governance documentation, control plane designs.
- Change management process.
- Rollback and fail-safe mechanisms.
- Human override and escalation paths.
- AI and automation boundaries (if applicable).
Scoring guidance: 5 = explicit controls; 3 = informal controls; 1 = vendor-controlled changes without safeguards.
Category D — Security, privacy, and compliance (15%)
Exposure to regulatory, data, and security risk.
Evidence required: audit reports, security attestations, incident disclosures.
- Security incident history.
- Compliance certifications (where relevant).
- Data residency and access controls.
- Vulnerability response time.
Scoring guidance: 5 = proactive security; 3 = compliant but reactive; 1 = repeated or opaque security issues.
Category E — Dependency and lock-in risk (10%)
How difficult it is to replace or exit the vendor.
Evidence required: contract clauses, architecture diagrams, data export tests.
- Data portability.
- API openness and standards.
- Exit clauses and transition support.
- Substitute vendor feasibility.
Scoring guidance: 5 = low dependency; 3 = moderate friction; 1 = high lock-in and proprietary dependencies.
Category F — Financial integrity and cost predictability (10%)
Whether costs are predictable and aligned to value.
Evidence required: invoice analysis, usage telemetry, cost trend reports.
- Cost variance vs forecast.
- Pricing transparency.
- Overage frequency.
- Billing dispute rate.
Scoring guidance: 5 = predictable and transparent; 3 = occasional variance; 1 = frequent surprises or opaque pricing.
Category G — Responsiveness and accountability (10%)
How the vendor behaves under stress.
Evidence required: incident communications, action item tracking, escalation records.
- Incident response time.
- Quality of communication.
- Ownership during failures.
- Follow-through on remediation.
Scoring guidance: 5 = proactive and accountable; 3 = responsive but reactive; 1 = delayed or evasive responses.
Overall risk rating (derived)
- Green: ≥ 4.2 — Preferred / renewal candidate.
- Amber: 3.2–4.1 — Conditional / remediation required.
- Red: < 3.2 — Exit planning or replacement recommended.
Governance actions by score
- Green: renew, consider expanded scope.
- Amber: issue remediation plan with deadlines.
- Red: initiate exit strategy; restrict new dependency.
sys3(a)i value add (why this works)
- Designing telemetry that maps directly to contract terms.
- Providing independent observability across vendors.
- Reducing reliance on vendor self-reporting.
- Enabling objective renewal, renegotiation, or exit decisions.
- Preserving procurement leverage over time.
One-line procurement summary
This scorecard converts vendor performance from assumed compliance into measurable, enforceable outcomes using independent telemetry and governance controls.
sys3(a)i POV: We approach critical systems work by stress-testing architectures, integrating observability and governance from day one, and designing sovereign or edge footprints where independence and continuity matter most.