Testplan Integration

PyUCIS can embed a structured testplan inside each NCDB .cdb file. A testplan describes the verification tasks (testpoints) and functional coverage groups expected for a design. Together with the binary test history (see Test History) it enables:

Closure reporting — did every testpoint’s tests actually pass?
Stage gate evaluation — are all V1/V2/V3 testpoints closed?
Merge propagation — the testplan travels with the database so reports always use the correct plan.

Quick-start 

Import an OpenTitan-style Hjson testplan and embed it in a .cdb:

from ucis.ncdb.testplan_hjson import import_hjson
from ucis.ncdb.ncdb_ucis import NcdbUCIS
from ucis.ncdb.ncdb_writer import NcdbWriter

plan = import_hjson("uart_testplan.hjson",
                    substitutions={"baud": ["9600", "115200"]})

db = NcdbUCIS("coverage.cdb")
db.setTestplan(plan)
NcdbWriter().write(db, "coverage.cdb")

Compute closure against the embedded testplan:

from ucis.ncdb.testplan_closure import compute_closure, stage_gate_status
from ucis.ncdb.testplan import get_testplan

db = NcdbUCIS("coverage.cdb")
plan = db.getTestplan()
results = compute_closure(plan, db)

for r in results:
    print(f"{r.testpoint.name:30s} {r.status.value}")

gate = stage_gate_status(results, "V2", plan)
print(gate["message"])

Testplan format 

A testplan is stored as testplan.json inside the NCDB ZIP and is also exportable as a standalone JSON file. The schema is:

{
  "format_version": 1,
  "source_file": "path/to/uart.hjson",
  "import_timestamp": "2025-01-01T00:00:00+00:00",
  "testpoints": [
    {
      "name": "uart_reset",
      "stage": "V1",
      "desc": "Verify reset behaviour",
      "tests": ["uart_smoke", "uart_init_*"],
      "tags": ["smoke"],
      "na": false,
      "source_template": ""
    }
  ],
  "covergroups": [
    {"name": "cg_uart_reset", "desc": "Reset coverage"}
  ]
}

Stages follow the OpenTitan V1 → V2 → V2S → V3 hierarchy; custom strings are also accepted and sort after V3 in gate evaluation.

Importing Hjson 

Use import_hjson() to parse an OpenTitan .hjson testplan (or a standard .json file):

plan = import_hjson(
    "uart_testplan.hjson",
    substitutions={
        "uart":  ["uart0", "uart1"],
        "mode":  ["loopback", "normal"],
    },
)

The substitutions dict provides values for {key} placeholders in test name templates. A list value generates the cartesian product of all combinations:

# Template: "{uart}_{mode}_test"
# Substitutions: uart=["uart0","uart1"], mode=["loopback","normal"]
# Expands to: uart0_loopback_test, uart0_normal_test,
#             uart1_loopback_test, uart1_normal_test

Tests listed as ["N/A"] are treated as intentionally unmapped (testpoint.na = True).

Closure computation 

compute_closure() evaluates each testpoint against the test history stored in the database:

Status	Meaning
`CLOSED`	All mapped tests have at least one passing run
`PARTIAL`	Some passing, some failing
`FAILING`	All mapped tests failed
`NOT_RUN`	None of the mapped tests appear in the database
`N/A`	Testpoint has `na = True`
`UNIMPLEMENTED`	Testpoint has an empty `tests` list

Test name matching uses three strategies in order:

Exact — the test name appears literally in testpoint.tests.
Seed-suffix strip — trailing _\d+ is removed and the result matched exactly (e.g. uart_smoke_42 → uart_smoke).
Wildcard — any testpoint.tests entry ending in _* is used as a prefix match.

Stage gate evaluation 

stage_gate_status() determines whether a regression is ready to advance to the next stage:

gate = stage_gate_status(results, "V2", plan)
if gate["passed"]:
    print("Ready to tape-out!")
else:
    for r in gate["blocking"]:
        print(f"  BLOCKING: {r.testpoint.name}")

The gate passes when all testpoints at the target stage and all stages below it (V1 < V2 < V2S < V3) are CLOSED or N/A.

Waivers 

Coverage and test failures can be suppressed with a WaiverSet:

from ucis.ncdb.waivers import Waiver, WaiverSet

ws = WaiverSet([
    Waiver(
        id="W-001",
        scope_pattern="top/uart/**",
        bin_pattern="reset_*",
        rationale="Reset coverage deferred to V2",
        approver="eng",
        approved_at="2025-01-01T00:00:00",
        expires_at="2026-01-01T00:00:00",
    )
])

db.setWaivers(ws)
NcdbWriter().write(db, "coverage.cdb")

Scope patterns use glob syntax: * matches a single path segment, ** matches any number of segments. Expiry enforcement is the caller’s responsibility — use active_at() to filter out expired waivers before passing to closure:

import time
now = time.strftime("%Y-%m-%dT%H:%M:%S")
active_waivers = ws.active_at(now)

Modes A and B 

Mode A (embedded) — testplan stored inside the .cdb:

db.setTestplan(plan)
NcdbWriter().write(db, "coverage.cdb")

# Read back — travels with the database
db2 = NcdbUCIS("coverage.cdb")
plan2 = db2.getTestplan()

Mode B (standalone) — testplan kept as a separate file:

plan.save("uart_testplan_snapshot.json")

# Load later and pass to analysis functions
plan = Testplan.load("uart_testplan_snapshot.json")
results = compute_closure(plan, db)

Both modes produce the same Testplan object. The helper get_testplan() works with both:

from ucis.ncdb.testplan import get_testplan
plan = get_testplan(db)   # works for NcdbUCIS or MemUCIS

API reference 

ucis.ncdb.testplan_hjson.import_hjson(hjson_path: str, substitutions: Dict[str, object] | None = None) → Testplan

Parse an OpenTitan-style Hjson testplan and return a Testplan.

Parameters:

hjson_path – Path to the .hjson (or .json) file.
substitutions – Optional dict of {key: value_or_list} pairs used for wildcard expansion in test names.

Returns:

A fully expanded Testplan with all {key} templates replaced.

class ucis.ncdb.testplan.Testplan(format_version: int = 1, source_file: str = '', import_timestamp: str = '', testpoints: List[Testpoint] = <factory>, covergroups: List[CovergroupEntry] = <factory>, _tp_by_name: dict = <factory>, _tp_by_test: dict = <factory>, _indexed: bool = False)

Structured verification testplan.

format_version

Schema version (currently 1).

Type:: int

source_file

Path to the source .hjson (informational only).

Type:: str

import_timestamp

ISO-8601 UTC timestamp set when embedded in a .cdb.

Type:: str

testpoints

Ordered list of Testpoint objects.

Type:: List[ucis.ncdb.testplan.Testpoint]

covergroups

Ordered list of CovergroupEntry objects.

Type:: List[ucis.ncdb.testplan.CovergroupEntry]

add_testpoint(tp: Testpoint) → None: Append tp and invalidate the lookup indices.

classmethod from_bytes(data: bytes) → Testplan: Reconstruct from JSON bytes (inverse of serialize()).

getTestpoint(name: str) → Testpoint | None: Return the testpoint with name, or None.

classmethod load(path: str) → Testplan: Load a testplan from a standalone JSON/hjson file (Mode B).

save(path: str) → None: Write this testplan to a standalone JSON file (Mode B).

serialize() → bytes: Serialise to compact JSON bytes (for ZIP embedding).

stages() → List[str]: Return the ordered unique stages present in the testplan.

stamp_import_time() → None: Set import_timestamp to the current UTC time.

testpointForTest(test_name: str) → Testpoint | None

Return the testpoint that owns test_name.

Match order:

Exact — test_name appears literally in testpoint.tests.
Seed-suffix strip — strip a trailing _\d+ (e.g. uart_smoke_42 → uart_smoke) and retry exact match.
Wildcard — any testpoint.tests entry ending in _* whose prefix matches test_name.

Returns None if no testpoint matches.

testpointsForStage(stage: str) → List[Testpoint]: Return all testpoints targeting stage (e.g. "V2").

class ucis.ncdb.testplan.Testpoint(name: str, stage: str, desc: str = '', tests: List[str] = <factory>, tags: List[str] = <factory>, na: bool = False, source_template: str = '', requirements: List[RequirementLink] = <factory>): One verification task (maps to one or more test names).

class ucis.ncdb.testplan.CovergroupEntry(name: str, desc: str = ''): One functional-coverage group expected to be exercised by the design.

class ucis.ncdb.testplan.RequirementLink(system: str = '', project: str = '', item_id: str = '', url: str = ''): Reference to an external requirement item (e.g. ALM/JIRA).

ucis.ncdb.testplan_closure.compute_closure(testplan: Testplan, db, waivers=None) → List[TestpointResult]

Compute pass/fail closure for every testpoint against db.

Parameters:

testplan – The testplan to evaluate.
db – Any UCIS database object (must expose historyNodes()).
waivers – Optional WaiverSet; reserved for future use (currently ignored).

Returns:

One TestpointResult per testpoint, in testplan order.

ucis.ncdb.testplan_closure.stage_gate_status(results: List[TestpointResult], stage: str, testplan: Testplan, require_flake_score_below: float | None = None, require_coverage_pct: float | None = None) → dict

Determine whether the gate for stage is met.

A stage gate passes when ALL testpoints at stage and all stages with a lower standard index are CLOSED (or N/A).

Parameters:

results – Output of compute_closure().
stage – Stage to evaluate (e.g. "V2").
testplan – The testplan (used for stage ordering).
require_flake_score_below – Reserved — flakiness threshold (future).
require_coverage_pct – Reserved — coverage threshold (future).

Returns:

Dict with keys passed (bool), stage, blocking (list of TestpointResult that prevent the gate from passing), and message (human-readable summary string).

class ucis.ncdb.testplan_closure.TPStatus(*values): Closure status of one testpoint.

class ucis.ncdb.testplan_closure.TestpointResult(testpoint: Testpoint, status: TPStatus, matched_tests: List[str], pass_count: int = 0, fail_count: int = 0): Closure result for one testpoint.

class ucis.ncdb.waivers.WaiverSet(waivers: List[Waiver] | None = None)

Collection of Waiver objects.

waivers: Ordered list of waivers.

active_at(timestamp: str) → WaiverSet

Return a new WaiverSet containing only waivers that are active at timestamp (ISO-8601 string).

A waiver is active when:

status == "active"
expires_at is empty OR expires_at > timestamp

add(waiver: Waiver) → None: Append waiver to the set.

get(waiver_id: str) → Waiver | None: Return the waiver with waiver_id, or None.

classmethod load(path: str) → WaiverSet: Load from a standalone JSON file.

matches_scope(scope_path: str, bin_name: str = '') → bool: Return True if any waiver covers scope_path / bin_name.

save(path: str) → None: Write to a standalone JSON file.

serialize() → bytes: Serialise to compact JSON bytes (for ZIP embedding).

class ucis.ncdb.waivers.Waiver(id: str, scope_pattern: str, bin_pattern: str = '*', rationale: str = '', approver: str = '', approved_at: str = '', expires_at: str = '', status: str = 'active')

A single waiver entry.

id

Unique identifier (e.g. "W-001").

Type:: str

scope_pattern

Glob-style pattern matched against UCIS scope paths. * matches any single path segment; ** matches any number of segments.

Type:: str

bin_pattern

Glob-style pattern matched against bin names within the matched scope. Use "*" to waive the entire scope.

Type:: str

rationale

Human-readable explanation.

Type:: str

approver

Name/username of approver.

Type:: str

approved_at

ISO-8601 UTC timestamp of approval.

Type:: str

expires_at

ISO-8601 UTC timestamp after which this waiver expires. Empty string means “never expires”.

Type:: str

status

"active" | "expired" | "revoked".

Type:: str