Reports and CI/CD Integration

The ucis.ncdb.reports and ucis.ncdb.testplan_export modules provide structured reports for testplan closure, regression delta, and CI/CD export. Every report function returns a typed dataclass with a to_json() method; companion format_*() functions render the dataclass to human-readable text.

Closure and gate reports 

from ucis.ncdb.ncdb_ucis import NcdbUCIS
from ucis.ncdb.testplan import get_testplan
from ucis.ncdb.testplan_closure import compute_closure
from ucis.ncdb.reports import (
    report_testpoint_closure,
    format_testpoint_closure,
    report_stage_gate,
    format_stage_gate,
)

db = NcdbUCIS("coverage.cdb")
plan = get_testplan(db)
results = compute_closure(plan, db)

# Print the closure table
summary = report_testpoint_closure(results)
print(format_testpoint_closure(summary))

# Evaluate a stage gate
gate = report_stage_gate(results, "V2", plan)
print(format_stage_gate(gate))

# Machine-readable JSON
import json
data = json.loads(summary.to_json())

Stage-rollup output example:

Testpoint                          Stage  Status     Pass   Fail
----------------------------------------------------------------
uart_reset                         V1     ✓ CLOSED      5      0
uart_loopback                      V2     ✗ FAILING     0      3
----------------------------------------------------------------

Stage roll-up:
  V1     [████████████████████] 1/1 (100.0%)
  V2     [░░░░░░░░░░░░░░░░░░░░] 0/1 (0.0%)

Total: 1/2 closed  (0 N/A)

Regression delta 

Compare two closure result sets to find testpoints that changed status between runs:

from ucis.ncdb.reports import report_regression_delta, format_regression_delta

# Load two snapshots
results_baseline = compute_closure(plan, db_baseline)
results_current  = compute_closure(plan, db_current)

delta = report_regression_delta(results_current, results_baseline)
print(format_regression_delta(delta))
# Regression delta: +1 closed, -0 newly failing, 1 still open

# Machine-readable
print(delta.to_json())

Reliability report 

Compute per-testpoint flake scores from v2 history data:

from ucis.ncdb.reports import report_testpoint_reliability, format_testpoint_reliability

report = report_testpoint_reliability(results, db)
print(format_testpoint_reliability(report))

Output example:

Testpoint                          Flake    Pass    Fail
--------------------------------------------------------
uart_loopback                      0.800       2       8  ⚠
uart_reset                         0.000      10       0

Unexercised covergroups 

Identify zero-hit or low-coverage covergroups:

from ucis.ncdb.reports import (
    report_unexercised_covergroups,
    format_unexercised_covergroups,
)

report = report_unexercised_covergroups(db, plan, low_threshold=50.0)
print(format_unexercised_covergroups(report))

Coverage contribution 

Show which tests contribute the most unique coverage bins (requires v2 history with contribution data):

from ucis.ncdb.reports import (
    report_coverage_contribution,
    format_coverage_contribution,
)

report = report_coverage_contribution(db)
print(format_coverage_contribution(report))

JUnit XML export 

Export closure results as a JUnit XML file for CI dashboards:

from ucis.ncdb.testplan_export import export_junit_xml

export_junit_xml(results, "closure_results.xml")

Or via the CLI:

pyucis testplan export-junit coverage.cdb --out closure_results.xml

The XML maps each testpoint to a <testcase> element. FAILING and PARTIAL testpoints become <failure> elements; NOT_RUN becomes <skipped>.

GitHub Annotations 

Emit GitHub Actions workflow commands for inline PR annotations:

from ucis.ncdb.testplan_export import export_github_annotations

export_github_annotations(results)  # writes to stdout

# Or capture to a string
import io
buf = io.StringIO()
export_github_annotations(results, output=buf)
print(buf.getvalue())

In a GitHub Actions workflow:

- name: Compute closure
  run: |
    python -c "
    from ucis.ncdb.ncdb_ucis import NcdbUCIS
    from ucis.ncdb.testplan import get_testplan
    from ucis.ncdb.testplan_closure import compute_closure
    from ucis.ncdb.testplan_export import export_github_annotations
    db = NcdbUCIS('coverage.cdb')
    plan = get_testplan(db)
    results = compute_closure(plan, db)
    export_github_annotations(results)
    "

Test budget by stage 

Estimate CPU-hour cost per stage from v2 test history mean CPU times:

from ucis.ncdb.ncdb_ucis import NcdbUCIS
from ucis.ncdb.testplan import get_testplan
from ucis.ncdb.reports import report_test_budget, format_test_budget

db   = NcdbUCIS("coverage.cdb")
plan = db.getTestplan()
rpt  = report_test_budget(plan, db)
print(format_test_budget(rpt))

# JSON export
import json
print(json.loads(rpt.to_json())["stage_totals"])

Testpoints whose tests have no CPU time recorded appear in rpt.missing_stats.

Safety traceability matrix 

Build a requirement-to-testpoint matrix (suitable for safety audits):

from ucis.ncdb.reports import report_safety_matrix, format_safety_matrix

rpt = report_safety_matrix(results)          # results from compute_closure
print(format_safety_matrix(rpt))

# CSV for a spreadsheet or audit tool
with open("traceability.csv", "w") as f:
    f.write(rpt.to_csv())

# Add a WaiverSet to flag waived testpoints
from ucis.ncdb.waivers import WaiverSet
waivers = WaiverSet.from_file("waivers.hjson")
rpt = report_safety_matrix(results, waivers=waivers)

Seed reliability heat-map 

Identify seeds that are disproportionately flaky:

from ucis.ncdb.ncdb_ucis import NcdbUCIS
from ucis.ncdb.reports import report_seed_reliability, format_seed_reliability

db  = NcdbUCIS("coverage.cdb")
rpt = report_seed_reliability(db, "uart_smoke")
print(format_seed_reliability(rpt))
# Seeds with flake_score ≥ 0.2 are flagged with ⚠

# JSON heat-map for a custom dashboard
import json
data = json.loads(rpt.to_json())
for row in data["rows"]:
    if row["flake"] >= 0.2:
        print(f"Seed {row['seed']}: {row['fail']} failures")

GitHub Step Summary 

Write a markdown table to $GITHUB_STEP_SUMMARY:

import os
from ucis.ncdb.testplan_export import export_summary_markdown

md = export_summary_markdown(results, stage_gate=gate)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
    f.write(md)

The output includes a stage roll-up table, per-testpoint status rows, and (when stage_gate is supplied) a gate verdict with a list of blocking testpoints.

API reference 

ucis.ncdb.reports.report_testpoint_closure(results: List[TestpointResult]) → ClosureSummary

Compute a closure summary from testpoint results.

Parameters:: results – List of TestpointResult objects (output of compute_closure()).
Returns:: ClosureSummary with per-stage roll-up and totals.

ucis.ncdb.reports.format_testpoint_closure(summary: ClosureSummary, *, show_all: bool = False) → str

Render a ClosureSummary as a terminal table.

Parameters:

summary – Output of report_testpoint_closure().
show_all – If False (default), skip N/A and UNIMPLEMENTED rows.

Returns:

Human-readable multiline string.

class ucis.ncdb.reports.ClosureSummary(results: List[TestpointResult], by_stage: Dict[str, Dict], total_closed: int, total_na: int, total: int)

Result of report_testpoint_closure().

Parameters:

results – Per-testpoint closure results.
by_stage – Stage-level roll-up: stage → {closed, total, pct}.
total_closed – Number of testpoints with status CLOSED.
total_na – Number of testpoints with status N/A.
total – Total testpoint count.

ucis.ncdb.reports.report_stage_gate(results: List[TestpointResult], stage: str, testplan: Testplan, require_flake_score_below: float | None = None, require_coverage_pct: float | None = None) → StageGateReport

Evaluate a stage gate (go/no-go for advancing to next stage).

Parameters:

results – Output of compute_closure().
stage – Target stage to evaluate ("V1", "V2", etc.).
testplan – The Testplan being evaluated.
require_flake_score_below – Optional flake threshold (0–1).
require_coverage_pct – Optional minimum coverage percentage.

Returns:

StageGateReport.

ucis.ncdb.reports.format_stage_gate(report: StageGateReport) → str

Render a StageGateReport as a terminal summary.

Parameters:: report – Output of report_stage_gate().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.StageGateReport(stage: str, passed: bool, blocking: List[TestpointResult], message: str, gate_detail: dict)

Result of report_stage_gate().

Parameters:

stage – Target stage (e.g. "V2").
passed – Whether the gate passes.
blocking – Testpoints that are not yet CLOSED (and not N/A).
message – Human-readable verdict line.
gate_detail – Raw detail dict from stage_gate_status().

ucis.ncdb.reports.report_regression_delta(results_new: List[TestpointResult], results_old: List[TestpointResult]) → RegressionDelta

Compute the testplan closure delta between two regression runs.

Parameters:

results_new – Closure results for the current regression.
results_old – Closure results for the baseline regression.

Returns:

RegressionDelta.

ucis.ncdb.reports.format_regression_delta(report: RegressionDelta) → str

Render a RegressionDelta as a terminal summary.

Parameters:: report – Output of report_regression_delta().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.RegressionDelta(newly_closed: List[TestpointResult], newly_failing: List[TestpointResult], unchanged_open: List[TestpointResult], summary: str)

Delta between two closure result sets.

Parameters:

newly_closed – Testpoints that moved to CLOSED.
newly_failing – Testpoints that were not FAILING but are now.
unchanged_open – Testpoints that remain open/partial.
summary – One-line summary string.

ucis.ncdb.reports.report_testpoint_reliability(results: List[TestpointResult], db, flaky_threshold: float = 0.2) → TestpointReliability

Compute per-testpoint flake scores from v2 test_stats.

Parameters:

results – Output of compute_closure().
db – An open NcdbUCIS instance.
flaky_threshold – Flake score above which a testpoint is flagged.

Returns:

TestpointReliability.

ucis.ncdb.reports.format_testpoint_reliability(report: TestpointReliability) → str

Render a TestpointReliability as a terminal table.

Parameters:: report – Output of report_testpoint_reliability().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.TestpointReliability(rows: List[Tuple[str, float, int, int]], flaky_threshold: float = 0.2)

Per-testpoint flake scores.

Parameters:

rows – List of (testpoint_name, flake_score, pass_count, fail_count). Sorted by flake_score descending.
flaky_threshold – Score above which a testpoint is considered flaky.

ucis.ncdb.reports.report_unexercised_covergroups(db, testplan: Testplan, low_threshold: float = 50.0) → UnexercisedCovergroups

Identify covergroups with zero or low coverage.

Parameters:

db – An open UCIS database.
testplan – The active testplan (used to filter to plan-tracked groups).
low_threshold – Percentage below which a covergroup is flagged as low-hit (default 50%).

Returns:

UnexercisedCovergroups.

ucis.ncdb.reports.format_unexercised_covergroups(report: UnexercisedCovergroups) → str

Render an UnexercisedCovergroups report as terminal text.

Parameters:: report – Output of report_unexercised_covergroups().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.UnexercisedCovergroups(zero_hit: List[str], low_hit: List[Tuple[str, float]], threshold: float = 50.0)

Covergroups with zero hits.

Parameters:

zero_hit – List of covergroup names with 0% coverage.
low_hit – List of (name, pct) tuples with 0 < pct < threshold.
threshold – Low-hit threshold used.

ucis.ncdb.reports.report_coverage_contribution(db) → CoverageContribution

Report per-test unique coverage bin contribution from v2 contrib data.

Parameters:: db – An open NcdbUCIS instance.
Returns:: CoverageContribution.

ucis.ncdb.reports.format_coverage_contribution(report: CoverageContribution) → str

Render a CoverageContribution as a terminal table.

Parameters:: report – Output of report_coverage_contribution().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.CoverageContribution(rows: List[Tuple[str, int, int]], total_bins: int)

Per-test unique bin contribution.

Parameters:

rows – List of (test_name, unique_bins, total_hits) sorted by unique_bins descending.
total_bins – Total covered bins in the database.

ucis.ncdb.reports.report_test_budget(testplan: Testplan, db) → TestBudget

Estimate CPU-hour budget per stage from v2 test_stats mean CPU times.

For each testpoint the mean CPU time of all its mapped tests is summed. Testpoints with no CPU stats are listed in missing_stats.

Parameters:

testplan – The active Testplan.
db – An open NcdbUCIS instance.

Returns:

TestBudget.

ucis.ncdb.reports.format_test_budget(report: TestBudget) → str

Render a TestBudget as a terminal table.

Parameters:: report – Output of report_test_budget().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.TestBudget(rows: List[Tuple[str, str, float, int]], stage_totals: Dict[str, float], missing_stats: List[str])

CPU time budget by stage.

Parameters:

rows – List of (stage, testpoint_name, mean_cpu_sec, total_runs) sorted by stage rank then mean_cpu_sec descending.
stage_totals – Mapping of stage → total estimated CPU seconds.
missing_stats – Testpoint names for which no CPU stats are available.

ucis.ncdb.reports.report_safety_matrix(results: List[TestpointResult], waivers=None) → SafetyMatrix

Build a requirement to testpoint traceability matrix.

Parameters:

results – Output of compute_closure().
waivers – Optional WaiverSet.

Returns:

SafetyMatrix.

ucis.ncdb.reports.format_safety_matrix(report: SafetyMatrix) → str

Render a SafetyMatrix as a text table.

Parameters:: report – Output of report_safety_matrix().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.SafetyMatrix(rows: List[Tuple[str, str, str, str, bool]], untested_requirements: List[str])

Requirement x testpoint traceability matrix with waiver flags.

Parameters:

rows – List of (req_id, req_desc, testpoint_name, status, waived).
untested_requirements – Requirement IDs with no linked testpoints.

ucis.ncdb.reports.report_seed_reliability(db, test_name: str) → SeedReliability

Compute per-seed pass/fail counts from v2 history buckets.

Parameters:

db – An open NcdbUCIS instance.
test_name – Name of the test to analyse.

Returns:

SeedReliability.

ucis.ncdb.reports.format_seed_reliability(report: SeedReliability) → str

Render a SeedReliability as a terminal heat-map table.

Parameters:: report – Output of report_seed_reliability().
Returns:: Human-readable multiline string.

class ucis.ncdb.reports.SeedReliability(test_name: str, rows: List[Tuple[int, int, int, float]], total_seeds: int)

Per-seed pass/fail counts for a given test name.

Parameters:

test_name – The queried test name.
rows – List of (seed_id, pass_count, fail_count, flake_score) sorted by fail_count descending.
total_seeds – Total unique seeds seen.

ucis.ncdb.testplan_export.export_junit_xml(results: List[TestpointResult], output_path: str, suite_name: str = 'testplan_closure') → None

Write closure results as a JUnit XML file.

Each testpoint becomes a <testcase>. Testpoints with status FAILING or PARTIAL get a <failure> element; NOT_RUN gets a <skipped> element; CLOSED is a plain pass.

Parameters:

results – Output of compute_closure().
output_path – Destination .xml file path.
suite_name – Value of the name attribute on <testsuite>.

Example:

from ucis.ncdb.testplan_export import export_junit_xml
export_junit_xml(results, "closure_results.xml")

ucis.ncdb.testplan_export.export_github_annotations(results: List[TestpointResult], file: str = 'testplan', *, output=None) → None

Write GitHub Actions workflow command annotations to output.

FAILING testpoints emit ::error:: lines; PARTIAL and NOT_RUN emit ::warning:: lines. CLOSED and N/A produce no output.

Parameters:

results – Output of compute_closure().
file – Value used in the file= annotation field (defaults to "testplan").
output – File-like object to write to (defaults to sys.stdout).

Example:

from ucis.ncdb.testplan_export import export_github_annotations
export_github_annotations(results)  # writes to stdout

ucis.ncdb.testplan_export.export_summary_markdown(results: List[TestpointResult], stage_gate: StageGateReport | None = None, history_db=None) → str

Generate a GitHub Actions Job Summary–compatible markdown string.

Parameters:

results – Output of compute_closure().
stage_gate – Optional StageGateReport to include a gate verdict section.
history_db – Unused; reserved for future trend lines.

Returns:

A markdown string suitable for appending to $GITHUB_STEP_SUMMARY.

Example:

from ucis.ncdb.testplan_export import export_summary_markdown
md = export_summary_markdown(results, stage_gate=gate)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
    f.write(md)