crashtest.hxsmith.runner
Run many programs in parallel worker processes and summarise where they land on the ladder.
A run is a list of Jobs from any source: the random generator (fuzz), a corpus
(crashtest.hxcorpus), or a file. execute_jobs is the one place that spawns workers,
enforces the time limit, keeps artifacts and aggregates a Report.
class Report
results: List[Result]
results = field(default_factory=list)
valid: List[Result]
invalid: List[Result]
histogram
def histogram() -> Dict[Quality, int]
clusters
def clusters() -> List[Tuple[Quality, str, List[str]]]
Failures (below BEHAVES) grouped by signature, biggest first: the fix queue.
fix_order
def fix_order() -> List[Tuple[str, int, int]]
Compile errors in the order to fix them: (kind, programs with it, programs it clears).
Greedy: each step takes the kind that makes the most programs compile-clean given the kinds already chosen (ties and dead ends fall back to the most frequent), so the first entries are the fixes that unblock the most programs, not just the most common message.
by_feature
def by_feature() -> List[Tuple[str, int, float, float]]
Per feature: programs using it, their mean rung, and the share below BEHAVES. A feature with a low share-good is where the decompiler is weakest.
mean_quality
def mean_quality() -> float
render
def render(top: int = 12) -> str
from_json
classmethod
@classmethod
def from_json(data: dict) -> 'Report'
to_json
def to_json() -> dict
init
def __init__(results: List[Result] = list()) -> None
execute_jobs
def execute_jobs(makers: Sequence[Callable[[], Result]], jobs: int, out: Path, progress: Optional[Callable[[int, int, Result], None]] = None) -> Report
Run the callables in a thread pool (each drives a worker process) and write report.json.
fuzz
def fuzz(seeds: Iterable[int], only: Optional[Iterable[str]] = None, exclude: Iterable[str] = (), jobs: int = 0, out: Optional[Path] = None, config: Optional[Config] = None, keep_above: Quality = Quality.BEHAVES, swarm: float = 0.0, progress: Optional[Callable[[int, int, Result], None]] = None) -> Tuple[Report, Path]
Generate a program per seed and run each.
keeper
def keeper(keep_above: Quality) -> Callable[[Result], bool]
Keep artifacts of every program that did not reach keep_above, and of every invalid one.
lint
def lint(seeds: Iterable[int], only: Optional[Iterable[str]] = None, exclude: Iterable[str] = (), jobs: int = 0, config: Optional[Config] = None, swarm: float = 0.0) -> List[Tuple[int, str]]
Check that the generator's programs are valid tests, without decompiling anything: each must compile, run, finish and repeat its own output. Returns (seed, problem) for the ones that do not. This is the loop for developing a feature.
lint_jobs
def lint_jobs(items: Iterable[Job], jobs: int = 0) -> List[Tuple[str, str]]
lint for arbitrary jobs: (name, problem) for each that is not a valid test.
run_job
def run_job(job: Job, out: Path, keep: Callable[[Result], bool], timeout: float = WORKER_TIMEOUT_S) -> Result
Evaluate one job in a worker. The artifact folder survives only if keep(result).
run_jobs
def run_jobs(items: Iterable[Job], jobs: int = 0, out: Optional[Path] = None, keep_above: Quality = Quality.BEHAVES, progress: Optional[Callable[[int, int, Result], None]] = None) -> Tuple[Report, Path]
run_worker
def run_worker(workdir: Path, timeout: float = WORKER_TIMEOUT_S) -> Optional[str]
Run the pipeline for the job in workdir in its own process group. Returns why it died, if it did.
RUNS
RUNS = Path(__file__).resolve().parent.parent / 'hxsmith-runs'
WORKER_TIMEOUT_S
WORKER_TIMEOUT_S = 180.0