README excerpt
# Pharo Bench Tester
This project packages a Pharo-based benchmark runner in a Docker image. It takes a JSONL file of LLM completions, converts each task into Tonel packages, runs the generated Pharo tests, and writes a JSONL results report. The container uses Pharo 13 and the `test_runner.py` as the entrypoint.
**Note:** `libLargeIntegers.so` in this repository is a fixed version of the original Pharo library. The original contained a missing NULL check bug; this version includes the fix.
## Docker Usage
You can use the `../5-benchmark-evaluation/5_9_run_tests.sh` script to build the image and run the benchmarks.
Otherwise you first need to copy the tree-sitter-pharo repository into this repository:
```bash
cp -r ../tree-sitter-pharo ./tree-sitter-pharo
```
Build the image:
```bash
docker build -t pharo-bench-tester .
```
Prepare input/output files on the host and run:
```bash
mkdir -p ./data
# Put your input JSONL at ./data/completions.jsonl
docker run --rm -it -v "$(pwd)/data:/data" pharo-bench-tester \
--input /data/completions.jsonl \
--output /data/results.jsonl
```
The results will be written to `./data/results.jsonl` on the host.
To speed up execution, you can enable parallel processing (default is CPU count minus one), which splits the tasks into multiple chunks, runs them concurrently on separate Pharo VMs, and merges the results at the end
To keep the temporary per-chunk Pharo working directories for inspection, set `BENCH_KEEP_DIR` to a host-mounted path:
```bash
docker run --rm -it -v "$(pwd)/data:/data" -e BENCH_KEEP_DIR="/data/tmp" pharo-bench-tester \
--input /data/completions.jsonl \
--output /data/results.jsonl
```
### CLI Options
`test_runner.py` is the container entrypoint. It accepts:
- `--input`, `-i`: Path to input JSONL inside the container.
- `--output`, `-o`: Path to output JSONL inside the container.
- `--parallel`: Number of parallel processes (default: CPU count minus one).
- `--process-log`: Path to log file inside the container.
- `--timeout`: Default test timeout in seconds.
### Input JSONL Format
Each line should be a JSON object with at least these fields:
- `task_id`: Task identifier using a `*-x-y` pattern (e.g. `example-task-0-1`), where x is the task ID and y the mask ID.
- `task_name`: Human-readable task name.
- `prefix`: Pharo method signature plus prefix body.
- `suffix`: Suffix appended after the completion.
- `tests`: Pharo test method source used to validate completions.
- `completions`: Array of completion strings.