README excerpt
# Soil for the VAST Platform
**A VAST Platform (VA Smalltalk) port of [Soil](https://github.com/ApptiveGrid/Soil)**,
the transactional object database for Pharo by ApptiveGrid.
> ⚠️ **This is not the Pharo Soil.** It is an independent port targeting
> **Instantiations' VAST Platform**. The code, packaging, on-disk format, and
> several implementation details differ from upstream — see
> [Porting changes](#porting-changes-vast-vs-pharo). It loads with VAST's
> `TonelLoader` (not Metacello) and has no Pharo/Zinc/FileSystem dependencies.
Soil stores object graphs directly (no ORM, no SQL) in a simple, inspectable
on-disk format. This port brings Soil's **core object store** to VAST.
## Status
**The core object store works and is tested on VAST 2026.** You can create a
database, store an object graph in a transaction, commit, reopen, and read it
back — including large graphs with cycles, shared references, and lazily-loaded
sub-clusters.
### What works
- **Serialization** of the common Smalltalk types: `nil`, `true`/`false`,
`SmallInteger` / `LargePositiveInteger` / `LargeNegativeInteger`, `Float`
(bit-exact), `Character`, `String`, `Symbol`, `ByteArray`, `Array`,
`OrderedCollection`, `Dictionary`, `IdentityDictionary`, `Association`,
`Class`, and generic fixed-instance-variable objects — with **object identity**
preserved across cycles and shared references.
- **Persistent storage**: append-only object files + index files organised into
segments, with 8-byte object ids (`segment:index`).
- **Transactions**: `createOnPath:` / `openOnPath:` / `newTransaction` / `root:` /
`root` / `commit`.
- **Multi-cluster graphs with lazy loading**: objects made *roots* are stored as
separate clusters and returned as lazy proxies that resolve on demand.
- **`SoilPersistentDictionary`**: keys stored inline, values as separate,
lazily-loaded roots — the typical Soil database root.
Every source file ships a self-test; e.g. `SoilStoreTest run` and
`SoilPersistentDictionaryTest run` answer a report ending in `ALL PASS`.
### Not yet ported
- **Disk B-tree / skip-list indexes** (`SoilIndexedDictionary`) — sorted /
range-query indexes (~35 page-based classes, entangled with the journal). The
behaviour registry currently uses a linear meta-segment scan instead of the
skip-list.
- **Write-ahead journal**, checkpoints, and crash recovery.
- **Concurrency / file locking** — single-image use for now (CFS region-lock
hooks are stubbed; see [Porting changes](#porting-changes-vast-vs-pharo)).
- Schema migration, CompiledMethod/block (bytecode) serialization, Roassal
visualisation, garbage collection, and Rhizome distribution.
## Installation
This is a **Tonel** project. Clone it and load with VAST's `TonelLoader`:
```smalltalk
(TonelLoader readFromPath: 'path/to/Soil-in-VAST' asPath) loadAllApplications.
```
That loads the single `Soil` application. Its prerequisites — `CLDT`,
`CommonFileSystem`, and `CfsPaths` — are part of every standard VAST image.
The port ships self-tests; after loading, evaluate e.g. `SoilStoreTest run` or
`SoilPersistentDictionaryTest run` (each answers a report ending in `ALL PASS`).
### Layout
```
.project Tonel project descriptor (srcDirectory: 'source')
source/.properties tonel format marker
source/Soil/package.st the Soil application (name + prerequisites)
source/Soil/*.class.st classes
source/Soil/*.extension.st extensions to base classes (new soil-* selectors)
```
## Usage
```smalltalk
"Create a database and store an object graph"
db := SoilDatabase createOnPath: 'mydb'.
txn := db newTransaction.
txn root: myModelRoot.
txn commit.
db close.
"Reopen and read it back"
db := SoilDatabase openOnPath: 'mydb'.
myModelRoot := db newTransaction root.
```
`Soil createOnPath:` / `Soil openOnPath:` also work — see the naming note below.
### Persistent dictionary (lazy-loading root)
```smalltalk
db := SoilDatabase createOnPath: 'company'.
txn := db newTransaction.
people := SoilPersistentDictionary new.
people at: #alice put: (Person name: 'Alice'). "each value becomes its own cluster"
people at: #bob put: (Person name: 'Bob').
txn root: people.
txn commit.
db close.
db := SoilDatabase openOnPath: 'company'.
people := db newTransaction root. "keys are loaded; values are not"
(people at: #alice) name. "-> 'Alice' (this value loads on demand)"
```
## Porting changes (VAST vs Pharo)
This section documents where and why the VAST port diverges from upstream Soil.
### Packaging & naming
- **Tonel format**, loaded with VAST's `TonelLoader` (not Metacello). The whole
port is a single VAST application (`Soil`), with proper VAST prerequisites
(`CLDT`, `CommonFileSystem`, `CfsPaths`).
- **Main class is `SoilDatabase`, not `Soil`.** In VAST the `Soil` global is the
ENVY *application* (a class whose superclass is `Application`), so a domain
class can't take that name. Class-side redirects on the `Soil` application
forward the documented `Soil createOnPath:` / `Soil openOnPath:` entry points
to `SoilDatabase`.
- **No kernel-class methods are overwritten.** Extensions to core classes use
new, `soil`-prefixed selectors (e.g. `ByteArray>>soilAsInteger`, not
`asInteger`). Overwriting an existing base-class method in VAST requires a
scratch edition of the owning system application (CLDT); adding a *new*
selector does not, so the port sticks to new selectors.
- **`new` does not auto-send `#initialize` in VAST.** Classes that need setup
define an explicit class-side `new` (`^super new initialize`), rather than
relying on Pharo's `Object class>>new`.
### Dependencies replaced
- **No Zinc.** Soil's only `Zn*` usage was buffered byte streams and a UTF-8
encoder (not HTTP). Replaced with:
- `SoilLockableStream` — a binary stream built directly on VAST's
`CfsFileDescriptor` (`open:oflag:`, `lseek:whence:`, `read:/write:startingAt:nbyte:`,
`flush`), instead of `ZnBufferedReadWriteStream`.
- `SoilReadStream` / `SoilWriteStream` — small in-memory byte streams
(avoids relying on VAST `ReadStream`/`WriteStream` behaviour for binary data).
- `SoilUtf8Codec` — a UTF-8 encoder/decoder, instead of `ZnCharacterEncoder`.
- **No Pharo FileSystem.** `SoilFileReference` is a minimal facade over VAST's
CFS (`CfsDirectoryDescriptor mkdir:`/`opendir:pattern:mode:`, `CfsStat`,
`CfsFileDescriptor remove:`) covering the subset of `FileReference` Soil uses.
### Serializer / materializer
- **VAST reflection instead of Pharo's metaobject model.** Pharo Soil dispatches
serialization through the class *layout* objects (`FixedLayout`, `PointerLayout`,
`VariableLayout`, …) and `Slot`s. This port uses plain VAST reflection —
`class allInstVarNames`, `instVarAt:` / `instVarAt:put:`, `basicNew` — and an
explicit type-code dispatch in the materializer (no layout classes, no
block/class dispatch table).
- **Type codes are class variables** on `SoilObjectCodec`, not a Pharo
`SharedPool`.
- **`Float` is stored as sign + integer mantissa + power-of-two exponent**
(extracted with basic float arithmetic), because VAST `Float` has no
`timesTwoPower:`. This is bit-exact but **not binary-compatible** with Pharo
Soil's float encoding — and, more generally, **the on-disk format is not meant
to be read by Pharo Soil**. The port is free to use VAST-friendly encodings.
- Transient instance variables are honoured via `soilTransientInstVars` /
`soilPersistentInstVars` (used by `SoilPersistentDictionary` to avoid
serialising its back-reference to the transaction).
### Multi-cluster & proxies
- **Proxies subclass `Object`** — VAST has no `ProtoObject`. A proxy forwards
unknown messages via `doesNotUnderstand:` → `aMessage sendTo: self soilRealObject`.
- **Proxies are not fully transparent.** Because a proxy inherits `Object`
methods, selectors that `Object` already implements (e.g. `name`) are answered
by the proxy rather than forwarded. To keep behaviour predictable,
`SoilPersistentDictionary>>at:` **resolves** the proxy to the real object on
access (still lazy — only accessed keys load). `rawAt:` returns the unresolved
proxy.
- **Simplified commit.** A commit serialises every cluster record (discovering
new roots along the way), allocates object ids, and writes them with `fsync`
— there is no write-ahead journal / checkpoint / recovery yet.
### VAST API notes encountered
- VAST `String` has no `asInteger` (use `asNumber`) and no `endsWith:` (use
`includesSubString:`).
- On reopen, segment file streams must be explicitly opened before the file
helpers touch their `stream` ivar.
## Credits & licence
A port of [Soil](https://github.com/ApptiveGrid/Soil) by ApptiveGrid. Original
design and code © the Soil authors; this VAST port maintains the upstream
licence.