Program to extract constants from C headers and integrate that as FFI SharedPools
[](https://travis-ci.org/marianopeck/FFICHeaderExtractor)
====
# FFICHeaderExtractor
In short, FFICHeaderExtractor is a program to extract information (e.g. constants) from `C` headers and integrate that into Smalltalk `SharedPools`.
When we use FFI to call a certain library, it's quite common that we need to pass specific constants (for example, `SIGKILL` to `kill()`). These constants are defined in `C` header files and can even change their values in different paltforms. Sometimes, these constants also are defined by the `C` preprocessor and so there is not way to get those values from FFI. If you don't have the value of those constants, you cannot make the FFI call. In other words, if I cannot know that the value of `SIGKILL` is `9`, how do I call `kill()` from FFI?
This tool allows the developers of a FFI tool (any project which uses FFI to call a certain library), to automatically create a `C` program that writes all the user-defined constants values, compile it, run it, and create a Smalltalk init method which initializes the shared pool constants based on `C` program output. This autogenerated init method can then be distributed with the rest of the FFI tool. FFICHeaderExtractor will also automatically initialize (searching and executing the previously autogenerated init method for the current platform) a `SharedPool` upon system startup.
##Table of Contents
* [FFICHeaderExtractor](#fficheaderextractor)
* [Table of Contents](#table-of-contents)
* [Installation](#installation)
* [Getting Started](#getting-started)
* [Defining and using a FFISharedPool with FFI](#defining-and-using-a-ffisharedpool-with-ffi)
* [Extracting and storing constants information](#extracting-and-storing-constants-information)
* [Initializing variables from autogenerated method](#initializing-variables-from-autogenerated-method)
* [Deploying the FFI tool with autogenerated methods](#deploying-the-ffi-tool-with-autogenerated-methods)
* [Running the tests](#running-the-tests)
* [Contributing](#contributing)
* [History](#history)
* [Future work](#future-work)
* [Authors](#authors)
* [License](#license)
* [Acknowledgments](#acknowledgments)
* [Funding](#funding)
## Installation
**FFICHeaderExtractor currently only works in Pharo 5.0 with Spur VM**. Until Pharo 5.0 is released, we recommend to always grab a latest image and VM. You can do that in via command line:
```bash
wget -O- get.pharo.org/alpha+vm | bash
```
Then, from within Pharo, execute the following to install FFICHeaderExtractor:
```Smalltalk
Metacello new
configuration: 'FFICHeaderExtractor';
repository: 'github://marianopeck/FFICHeaderExtractor:master/repository';
version: #stable;
load.
```
Besides the above installation instructions, FFICHeaderExtractor can also be installed from the `Catalog Browser`, already present in Pharo. Just open it, search for FFICHeaderExtractor, then right click, `Install stable version`.
> Important: so far FFICHeaderExtractor works only in OSX and Unix.
## Getting Started
The user of this tool will be a developer of a FFI-based project. As an example, let's say we want to call some functions from the `libc` (the standard C library) via FFI. `libc` is huge and it has lots of functions and constants. For our example we will take only a small portion of it.
### Defining and using a FFISharedPool with FFI
The first step is to define your own subclass of `FFISharedPool` and define all the constants you want to retrieve their values, as class variables:
```Smalltalk
FFISharedPool subclass: #LibCSharedPool
instanceVariableNames: ''
classVariableNames: 'EINVAL EPERM SIGCHLD SIGCONT SIGINFO SIGKILL SIGPOLL SIGSTOP SIGTERM'
package: 'LibC'
```
In this example, we pick up only some signals and some errors. Let's now see the user of this `SharedPool`:
```Smalltalk
Object subclass: #LibCWrapper
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'LibCSharedPool'
package: 'LibC'
```
The imporant line there is `poolDictionaries: 'LibCSharedPool'` were we declare that we want to use `LibCSharedPool`. The `LibCWrapper` is then able to do this:
```Smalltalk
LibCWrapper >> stopProcess: aPidNumber
self primitiveKill: aPidNumber signal: SIGSTOP.
```
```Smalltalk
LibCWrapper >> continueProcess: aPidNumber
self primitiveKill: aPidNumber signal: SIGCONT.
```
`SIGSTOP` and `SIGCONT` are signals which, via `kill()`, can pause a process with it's current state and then resume it's execution later. The details of `primitiveKill:signal:` is not important for this example as it is the simple FFI call to `int kill(pid_t pid, int sig)`.
The important part here is that thanks to `SharedPools`, the `LibCSharedPool` code can access directly to `SIGSTOP` and `SIGCONT` as if they were class variables. However, these are the class variables defined in `LibCWrapper`. For above code to really work, the class variables of `LibCSharedPool` must be initialized with the correct values. If we were using `SIGKILL` or `SIGTERM` for example, it's very likely that the values of them are `9` and `15` in almost every possible platform. But that's not the case of `SIGSTOP` and `SIGCONT`. Do you know their values? Are you sure they do not change among platforms? As we will read later, these constants **do change** between platforms.
This problematic is part of what FFICHeaderExtractor solves. Let's see how.
### Extracting and storing constants information
Once you have defined your own `FFISharedPool` subclass and the constants for which you would like to get the values, the next step is to give more information to FFICHeaderExtractor so that it can extract the definitions from `C`.
The created `FFISharedPool` subclass, must implement the class side method `#headersContainingVariables` which answers an array with the `C` header names that define all the defined constants. In the example of `LibCSharedPool` we have singal constants (defined in `signal.h`) and error constants (defined in `errno.h`). Therefore:
```Smalltalk
LibCSharedPool class >> headersContainingVariables
^ #( 'errno.h' 'signal.h' )
```
<!-- TO-DO: put sec ref -->
> For many cases, this is all the needed information. We will later see cases where more definitions are needed.
The last step for the developer of the FFI tool (`LibCWrapper` in our example) is to trigger the extraction itself. The way of doing this is:
```Smalltalk
LibCSharedPool extractAndStoreHeadersInformation
```
<!-- TO-DO: put sec ref -->
As explained later, to be able to run `#extractAndStoreHeadersInformation` the user must have installed and reached by `$PATH`, `cc` (LLVM Clang) in OSX and `gcc` in Unix.
The method `#extractAndStoreHeadersInformation` first extracts all the constants values (defined in C header files) **from the current running platform** and then creates a Smalltalk init method which is then compiled into the SharedPool. So, for example, if I execute such a method in OSX with a Pharo 32 bits VM, then the resulting autogenerated method is this:
```Smalltalk
LibCSharedPool class >> initVariablesMacOS32
"Method automatically generated by FFICHeaderExtractor. Read more at https://github.com/marianopeck/FFICHeaderExtractor"
<platformName: 'Mac OS' wordSize: 4>
EINVAL := 22.
EPERM := 1.
SIGCHLD := 20.
SIGCONT := 19.
SIGINFO := 29.
SIGKILL := 9.
SIGPOLL := nil."SIGPOLL is UNDEFINED for this platform"
SIGSTOP := 17.
SIGTERM := 15.
```
Note that the selector of the autogenerated method is `#initVariablesMacOS32` which matches the platform used to run `#extractAndStoreHeadersInformation`. If later we have Pharo 64 bits VM, then we should also run `#extractAndStoreHeadersInformation` with such a VM. Here is the example when running in Linux 32 bits:
```Smalltalk
initVariablesunix32
"Method automatically generated by FFICHeaderExtractor. Read more at https://github.com/marianopeck/FFICHeaderExtractor"
<platformName: 'unix' wordSize: 4>
EINVAL := 22.
EPERM := 1.
SIGCHLD := 17.
SIGCONT := 18.
SIGINFO := nil."SIGINFO is UNDEFINED for this platform"
SIGKILL := 9.
SIGPOLL := 29.
SIGSTOP := 19.
SIGTERM := 15.
```
If you compare `#initVariablesunix32` and `initVariablesMacOS32` you can get some interesting conclusions:
* Some constants do not have the same value in OSX and in Linux. In fact, the ones we use in this example (`SIGSTOP` and `SIGCONT`) are different. So, without FFICHeaderExtractor this example would have been difficult to implement.
* Not all constants are defined in all OS. For example, `SIGINFO` is defined in OSX but not in Linux, and `SIGPOLL` the other way around.
The information embebed as part of the selectors of the methods is just for organization. When FFICHeaderExtractor needs to search the correct method for the current platform, it uses the `platformName` and `wordSize` of the pragma of the autogenerated method, not the selector itself.
The developer of the FFI tool is responsible of running `#extractAndStoreHeadersInformation` for every platform he needs. Ideally, the developer could have a CI (continuous integration) server that automatically runs this for every platform.
### Initializing variables from autogenerated method
Upon Pharo startup, `FFISharedPool` analyses each subclass to see which ones require the initialization of the class variables. When the initialization of class variables does happen, then the used `platformName` and `wordSize` are stored in the SharedPool. This way, next system startup we can check if the platform has changed compared to what we have stored (like running the image in another OS). If the platform has changed or if the class variables were never initialized before, then `FFISharedPool` will try to do the initialization by sending the message `#initializeVariables`.
`#initializeVariables` simply searches an autogenerated method whose