ingle -> single

.gitignore

@@ -22,6 +22,7 @@ src/compiler/target
 src/runtime/Config
 src/runtime/TAGS
 src/runtime/genesis
+src/runtime/linkage-table-prelink-info.c
 src/runtime/openbsd-sigcontext.h
 src/runtime/sbcl
 src/runtime/sbcl.exe

README.static-executable

@@ -0,0 +1,250 @@
+This branch of SBCL is maintained by Eric Timmons (@daewok) and contains a set
+of patches necessary to build a completely static executable with SBCL. Such an
+executable has all necessary foreign libraries statically linked into the
+runtime and has no support for dynamic loading and unloading of
+libraries. While the lack of dynamic loading support is certainly constraining,
+the benefit of building an executable this way is it requires no libraries to
+be installed by the user of the executable. This makes it ideal for archival
+purposes, distributing executables to a non-technical audience, distributing an
+executable where you must know the exact versions of foreign libraries used at
+runtime, or distributing executables that Just Work^TM (like many executables
+written in golang).
+
+While other solutions exist to statically link foreign libraries into the SBCL
+runtime, to the best of my knowledge there has been no publicly advertised
+method of building SBCL with libc statically linked. The lack of static linking
+for libc means that the user of the executable must have a compatible libc
+installed. Unfortunately, the most commonly used libc in the Linux world
+(glibc) is frequently not backward compatible with itself. For evidence of
+this, see the fact that SBCLs built on Debian Buster (like the official
+releases since 1.5.6) do not run on Debian Stretch.
+
+Unfortunately, glibc doesn't even really support static linking at
+all. Therefore, I recommend that static SBCL executables be built with musl
+libc. Musl is designed with static linking in mind and is broadly compatible
+with most libraries that don't do tricksy things with libc. And if you find a
+library not compatible with musl libc, it seems most maintainers are welcoming
+to patches that add support.
+
+Alpine Linux is a great OS for building statically linked executables as it
+uses musl libc by default. I further recommend using Docker for building static
+executables so that you don't need to maintain a separate Alpine install. Plus,
+you can use the clfoundation/sbcl:alpine3.13 image as a starting point.
+
+THEORY
+
+The biggest issue with creating a static executable is ensuring that foreign
+symbols are accessible from the Lisp core. In normal, dynamic use, SBCL uses
+dlsym to look up the address of symbols and stores them in a vector in foreign
+memory called the "linkage table". The lisp core then maintains a hash table
+mapping foreign symbol names to their index in the linkage table. This is
+called the linkage info.
+
+In a static executable, we cannot count on having a working dlsym, even if
+libdl is linked into the runtime. When performing static linkage, musl libc
+replaces all libdl functions with stubs that simply return errors. Therefore,
+we have to use the system linker to resolve the references for us. But in order
+to have the linker do that for us, we need to know at link time which foreign
+symbols our lisp code will want to use!
+
+EXTRACTING LINKAGE INFO
+
+There are two approaches described below to generate a static executable. Both
+of them require a file describing the desired linkage info. While you could
+generate this by hand, it is easiest to extract it from a core.
+
+In order to extract the linkage info from a running core, use
+tools-for-build/dump-linkage-info.lisp. After loading that into the core,
+evaluate (sb-dump-linkage-info:dump-to-file #p"/path/to/output.sexp"). It also
+takes an keyword argument :make-undefined, a list of symbol names to make
+undefined in the output. This is useful for approach two below.
+
+The sexp written to the output file is a single list of lists. Each sublist has
+three elements. The first is a string naming the symbol. The second is T if the
+symbol is entered into the linkage info as data (it is a foreign variable) and
+NIL otherwise (it is a foreign function). The third is T if the symbol is
+undefined and NIL otherwise. It is critical that undefined symbols be
+maintained for approach one below.
+
+The following two sections describe two approaches on how to generate a static
+executable, step-by-step. The demo static executable contains the sb-gmp
+contrib and runs its test quite when executed. It requires that the static
+libraries for libgmp and libz are installed on your system. There is some
+weirdness with how the tests are loaded. This is because the tests do not seem
+to work after being dumped: I have not yet figured out why this is.
+
+BUILDING A STATIC EXECUTABLE - APPROACH ONE
+
+This approach to building a static executable is preferred if you're you want
+to minimize the amount of time compiling C and Lisp code. It takes advantage of
+the fact that musl inserts stub functionality for libdl such that it can still
+be linked against.
+
+The general process for this approach is:
+
+0. Build SBCL with the :sb-prelink-linkage-table feature (:sb-linkable-runtime
+is also strongly recommended).
+
+1. Build a core containing the lisp code you want to package in the static
+executable.
+
+2. Dump the linkage info to a file.
+
+3. Dump the core to a file (with save-lisp-and-die).
+
+4. Generate a C file that contains the info needed to build the linkage table.
+
+5. Relink the runtime. This time statically *and* with the object file
+generated from the C file in step 4.
+
+6. Load the saved core into the new static runtime, dumping again with
+:executable t if desired.
+
+Some notes about this approach:
+
++ The build IDs of the dynamic runtime (used to generate the core in step 1)
+and the static runtime *must* match. The easiest way to achieve this is to
+install SBCL with the feature :sb-linkable-runtime. This installs sbcl.o (the
+SBCL runtime in a single object file) along with everything else.
+
++ No modifications must be made to the linkage info file generated in step 2
+and no symbols can be filtered out of it.
+
+Here is a step-by-step procedure to build the demo static executable using this
+approach.
+
+Step 0:
+
+    sh make.sh --fancy --with-sb-linkable-runtime --with-sb-prelink-linkage-table
+    sh install.sh
+
+Steps 1-3:
+
+    sbcl --non-interactive \
+         --no-sysinit --no-userinit \
+         --eval '(require :uiop)' \
+         --eval '(require :sb-gmp)' \
+         --eval '(require :sb-rt)' \
+         --eval '(defvar *sb-gmp-tests* (uiop:read-file-string "contrib/sb-gmp/tests.lisp"))' \
+         --load tools-for-build/dump-linkage-info.lisp \
+         --eval '(sb-dump-linkage-info:dump-to-file "/tmp/linkage-info.sexp")' \
+         --eval '(sb-ext:save-lisp-and-die "/tmp/sb-gmp-tester.core")'
+
+Step 4:
+
+    sbcl --no-sysinit --no-userinit \
+         --script tools-for-build/create-linkage-table-prelink-info-override.lisp \
+         /tmp/linkage-info.sexp \
+         /tmp/linkage-table-prelink-info-override.c
+
+Step 5:
+
+    # Get all the variables SBCL used to build defined in the current environment.
+    while read l; do
+      eval "${l%%=*}=\"${l#*=}\"";
+    done < /usr/local/lib/sbcl/sbcl.mk
+
+    $CC $CFLAGS -Wno-builtin-declaration-mismatch -o /tmp/linkage-table-prelink-info-override.o -c /tmp/linkage-table-prelink-info-override.c
+    $CC -no-pie -static $LINKFLAGS -o /tmp/static-sbcl /usr/local/lib/sbcl/$LIBSBCL /tmp/linkage-table-prelink-info-override.o -lgmp $LIBS
+
+Step 6:
+
+    /tmp/static-sbcl --core /tmp/sb-gmp-tester.core \
+                     --non-interactive \
+                     --no-sysinit --no-userinit \
+                     --eval '(sb-ext:save-lisp-and-die "/tmp/sb-gmp-tester" :executable t :toplevel (lambda () (uiop:load-from-string *sb-gmp-tests*) (sb-rt:do-tests) (exit)) :compression t)'
+
+
+Look at the dumped executable. You should see that it is a static executable.
+
+    ldd /tmp/sb-gmp-tester
+
+Test that it works!
+
+    /tmp/sb-gmp-tester
+
+BUILDING A STATIC EXECUTABLE - APPROACH TWO
+
+This approach results in an executable that is not linked with libdl at
+all. This makes it a little bit more "pure" than than the previous approach,
+but that comes at the cost of needing to fully recompile both the runtime and
+core after the necessary foreign symbols are determined.
+
+The general process for this approach is:
+
+1. Build a core containing the lisp code you want to package in the static
+executable.
+
+2. Dump the linkage info to a file.
+
+3. Recompile SBCL, passing in the linkage info during build.
+
+4. Rebuild your core with the new runtime and corresponding core.
+
+5. Dump with :executable t.
+
+Some notes about this approach:
+
++ The libdl symbols must be stripped out of the linkage info file generated in
+step 2. The easiest way to do this is pass sb-dump-linkage-info:*libdl-symbols*
+as the :make-undefined argument to dump-to-file.
+
++ Further modifications can be made to the linkage info file generated in step
+2. You can reorder the symbols at will. You can add new symbols. You probably
+don't want to remove any (besides libdl functions).
+
+Steps 1-2:
+
+    sh run-sbcl.sh --non-interactive \
+                   --no-sysinit --no-userinit \
+                   --eval '(require :uiop)' \
+                   --eval '(require :sb-gmp)' \
+                   --eval '(require :sb-rt)' \
+                   --eval '(defvar *sb-gmp-tests* (uiop:read-file-string "contrib/sb-gmp/tests.lisp"))' \
+                   --load tools-for-build/dump-linkage-info.lisp \
+                   --eval '(sb-dump-linkage-info:dump-to-file "/tmp/linkage-info.sexp" :remove-symbols sb-dump-linkage-info:*libdl-symbols*)'
+
+Step 3:
+
+    LDLIBS="-lgmp" LINKFLAGS="-no-pie -static" IGNORE_CONTRIB_FAILURES="yes" sh make.sh --extra-linkage-table-entries=/tmp/linkage-info.sexp --without-os-provides-dlopen --without-os-provides-dladdr --fancy
+
+Steps 4-5:
+
+    sh run-sbcl.sh --non-interactive \
+                   --no-sysinit --no-userinit \
+                   --eval '(require :uiop)' \
+                   --eval '(require :sb-gmp)' \
+                   --eval '(require :sb-rt)' \
+                   --eval '(defvar *sb-gmp-tests* (uiop:read-file-string "contrib/sb-gmp/tests.lisp"))' \
+                   --eval '(sb-ext:save-lisp-and-die "/tmp/sb-gmp-tester" :executable t :toplevel (lambda () (uiop:load-from-string *sb-gmp-tests*) (sb-rt:do-tests) (exit)) :compression t)'
+
+Look at the dumped executable. You should see that it is a static executable.
+
+    ldd /tmp/sb-gmp-tester
+
+Test that it works!
+
+    /tmp/sb-gmp-tester
+
+BUILDING A STATIC EXECUTABLE WITH DOCKER
+
+See the Dockerfile at tools-for-build/Dockerfile.static-executable-example for
+an example of how to build the demo executable using Docker and approach
+one. The benefit of Docker is that it is a cheap way to build with musl libc
+even if you use glibc locally.
+
+The following commands will build the demo executable inside docker and extract
+it from the image, placing it at /tmp/sb-gmp-tester on your local file
+system. The following commands also try to avoid polluting your Docker
+namespace by not tagging the image or naming the container used to extract the
+executable.
+
+    IMAGE_ID_FILE="$(mktemp)"
+    CONTAINER_ID_FILE="$(mktemp)"
+    rm "$CONTAINER_ID_FILE"
+    docker build --iidfile "$IMAGE_ID_FILE" -f tools-for-build/Dockerfile.static-executable-example .
+    docker create --cidfile "$CONTAINER_ID_FILE" "$(cat "$IMAGE_ID_FILE")"
+    docker cp "$(cat "$CONTAINER_ID_FILE"):/tmp/sb-gmp-tester" /tmp/sb-gmp-tester
+    docker rm "$(cat "$CONTAINER_ID_FILE")"
+    rm "$IMAGE_ID_FILE"
+    rm "$CONTAINER_ID_FILE"

contrib/sb-bsd-sockets/sb-bsd-sockets.asd

@@ -6,7 +6,7 @@
 ;;; 1003.1-2003 defines an alternative API, which is specified in the
 ;;; RFC to be thread-safe. If it seems to be available, use it.
 
-(when (sb-alien::find-dynamic-foreign-symbol-address "getaddrinfo")
+(when (sb-alien::find-foreign-symbol-address "getaddrinfo")
   (pushnew :sb-bsd-sockets-addrinfo *features*))
 
 (defsystem "sb-bsd-sockets"

contrib/sb-gmp/gmp.lisp

@@ -84,6 +84,7 @@
             #-(or win32 darwin) '("libgmp.so" "libgmp.so.10" "libgmp.so.3")
             #+darwin '("libgmp.dylib" "libgmp.10.dylib" "libgmp.3.dylib")
             #+win32 '("libgmp.dll" "libgmp-10.dll" "libgmp-3.dll"))
+      (sb-alien::find-foreign-symbol-address "__gmp_version")
       (warn "GMP not loaded.")))
 
 (defvar *gmp-features* nil)

make-config.sh

@@ -88,7 +88,10 @@ do
         ;;
       --without)
         WITHOUT_FEATURES="$WITHOUT_FEATURES :$optarg"
-	;;
+	    ;;
+      --extra-linkage-table-entries=)
+        $optarg_ok && SBCL_EXTRA_LINKAGE_TABLE_ENTRIES=$optarg
+        ;;
       --fancy)
         WITH_FEATURES="$WITH_FEATURES $FANCY_FEATURES"
         # Lower down we add :sb-thread for platforms where it can be built.
@@ -213,6 +216,17 @@ Options:
                   Transfer the files to/from directory /home/user/sbcl
                   on host-machine.
 
+  --extra-linkage-table-entries=<path> Specify extra C symbols to include in
+                                       the linkage table
+
+      Path to a file specifying symbols that must be included in the
+      SBCL linkage table. Useful for statically linking libraries
+      into the runtime and ensuring the linker does not remove them.
+      The file must contain a single list of two element lists. Each
+      sublist must have a string naming a C symbol as its first
+      element and NIL or T (if the symbol names a variable) as its
+      second element.
+
 EOF
   exit 1
 fi
@@ -254,6 +268,11 @@ find_gnumake
 
 ./generate-version.sh
 
+# Copy the extra linkage entries to output folder for Genesis to find.
+if [ -n "$SBCL_EXTRA_LINKAGE_TABLE_ENTRIES" ] && [ -f "$SBCL_EXTRA_LINKAGE_TABLE_ENTRIES" ]; then
+    cp "$SBCL_EXTRA_LINKAGE_TABLE_ENTRIES" output/extra-linkage-table-entries.lisp-expr
+fi
+
 # Now that we've done our option parsing and found various
 # dependencies, write them out to a file to be sourced by other
 # scripts.

make-genesis-2.lisp

@@ -24,7 +24,10 @@
          :core-file-name "output/cold-sbcl.core"
          ;; The map file is not needed by the system, but can be
          ;; very handy when debugging cold init problems.
-         :map-file-name "output/cold-sbcl.map")
+         :map-file-name "output/cold-sbcl.map"
+         :linkage-table-prefill-info-c-name "src/runtime/linkage-table-prelink-info.c"
+         :extra-linkage-table-entries (when (probe-file "output/extra-linkage-table-entries.lisp-expr")
+                                        (read-from-file "output/extra-linkage-table-entries.lisp-expr")))
 #+cmu (ext:quit)
 #+clisp (ext:quit)
 #+abcl (ext:quit)

make-host-1.sh

@@ -32,3 +32,16 @@ export LANG LC_ALL
 # environment.
 echo //building cross-compiler, and doing first genesis
 echo '(load "loader.lisp") (load-sbcl-file "make-host-1.lisp")' | $SBCL_XC_HOST
+
+# Use a little C program to grab stuff from the C header files and
+# smash it into Lisp source code.
+$GNUMAKE -C src/runtime clean
+$GNUMAKE -C src/runtime sbcl.h
+$GNUMAKE -C tools-for-build -I../src/runtime grovel-headers
+tools-for-build/grovel-headers > output/stuff-groveled-from-headers.lisp
+
+$GNUMAKE -C src/runtime after-grovel-headers
+
+if [ -n "$SBCL_HOST_LOCATION" ]; then
+    rsync -a output/stuff-groveled-from-headers.lisp "$SBCL_HOST_LOCATION/output"
+fi

make-target-1.sh

@@ -31,31 +31,15 @@ fi
 
 # Build the runtime system and symbol table (.nm) file.
 #
-# (This C build has to come after the first genesis in order to get
-# the sbcl.h the C build needs, and come before the second genesis in
-# order to produce the symbol table file that second genesis needs. It
-# could come either before or after running the cross compiler; that
-# doesn't matter.)
-#
-# Note that the latter requirement does not apply to :linkage-table
-# builds, since the cross compiler does not depend on symbol tables in
-# that case.  Only because sbcl.nm is convenient for debugging purposes
-# is its generation left enabled even for those builds.
+# This C build has to come after the first genesis in order to get
+# the sbcl.h the C build needs.
 echo //building runtime system and symbol table file
 
 $GNUMAKE -C src/runtime clean
 # $GNUMAKE -C src/runtime depend
 $GNUMAKE $SBCL_MAKE_JOBS -C src/runtime all
 
-# Use a little C program to grab stuff from the C header files and
-# smash it into Lisp source code.
-$GNUMAKE -C tools-for-build -I../src/runtime grovel-headers
-tools-for-build/grovel-headers > output/stuff-groveled-from-headers.lisp
-
-$GNUMAKE -C src/runtime after-grovel-headers
-
 if [ -n "$SBCL_HOST_LOCATION" ]; then
     echo //copying target-1 output files to host
     rsync -a src/runtime/sbcl.nm "$SBCL_HOST_LOCATION/src/runtime/"
-    rsync -a output/stuff-groveled-from-headers.lisp "$SBCL_HOST_LOCATION/output"
 fi

make-target-2-load.lisp

@@ -35,6 +35,10 @@
           ;; more-or-less confined to serve-event, except for a test which now
           ;; detects whether COMPUTE-POLLFDS is defined and therefore testable.
           :OS-PROVIDES-POLL
+          ;; Used by genesis and C. Genesis uses presence of this feature to
+          ;; determine if a C file should be written to contain the linkage
+          ;; info.
+          :SB-PRELINK-LINKAGE-TABLE
           ;; The final batch of symbols is strictly for C. The LISP_FEATURE_
           ;; prefix on the corresponding #define is unfortunate.
           :GCC-TLS :USE-SYS-MMAP

make-target-contrib.sh

@@ -114,6 +114,6 @@ EOF
   fi
 done
 
-if [ $HEADER_HAS_BEEN_PRINTED = true ]; then
+if [ $HEADER_HAS_BEEN_PRINTED = true ] && [ "$IGNORE_CONTRIB_FAILURES" != "yes" ]; then
   exit 1
 fi