just learned about this project erjic that does seccomp+bwrap sandboxing https://codeberg.org/prisixia/erjic the interface is really nicely designed and i believe will fit the needs of my build system perfectly
-
bazel doesn't support this, because google hates you. nix and guix also don't support this—but recall that nix and guix are actually distros, and distros benefit from system software sharing library versions (nix and guix can i believe have multiple versions of a library across dependency subgraphs—cc @alina is that right?).
bazel is unusable as a monorepo build system for this reason, and the google codebase is full of known vulnerabilities from millions of vendored copies of open source code that are never maintained or updated. that's how engineers are forced to deal with build systems that fail to describe the communication structure between subprojects—i.e. the human communication mechanisms within a corporation.
imho, guix especially but also nix have more powerful dependency structures than the typical distro methodology, and this is why users are encouraged to extend package recipes themselves, instead of imposing a stark distinction between distro packagers and end users.
this is wonderful in some ways. but like the lockfile PEP, it also tasks end users to develop the specialized expertise of a distro packager, who is tasked with carefully understanding changes across a dependency graph. guix and nix have configuration languages and introspection tools for this purpose, but tooling cannot replace expertise and judgement!
-
imho, guix especially but also nix have more powerful dependency structures than the typical distro methodology, and this is why users are encouraged to extend package recipes themselves, instead of imposing a stark distinction between distro packagers and end users.
this is wonderful in some ways. but like the lockfile PEP, it also tasks end users to develop the specialized expertise of a distro packager, who is tasked with carefully understanding changes across a dependency graph. guix and nix have configuration languages and introspection tools for this purpose, but tooling cannot replace expertise and judgement!
recall "b2b vs b2c"—this distinction can also be observed cryptographically. the standard distro model (where heroes like mgorny or q66 go through hell so you don't have to) also enables them to sign kernel and package build artifacts with a private key. that cryptographic signature codifies trust: "i personally have verified this is safe, with my expertise as an engineer and my love for the user". and if there's a problem, their reputation is on the line.
-
recall "b2b vs b2c"—this distinction can also be observed cryptographically. the standard distro model (where heroes like mgorny or q66 go through hell so you don't have to) also enables them to sign kernel and package build artifacts with a private key. that cryptographic signature codifies trust: "i personally have verified this is safe, with my expertise as an engineer and my love for the user". and if there's a problem, their reputation is on the line.
where do you draw that line, in a model which fails to distinguish between distro and user? forget mathematics—where can you even provide a statement of trust? who verifies safety?
nix and guix have landed upon "reproducibility" to answer this, where guix is imho much more thoughtful and ahierarchical about it. this particular cryptographic definition of "reproducibility" requires absolute bit-for-bit matching, across the whole dependency graph. there is no room for configuration—not without forking a whole subgraph.
-
where do you draw that line, in a model which fails to distinguish between distro and user? forget mathematics—where can you even provide a statement of trust? who verifies safety?
nix and guix have landed upon "reproducibility" to answer this, where guix is imho much more thoughtful and ahierarchical about it. this particular cryptographic definition of "reproducibility" requires absolute bit-for-bit matching, across the whole dependency graph. there is no room for configuration—not without forking a whole subgraph.
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
-
@hipsterelectron was the single version really such a downside at Twitter? Want to hear you perspective from EE side.
From Data Platform I helped many times with the typical suspects, jackson, guava and other libraries messing everything up. We had the main issue of Hadoop being outside of the monorepo and having surprise runtime dependencies and failures and during my time we switched every to classpath isolation. I felt we were the “pain sponges” and that was good.
@hipsterelectron at Netflix in the ML Platform the freedom of dependencies was an absolute nightmare in JVM world. Everything was service based but implying client libraries with heavy dependency trees. In ML you want every input you can get your hands on, so bring the entire universe together. Then you want to apply ML everywhere, thus take the whole universe everywhere else. Every single project with so much gradle boilerplate to resolve same issue all the time.
-
@hipsterelectron at Netflix in the ML Platform the freedom of dependencies was an absolute nightmare in JVM world. Everything was service based but implying client libraries with heavy dependency trees. In ML you want every input you can get your hands on, so bring the entire universe together. Then you want to apply ML everywhere, thus take the whole universe everywhere else. Every single project with so much gradle boilerplate to resolve same issue all the time.
@hipsterelectron on top of that, you want ML in batch processing, in stream processing and in services so every very heavy context needs to be compatible. The freedom we had meant embarrassing things like Spark and Flink used incompatible Avro and Iceberg versions, different JDK version, different Java source version, different Scala versions.
-
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
there is an intrinsic pull towards monopolistic control (not maliciously nor intentionally) with systems that use classical cryptographic checksums for reproducibility. the checksum (often SHA-256, although i recommend SHA3-512/256 or BLAKE3) does not understand the concept of a "filesystem" like the kernel or libc. in fact, cryptographic indifferentiability ensures that the checksum cannot reveal any information about the thing it summarizes—or about how it changed.
a system which provides an environment matching the checksum therefore can tolerate no interaction with the host—or the user. "reproducibility", in this limited formulation, means that only this input will produce the desired output. this is why it necessitates forking entire subgraphs.
-
@hipsterelectron on top of that, you want ML in batch processing, in stream processing and in services so every very heavy context needs to be compatible. The freedom we had meant embarrassing things like Spark and Flink used incompatible Avro and Iceberg versions, different JDK version, different Java source version, different Scala versions.
@hipsterelectron also spent a whole year moving from Java 8 to 11 to 17 to 21, Scala 2.10 to 2.11 to 2.12 to 2.13 and Spark 2.1 to 2.4 to 3.x across tens (hundreds?) of projects. All updates intertwined like that because none could indepently jump to the end state.
Most aggravating was the lack of cross scala version support in Gradle. I had to patch so many projects to make it a variable to begin with, even on Java services consuming libraries, unaware of Scala.
-
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
@hipsterelectron
@guix (and @nixos_org I believe) use Linux containers for package builds, without network access.
#guix
#nixos
#reproduciblebuilds
@reproducible_builds -
there is an intrinsic pull towards monopolistic control (not maliciously nor intentionally) with systems that use classical cryptographic checksums for reproducibility. the checksum (often SHA-256, although i recommend SHA3-512/256 or BLAKE3) does not understand the concept of a "filesystem" like the kernel or libc. in fact, cryptographic indifferentiability ensures that the checksum cannot reveal any information about the thing it summarizes—or about how it changed.
a system which provides an environment matching the checksum therefore can tolerate no interaction with the host—or the user. "reproducibility", in this limited formulation, means that only this input will produce the desired output. this is why it necessitates forking entire subgraphs.
now, i fully believe in and trust the guix project not to force monopolies. and indeed, the guix form of "reproducibility" is very specifically defined in a communal manner: trust is when my friends and organizations i trust agree upon the checksum together. this is very thoughtful, and makes "trust" a bottom-up process instead of a top-down hierarchy.
guix making use of guile scheme is also much more appropriate—the language semantics are recursively defined in scheme (which unlike lisp, really does believe in constructing a tower of IRs—instead of implementation-defined, it's protocols all the way down).
-
@terezi @hipsterelectron the usual way to pass libraries into stdenv.mkDerivation is by referencing nixpkgs in the attrs to pkgs.callPackage like this https://github.com/NixOS/nixpkgs/blob/e99d0bffd2a4cafb19335ea4ec95edc9ca23a0a4/pkgs/os-specific/linux/systemd/default.nix#L36
and then pass them into nativeBuildInputs or buildInputs https://github.com/NixOS/nixpkgs/blob/e99d0bffd2a4cafb19335ea4ec95edc9ca23a0a4/pkgs/os-specific/linux/systemd/default.nix#L342
also nix is not an operating system, the OS is called NixOS. there is a functional programming language called nix, with its stdlib designed to write package expressions and declare & define options in a module system which NixOS builds upon
you can have multiple versions of a library but you have to pin the specific revision/tag/commit of nixpkgs from which you want to source the package definition as flake input (or with npins or fetching a specific git archive revision of nixpkgs with a trivial fetcher), or write it yourself and pin the revision/tag/commit of the source archive you want to build the same way
possibly you will have to use autoPatchelfHook to do the linking properly, but outside of library/binary contexts you can just template a package using a function with the library package and the binary name as parameters and have differently named binaries in your $PATH, or use multiple $PATH environments like `nix shell` and switch between contexts using direnv or tmux for example
-
@terezi @hipsterelectron the usual way to pass libraries into stdenv.mkDerivation is by referencing nixpkgs in the attrs to pkgs.callPackage like this https://github.com/NixOS/nixpkgs/blob/e99d0bffd2a4cafb19335ea4ec95edc9ca23a0a4/pkgs/os-specific/linux/systemd/default.nix#L36
and then pass them into nativeBuildInputs or buildInputs https://github.com/NixOS/nixpkgs/blob/e99d0bffd2a4cafb19335ea4ec95edc9ca23a0a4/pkgs/os-specific/linux/systemd/default.nix#L342
also nix is not an operating system, the OS is called NixOS. there is a functional programming language called nix, with its stdlib designed to write package expressions and declare & define options in a module system which NixOS builds upon
you can have multiple versions of a library but you have to pin the specific revision/tag/commit of nixpkgs from which you want to source the package definition as flake input (or with npins or fetching a specific git archive revision of nixpkgs with a trivial fetcher), or write it yourself and pin the revision/tag/commit of the source archive you want to build the same way
possibly you will have to use autoPatchelfHook to do the linking properly, but outside of library/binary contexts you can just template a package using a function with the library package and the binary name as parameters and have differently named binaries in your $PATH, or use multiple $PATH environments like `nix shell` and switch between contexts using direnv or tmux for example
@terezi @hipsterelectron here's how to do the ELF patching / relinking manually by the example of the package i wrote for binary ninja
each source tree file path i reference in a path literal value gets passed to its nativeBuildInputs and then string interpolated to in the call to patchelf in the installPhase shell script. declaring nativeBuildInputs is needed to ensure availability of the referenced nix package during build time, and due to the sandboxed environment also access to the nix store path in the builder's mount namespace
{
inputs = {
flake-parts.url = "github:hercules-ci/flake-parts";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
};outputs =
inputs@{ flake-parts, ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
systems = [
"x86_64-linux"
];
perSystem =
{
config,
self',
inputs',
pkgs,
system,
lib,
...
}:
{
packages.default = (
pkgs.callPackage (
{
curl,
lib,
qt6,
patchelf,
pkg-config,
pkgsStatic,
stdenv,
...
}:
let
inherit (lib)
concatStringsSep
concatMapStringsSep
getExe
;
qt6libs = (qt6.qtbase.overrideAttrs { dontWrapQtApps = true; });
libdrv =
name: path:
stdenv.mkDerivation {
dontWrapQtApps = true;
dontUnpack = true;
dontConfigure = true;
dontBuild = true;
src = path;
version = "1.0.0";
inherit name;
installPhase = ''
install -m755 -D $src $out/lib/${name}
'';
};
dynamicDeps = map (x: libdrv x) [
(libdrv "libbinaryninjacore.so.1" ./src/libbinaryninjacore.so.1)
(libdrv "libbinaryninjaui.so.1" ./src/libbinaryninjaui.so.1)
(libdrv "libQt6Widgets.so.6" ./src/libQt6Widgets.so.6)
(libdrv "libQt6Gui.so.6" ./src/libQt6Gui.so.6)
(libdrv "libQt6Core.so.6" ./src/libQt6Core.so.6)
(libdrv "libQt6DBus.so.6" ./src/libQt6DBus.so.6)
(libdrv "libQt6OpenGL.so.0" ./src/libQt6OpenGL.so.6)
];
includePath = concatStringsSep "/lib:" dynamicDeps;
in
stdenv.mkDerivation {
src = ./src/binaryninja;
name = "binja";
nativeBuildInputs = [
# autoPatchelfHook
pkg-config
curl.dev
stdenv.cc.cc.lib
qt6.wrapQtAppsHook
# qt6libs
# curl.dev
# stdenv.cc.cc.lib
];
buildInputs = [ qt6libs ];
# ++ dynamicDeps;
dontWrapQtApps = true;
dontUnpack = true;
dontConfigure = true;
dontBuild = true;
installPhase =
let
optionFormat = opt: {
option = "--${opt}";
sep = " ";
explicitBool = false;
};
argv = (
concatStringsSep " " (
lib.cli.toCommandLine optionFormat {
set-interpreter = "${stdenv.cc}/nix-support/dynamic-linker";
set-rpath = ''"${builtins.trace includePath includePath}"'';
}
)
);
in
''
runHook preInstall
install -m755 -D $src $out/bin/binja
${getExe patchelf} ${builtins.trace argv argv} $out/bin/binja
${
concatMapStringsSep "\n" (
x: "${getExe patchelf} --replace-needed ${x.meta.name} ${x}/lib/${x.name} $out/bin/binja"
) dynamicDeps
}
runHook postInstall
'';
# preFixup = ''
# wrapQtApp $out/bin/binja --prefix ${./src} : $out/bin/binja
# '';
meta = with lib; {
homepage = "https://binary.ninja";
description = "an interactive decompiler, disassembler, debugger, and binary analysis platform";
license = licenses.unfree;
platforms = platforms.linux;
};
}
) { }
);
};
flake = {
nixosModules.default = { lib, config, ... }: {
options.programs.binja = with lib.types; {
enable = lib.mkEnableOption "binary ninja";
};
};
};
};
} -
now, i fully believe in and trust the guix project not to force monopolies. and indeed, the guix form of "reproducibility" is very specifically defined in a communal manner: trust is when my friends and organizations i trust agree upon the checksum together. this is very thoughtful, and makes "trust" a bottom-up process instead of a top-down hierarchy.
guix making use of guile scheme is also much more appropriate—the language semantics are recursively defined in scheme (which unlike lisp, really does believe in constructing a tower of IRs—instead of implementation-defined, it's protocols all the way down).
so this is not a moral accusation but rather a technological observation: the bit-for-bit indifferentiability through cryptographic checksums tends toward a thin form of "reproducibility" which means:
- if you do everything exactly like everyone else, you will get the same result!
- if you do things differently, you can't reuse anything—you have to rebuild your new dependency graph from scratch.
i believe this (unintentionally) incentivizes the production of fiefdoms and silos, and especially makes it easier for capital to subvert guix freedoms. i have not observed any example of the latter in guix, but it is at this point trivial and well-understood to be a flaw of nix. lix is heroic, but swimming upstream is made more difficult in this shallow cryptographic reproducibility regime.
-
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
@hipsterelectron @janneke oh, FUSE! That’s an interesting idea.
-
so this is not a moral accusation but rather a technological observation: the bit-for-bit indifferentiability through cryptographic checksums tends toward a thin form of "reproducibility" which means:
- if you do everything exactly like everyone else, you will get the same result!
- if you do things differently, you can't reuse anything—you have to rebuild your new dependency graph from scratch.
i believe this (unintentionally) incentivizes the production of fiefdoms and silos, and especially makes it easier for capital to subvert guix freedoms. i have not observed any example of the latter in guix, but it is at this point trivial and well-understood to be a flaw of nix. lix is heroic, but swimming upstream is made more difficult in this shallow cryptographic reproducibility regime.
the distinction i happened upon today was "b2b vs b2c". "c" here means "end user", while "b" is "packager or codebase maintainer" (i.e. someone who agreed to take on responsibility for the end user's safety). this was inspired by the json profile format for the erjic sandboxing tool, which means my build tool can use it for sandboxing reliably (i'm "b" because i maintain the build tool, erjic is "b" because it's a codebase with a maintainer, and i'm taking responsibility for executing compiler subprocesses safely)
-
the distinction i happened upon today was "b2b vs b2c". "c" here means "end user", while "b" is "packager or codebase maintainer" (i.e. someone who agreed to take on responsibility for the end user's safety). this was inspired by the json profile format for the erjic sandboxing tool, which means my build tool can use it for sandboxing reliably (i'm "b" because i maintain the build tool, erjic is "b" because it's a codebase with a maintainer, and i'm taking responsibility for executing compiler subprocesses safely)
the packager who builds a codebase using my build tool? that's "b" too! the end user "c" is someone executing code built with my tool—and while i only interface with "b"s, we're all serving "c" together.
this formulation allows for "c" to be their own "b"(s), but it doesn't force them to, because it doesn't ablate the handoffs of accountability across the dependency hypergraph. while cryptographic indifferentiability is related to the problems of "reproducibility", it mostly just reflects the ablation of responsibility that removes standard trust boundaries.
-
the packager who builds a codebase using my build tool? that's "b" too! the end user "c" is someone executing code built with my tool—and while i only interface with "b"s, we're all serving "c" together.
this formulation allows for "c" to be their own "b"(s), but it doesn't force them to, because it doesn't ablate the handoffs of accountability across the dependency hypergraph. while cryptographic indifferentiability is related to the problems of "reproducibility", it mostly just reflects the ablation of responsibility that removes standard trust boundaries.
["we're all serving 'c' together" is intended to pun on queers serving cunt collectively. we're in your dependency graph and we are teaching the graph about gender]
-
["we're all serving 'c' together" is intended to pun on queers serving cunt collectively. we're in your dependency graph and we are teaching the graph about gender]
so of course i
spack immensely and one of the reasons why is because it maintains the capability of nix and guix in which end users and/or sysadmins are empowered to extend package recipes, but it works incredibly hard to reuse work as much as possible. a spack package recipe, at any time, describes every single version of a package at once—its build environment, its dependencies, where to fetch the sources, and how to configure and build and install it. -
so of course i
spack immensely and one of the reasons why is because it maintains the capability of nix and guix in which end users and/or sysadmins are empowered to extend package recipes, but it works incredibly hard to reuse work as much as possible. a spack package recipe, at any time, describes every single version of a package at once—its build environment, its dependencies, where to fetch the sources, and how to configure and build and install it.we also have a checksum graph and achieve bit-for-bit reproducibility, and we also (as of v1) build in a sandbox, because sandboxing is actually the easiest part of all this
-
as you may have inferred from the little orange bar in the codeberg link preview it is in rust which means i will have to make it an optional dep until gccrs is shocked with a bolt of lightning like frankenstein's monster. but sandboxing is so os-specific that it literally has to be an optional dep in any instance
@hipsterelectron an optional dep for what?
