just learned about this project erjic that does seccomp+bwrap sandboxing https://codeberg.org/prisixia/erjic the interface is really nicely designed and i believe will fit the needs of my build system perfectly
-
previously (in conversation with junyer, the late RE2 maintainer) i had arrived upon the motto "build xor pkg" (see e.g. https://circumstances.run/@hipsterelectron/115241855909691763) to codify what (imho) pants and spack do right: separating packaging concerns from build concerns:
- a package manager (pip, spack, npm, poetry, and every linux distro) describes relationships across codebases (and therefore necessarily defines an interdependent ecosystem).
- a build tool covers one specific codebase. it is a tool for maintainers to describe their code, to develop the code, and to generate release artifacts (which are definitionally consumed by package managers).
@hipsterelectron so, where does this philosophy place CPack and rpmbuild, respectively?
As a person who writes software and uses CMake as a build system and also has to package (and rebuild) other people’s software with rpmbuild, I have _opinions_. -
i am forcing myself not to derail into a discussion of task scheduling, but "multiple processes at once" does in fact specifically mean multiple live processes executing in parallel through some form of timesharing mechanism (i ignore kernel threads here). the essentially round-robin preemption model with process-specific nice values (ignoring threads) used by linux and all BSDs i know of wants to share as much memory as possible across processes, because at all times it is trying to execute them all at once.
that's kind of a strange assumption, isn't it?
once you execute a process (via
execvlp(), orposix_spawn(), or whatever) it's immediately off to the races! that process is added to the big soup of other processes. you can't even stop it without sending a signal—which doesn't pause it, but kills it. the process itself needs to decide to open up a semaphore or pipe you gave it in order to pause its execution.think about it: isn't this a form of cooperative scheduling? when you can't rein in a subprocess except by killing it entirely? how often do you genuinely want a subprocess to execute however long it wants—unless you yourself control the code and force it to yield?
-
once you execute a process (via
execvlp(), orposix_spawn(), or whatever) it's immediately off to the races! that process is added to the big soup of other processes. you can't even stop it without sending a signal—which doesn't pause it, but kills it. the process itself needs to decide to open up a semaphore or pipe you gave it in order to pause its execution.think about it: isn't this a form of cooperative scheduling? when you can't rein in a subprocess except by killing it entirely? how often do you genuinely want a subprocess to execute however long it wants—unless you yourself control the code and force it to yield?
we will now terminate this tangent, but the point of this is: shared libraries (a packaging mechanism) are motivated by sharing memory among parallel process executions. your hardworking beloved distro packager is able to package software to use shared libraries (they "control the code" in that sense) as much as possible, in order to enable their end users to use the OS efficiently in unexpected ways (watching videos, making music, building LLVM).
-
once you execute a process (via
execvlp(), orposix_spawn(), or whatever) it's immediately off to the races! that process is added to the big soup of other processes. you can't even stop it without sending a signal—which doesn't pause it, but kills it. the process itself needs to decide to open up a semaphore or pipe you gave it in order to pause its execution.think about it: isn't this a form of cooperative scheduling? when you can't rein in a subprocess except by killing it entirely? how often do you genuinely want a subprocess to execute however long it wants—unless you yourself control the code and force it to yield?
@hipsterelectron yeah as an osdever this bugs me
memory entitlement and processor scheduling are capabilities that should be controlled by the parent and not the child -
we will now terminate this tangent, but the point of this is: shared libraries (a packaging mechanism) are motivated by sharing memory among parallel process executions. your hardworking beloved distro packager is able to package software to use shared libraries (they "control the code" in that sense) as much as possible, in order to enable their end users to use the OS efficiently in unexpected ways (watching videos, making music, building LLVM).
this is another reason why distro package managers almost always support at most one version of any package at any time across the system—the distro package manager is a tool to maintain the user's system, and the distro packagers build the entire dependency graph together. a distro is a monorepo, and the package manager is their build tool.
-
once you execute a process (via
execvlp(), orposix_spawn(), or whatever) it's immediately off to the races! that process is added to the big soup of other processes. you can't even stop it without sending a signal—which doesn't pause it, but kills it. the process itself needs to decide to open up a semaphore or pipe you gave it in order to pause its execution.think about it: isn't this a form of cooperative scheduling? when you can't rein in a subprocess except by killing it entirely? how often do you genuinely want a subprocess to execute however long it wants—unless you yourself control the code and force it to yield?
@hipsterelectron uh but SIGSTOP and SIGCONT do stop/continue a process without killing it
arbitrary code you don't control might not deal with that well (e.g. a wayland program would miss the keep-alive pings, or something that uses timers would have the timer signals queue up), but they're there -
this is another reason why distro package managers almost always support at most one version of any package at any time across the system—the distro package manager is a tool to maintain the user's system, and the distro packagers build the entire dependency graph together. a distro is a monorepo, and the package manager is their build tool.
( cc @ireneista @SRAZKVT this is intentionally handwavey but i'm working on it )
-
we will now terminate this tangent, but the point of this is: shared libraries (a packaging mechanism) are motivated by sharing memory among parallel process executions. your hardworking beloved distro packager is able to package software to use shared libraries (they "control the code" in that sense) as much as possible, in order to enable their end users to use the OS efficiently in unexpected ways (watching videos, making music, building LLVM).
@hipsterelectron this tangent is a subprocess.
-
this is another reason why distro package managers almost always support at most one version of any package at any time across the system—the distro package manager is a tool to maintain the user's system, and the distro packagers build the entire dependency graph together. a distro is a monorepo, and the package manager is their build tool.
in fact, it's completely possible to use distinct library versions across subgraphs of a dependency graph. this is, crucially, something both pants and spack support!
it's more impressive in spack, which incorporates this into dependency solving, whereas pants simply lets you declare multiple dependency versions with separate names (and then subprojects just depend upon the one they need). this is absolutely crucial for corporate monorepos like twitter inc, where multiple teams work on completely separate projects that don't communicate with each other. imposing a single version of a dependency means you have to migrate the whole repo at once before anyone can upgrade. it's a complete fucking disaster
-
in fact, it's completely possible to use distinct library versions across subgraphs of a dependency graph. this is, crucially, something both pants and spack support!
it's more impressive in spack, which incorporates this into dependency solving, whereas pants simply lets you declare multiple dependency versions with separate names (and then subprojects just depend upon the one they need). this is absolutely crucial for corporate monorepos like twitter inc, where multiple teams work on completely separate projects that don't communicate with each other. imposing a single version of a dependency means you have to migrate the whole repo at once before anyone can upgrade. it's a complete fucking disaster
@hipsterelectron was the single version really such a downside at Twitter? Want to hear you perspective from EE side.
From Data Platform I helped many times with the typical suspects, jackson, guava and other libraries messing everything up. We had the main issue of Hadoop being outside of the monorepo and having surprise runtime dependencies and failures and during my time we switched every to classpath isolation. I felt we were the “pain sponges” and that was good.
-
@hipsterelectron uh but SIGSTOP and SIGCONT do stop/continue a process without killing it
arbitrary code you don't control might not deal with that well (e.g. a wayland program would miss the keep-alive pings, or something that uses timers would have the timer signals queue up), but they're there@hipsterelectron tbh a shell's job control relies on somewhat on multi-process preemptive multitasking
background jobs can be stopped and continued, not just by direct SIGSTOP/SIGCONT (`command &` and `bg`) , but also by trying to read from the tty while not being in the foreground (causes SIGTTIN) -- that also causes it to stop until it's the foreground process (via `fg`) -
in fact, it's completely possible to use distinct library versions across subgraphs of a dependency graph. this is, crucially, something both pants and spack support!
it's more impressive in spack, which incorporates this into dependency solving, whereas pants simply lets you declare multiple dependency versions with separate names (and then subprojects just depend upon the one they need). this is absolutely crucial for corporate monorepos like twitter inc, where multiple teams work on completely separate projects that don't communicate with each other. imposing a single version of a dependency means you have to migrate the whole repo at once before anyone can upgrade. it's a complete fucking disaster
bazel doesn't support this, because google hates you. nix and guix also don't support this—but recall that nix and guix are actually distros, and distros benefit from system software sharing library versions (nix and guix can i believe have multiple versions of a library across dependency subgraphs—cc @alina is that right?).
bazel is unusable as a monorepo build system for this reason, and the google codebase is full of known vulnerabilities from millions of vendored copies of open source code that are never maintained or updated. that's how engineers are forced to deal with build systems that fail to describe the communication structure between subprojects—i.e. the human communication mechanisms within a corporation.
-
bazel doesn't support this, because google hates you. nix and guix also don't support this—but recall that nix and guix are actually distros, and distros benefit from system software sharing library versions (nix and guix can i believe have multiple versions of a library across dependency subgraphs—cc @alina is that right?).
bazel is unusable as a monorepo build system for this reason, and the google codebase is full of known vulnerabilities from millions of vendored copies of open source code that are never maintained or updated. that's how engineers are forced to deal with build systems that fail to describe the communication structure between subprojects—i.e. the human communication mechanisms within a corporation.
imho, guix especially but also nix have more powerful dependency structures than the typical distro methodology, and this is why users are encouraged to extend package recipes themselves, instead of imposing a stark distinction between distro packagers and end users.
this is wonderful in some ways. but like the lockfile PEP, it also tasks end users to develop the specialized expertise of a distro packager, who is tasked with carefully understanding changes across a dependency graph. guix and nix have configuration languages and introspection tools for this purpose, but tooling cannot replace expertise and judgement!
-
imho, guix especially but also nix have more powerful dependency structures than the typical distro methodology, and this is why users are encouraged to extend package recipes themselves, instead of imposing a stark distinction between distro packagers and end users.
this is wonderful in some ways. but like the lockfile PEP, it also tasks end users to develop the specialized expertise of a distro packager, who is tasked with carefully understanding changes across a dependency graph. guix and nix have configuration languages and introspection tools for this purpose, but tooling cannot replace expertise and judgement!
recall "b2b vs b2c"—this distinction can also be observed cryptographically. the standard distro model (where heroes like mgorny or q66 go through hell so you don't have to) also enables them to sign kernel and package build artifacts with a private key. that cryptographic signature codifies trust: "i personally have verified this is safe, with my expertise as an engineer and my love for the user". and if there's a problem, their reputation is on the line.
-
recall "b2b vs b2c"—this distinction can also be observed cryptographically. the standard distro model (where heroes like mgorny or q66 go through hell so you don't have to) also enables them to sign kernel and package build artifacts with a private key. that cryptographic signature codifies trust: "i personally have verified this is safe, with my expertise as an engineer and my love for the user". and if there's a problem, their reputation is on the line.
where do you draw that line, in a model which fails to distinguish between distro and user? forget mathematics—where can you even provide a statement of trust? who verifies safety?
nix and guix have landed upon "reproducibility" to answer this, where guix is imho much more thoughtful and ahierarchical about it. this particular cryptographic definition of "reproducibility" requires absolute bit-for-bit matching, across the whole dependency graph. there is no room for configuration—not without forking a whole subgraph.
-
where do you draw that line, in a model which fails to distinguish between distro and user? forget mathematics—where can you even provide a statement of trust? who verifies safety?
nix and guix have landed upon "reproducibility" to answer this, where guix is imho much more thoughtful and ahierarchical about it. this particular cryptographic definition of "reproducibility" requires absolute bit-for-bit matching, across the whole dependency graph. there is no room for configuration—not without forking a whole subgraph.
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
-
@hipsterelectron was the single version really such a downside at Twitter? Want to hear you perspective from EE side.
From Data Platform I helped many times with the typical suspects, jackson, guava and other libraries messing everything up. We had the main issue of Hadoop being outside of the monorepo and having surprise runtime dependencies and failures and during my time we switched every to classpath isolation. I felt we were the “pain sponges” and that was good.
@hipsterelectron at Netflix in the ML Platform the freedom of dependencies was an absolute nightmare in JVM world. Everything was service based but implying client libraries with heavy dependency trees. In ML you want every input you can get your hands on, so bring the entire universe together. Then you want to apply ML everywhere, thus take the whole universe everywhere else. Every single project with so much gradle boilerplate to resolve same issue all the time.
-
@hipsterelectron at Netflix in the ML Platform the freedom of dependencies was an absolute nightmare in JVM world. Everything was service based but implying client libraries with heavy dependency trees. In ML you want every input you can get your hands on, so bring the entire universe together. Then you want to apply ML everywhere, thus take the whole universe everywhere else. Every single project with so much gradle boilerplate to resolve same issue all the time.
@hipsterelectron on top of that, you want ML in batch processing, in stream processing and in services so every very heavy context needs to be compatible. The freedom we had meant embarrassing things like Spark and Flink used incompatible Avro and Iceberg versions, different JDK version, different Java source version, different Scala versions.
-
bit-for-bit matching is, impressively, actually achievable, despite lack of OS support for filesystem transactions. it can be achieved by isolating a build process within a virtualized directory (i believe both nix and guix use FUSE—cc @janneke is this true?). FUSE lets you achieve any number of guarantees the kernel and filesystem provide no API for. however, it indicates a flaw in POSIX and linux.
as is typical with linux, sandboxing like this is very difficult without requiring root, but recall that nix and guix are distros—the distro naturally requires root. however, this also means a codebase cannot provide a guix recipe for non-guix systems—a portable codebase must support some other build system (i like meson and automake).
there is an intrinsic pull towards monopolistic control (not maliciously nor intentionally) with systems that use classical cryptographic checksums for reproducibility. the checksum (often SHA-256, although i recommend SHA3-512/256 or BLAKE3) does not understand the concept of a "filesystem" like the kernel or libc. in fact, cryptographic indifferentiability ensures that the checksum cannot reveal any information about the thing it summarizes—or about how it changed.
a system which provides an environment matching the checksum therefore can tolerate no interaction with the host—or the user. "reproducibility", in this limited formulation, means that only this input will produce the desired output. this is why it necessitates forking entire subgraphs.
-
@hipsterelectron on top of that, you want ML in batch processing, in stream processing and in services so every very heavy context needs to be compatible. The freedom we had meant embarrassing things like Spark and Flink used incompatible Avro and Iceberg versions, different JDK version, different Java source version, different Scala versions.
@hipsterelectron also spent a whole year moving from Java 8 to 11 to 17 to 21, Scala 2.10 to 2.11 to 2.12 to 2.13 and Spark 2.1 to 2.4 to 3.x across tens (hundreds?) of projects. All updates intertwined like that because none could indepently jump to the end state.
Most aggravating was the lack of cross scala version support in Gradle. I had to patch so many projects to make it a variable to begin with, even on Java services consuming libraries, unaware of Scala.
