just learned about this project erjic that does seccomp+bwrap sandboxing https://codeberg.org/prisixia/erjic the interface is really nicely designed and i believe will fit the needs of my build system perfectly
-
so this is not a moral accusation but rather a technological observation: the bit-for-bit indifferentiability through cryptographic checksums tends toward a thin form of "reproducibility" which means:
- if you do everything exactly like everyone else, you will get the same result!
- if you do things differently, you can't reuse anything—you have to rebuild your new dependency graph from scratch.
i believe this (unintentionally) incentivizes the production of fiefdoms and silos, and especially makes it easier for capital to subvert guix freedoms. i have not observed any example of the latter in guix, but it is at this point trivial and well-understood to be a flaw of nix. lix is heroic, but swimming upstream is made more difficult in this shallow cryptographic reproducibility regime.
the distinction i happened upon today was "b2b vs b2c". "c" here means "end user", while "b" is "packager or codebase maintainer" (i.e. someone who agreed to take on responsibility for the end user's safety). this was inspired by the json profile format for the erjic sandboxing tool, which means my build tool can use it for sandboxing reliably (i'm "b" because i maintain the build tool, erjic is "b" because it's a codebase with a maintainer, and i'm taking responsibility for executing compiler subprocesses safely)
-
the distinction i happened upon today was "b2b vs b2c". "c" here means "end user", while "b" is "packager or codebase maintainer" (i.e. someone who agreed to take on responsibility for the end user's safety). this was inspired by the json profile format for the erjic sandboxing tool, which means my build tool can use it for sandboxing reliably (i'm "b" because i maintain the build tool, erjic is "b" because it's a codebase with a maintainer, and i'm taking responsibility for executing compiler subprocesses safely)
the packager who builds a codebase using my build tool? that's "b" too! the end user "c" is someone executing code built with my tool—and while i only interface with "b"s, we're all serving "c" together.
this formulation allows for "c" to be their own "b"(s), but it doesn't force them to, because it doesn't ablate the handoffs of accountability across the dependency hypergraph. while cryptographic indifferentiability is related to the problems of "reproducibility", it mostly just reflects the ablation of responsibility that removes standard trust boundaries.
-
the packager who builds a codebase using my build tool? that's "b" too! the end user "c" is someone executing code built with my tool—and while i only interface with "b"s, we're all serving "c" together.
this formulation allows for "c" to be their own "b"(s), but it doesn't force them to, because it doesn't ablate the handoffs of accountability across the dependency hypergraph. while cryptographic indifferentiability is related to the problems of "reproducibility", it mostly just reflects the ablation of responsibility that removes standard trust boundaries.
["we're all serving 'c' together" is intended to pun on queers serving cunt collectively. we're in your dependency graph and we are teaching the graph about gender]
-
["we're all serving 'c' together" is intended to pun on queers serving cunt collectively. we're in your dependency graph and we are teaching the graph about gender]
so of course i
spack immensely and one of the reasons why is because it maintains the capability of nix and guix in which end users and/or sysadmins are empowered to extend package recipes, but it works incredibly hard to reuse work as much as possible. a spack package recipe, at any time, describes every single version of a package at once—its build environment, its dependencies, where to fetch the sources, and how to configure and build and install it. -
so of course i
spack immensely and one of the reasons why is because it maintains the capability of nix and guix in which end users and/or sysadmins are empowered to extend package recipes, but it works incredibly hard to reuse work as much as possible. a spack package recipe, at any time, describes every single version of a package at once—its build environment, its dependencies, where to fetch the sources, and how to configure and build and install it.we also have a checksum graph and achieve bit-for-bit reproducibility, and we also (as of v1) build in a sandbox, because sandboxing is actually the easiest part of all this
-
as you may have inferred from the little orange bar in the codeberg link preview it is in rust which means i will have to make it an optional dep until gccrs is shocked with a bolt of lightning like frankenstein's monster. but sandboxing is so os-specific that it literally has to be an optional dep in any instance
@hipsterelectron an optional dep for what? -
@hipsterelectron
@guix (and @nixos_org I believe) use Linux containers for package builds, without network access.
#guix
#nixos
#reproduciblebuilds
@reproducible_builds@janneke @hipsterelectron @guix @nixos_org @reproducible_builds I/we do this at work. Of corse, we’re beholden to RPM so nothing is _binary_ reproducible.
-
we also have a checksum graph and achieve bit-for-bit reproducibility, and we also (as of v1) build in a sandbox, because sandboxing is actually the easiest part of all this
this is why erjic can be an optional dependency for my build system. because sandboxing is not standardized by POSIX, it will need OS-specific implementations anyway. if you make sandboxing a core assumption of your dependency graph architecture, your dependency graph too will need to be replicated across OSes. and if you have no concept of a dependency except a checksum, you have no capability to make any universally-quantified logical statements—it's (at best) existential
-
this is why erjic can be an optional dependency for my build system. because sandboxing is not standardized by POSIX, it will need OS-specific implementations anyway. if you make sandboxing a core assumption of your dependency graph architecture, your dependency graph too will need to be replicated across OSes. and if you have no concept of a dependency except a checksum, you have no capability to make any universally-quantified logical statements—it's (at best) existential
and i do in fact mean higher-order mathematical logic. i'm not a logician, but i believe the analogy of cryptographic reproducibility is somewhere between propositional and first-order logic—while the checksum graph is starkly propositional, i suspect the nix and guix languages support enough abstraction for parameterized recipes, which seems first-order to me (but this is not my field).
-
and i do in fact mean higher-order mathematical logic. i'm not a logician, but i believe the analogy of cryptographic reproducibility is somewhere between propositional and first-order logic—while the checksum graph is starkly propositional, i suspect the nix and guix languages support enough abstraction for parameterized recipes, which seems first-order to me (but this is not my field).
the spack model would be appropriately summarized with BE NOT AFRAID: https://spack.readthedocs.io/en/latest/spec_syntax.html
Here is an example of using a complex spec to install a very specific configuration of mpileaks:
spack install mpileaks@1.2:1.4 +debug ~qt target=x86_64_v3 %gcc@15 ^libelf@1.1 %clang@20The figure below helps you get a sense of the various parts that compose this spec:
-
the spack model would be appropriately summarized with BE NOT AFRAID: https://spack.readthedocs.io/en/latest/spec_syntax.html
Here is an example of using a complex spec to install a very specific configuration of mpileaks:
spack install mpileaks@1.2:1.4 +debug ~qt target=x86_64_v3 %gcc@15 ^libelf@1.1 %clang@20The figure below helps you get a sense of the various parts that compose this spec:
we do happen to employ the ASP logic language through the clingo ASP solver. lots of good resources on ASP https://potassco.org/resources/, which is kind of like SAT/SMT except that universal quantifier expressions can be stated and solved over infinite graph-structured solution spaces
-
we do happen to employ the ASP logic language through the clingo ASP solver. lots of good resources on ASP https://potassco.org/resources/, which is kind of like SAT/SMT except that universal quantifier expressions can be stated and solved over infinite graph-structured solution spaces
i was hired by LLNL right as spack was transitioning from a messy non-backtracking serial graph solver (our own, in python, grown organically over the decade since spack was created) to clingo (this was a huge risk). the reason we could do this was because package recipes were already declarative, and already described all versions of a package across all architectures at once.
this is super important: we already had a logical model—we just needed a more powerful solver. the solver did not dictate our logic. we applied the solver to our use case.
-
i was hired by LLNL right as spack was transitioning from a messy non-backtracking serial graph solver (our own, in python, grown organically over the decade since spack was created) to clingo (this was a huge risk). the reason we could do this was because package recipes were already declarative, and already described all versions of a package across all architectures at once.
this is super important: we already had a logical model—we just needed a more powerful solver. the solver did not dictate our logic. we applied the solver to our use case.
this is the advantage of protocols. spack was made for LLNL to provide a service to our physicists. these physicists encompass mathematicians and programmers—LLNL does simulations, so like the spack team, they're necessarily applied scientists. the tools development group (TDG) does spack, but our larger org within LLNL computing also developed parallelism libraries, which themselves made use of memory management primitives
-
this is the advantage of protocols. spack was made for LLNL to provide a service to our physicists. these physicists encompass mathematicians and programmers—LLNL does simulations, so like the spack team, they're necessarily applied scientists. the tools development group (TDG) does spack, but our larger org within LLNL computing also developed parallelism libraries, which themselves made use of memory management primitives
the purpose of all of this is to bring deep hardware-dependent concerns all the way up to the application layer, because HPC is generally one of the fields that requires this at all times. but notice something very important—the application layer is not the spack team, nor the RAJA team, but the scientists we serve.
our goal is to make their scientific work more efficient than any alternative. that's how we broke the nuclear fusion net-positive efficiency barrier after 60 years. that's what differentiates LLNL from LANL, who still pretends quantum computing is totally real.
-
the purpose of all of this is to bring deep hardware-dependent concerns all the way up to the application layer, because HPC is generally one of the fields that requires this at all times. but notice something very important—the application layer is not the spack team, nor the RAJA team, but the scientists we serve.
our goal is to make their scientific work more efficient than any alternative. that's how we broke the nuclear fusion net-positive efficiency barrier after 60 years. that's what differentiates LLNL from LANL, who still pretends quantum computing is totally real.
the point of this hagiography is not to claim the US is good at science, or that the spack team is smarter than anyone else (except LANL, who is overrated). in fact, i believe the netherlands is far better at funding science, and i fervently hope their physicists develop better silicon fabrication techniques soon.
the US is in fact terrible at funding science. but government employees are different from military generals or failson politicians. and in this hostile environment with funding swings every 4 years, this is what we arrived at to defend science from being beholden to IBM
-
the point of this hagiography is not to claim the US is good at science, or that the spack team is smarter than anyone else (except LANL, who is overrated). in fact, i believe the netherlands is far better at funding science, and i fervently hope their physicists develop better silicon fabrication techniques soon.
the US is in fact terrible at funding science. but government employees are different from military generals or failson politicians. and in this hostile environment with funding swings every 4 years, this is what we arrived at to defend science from being beholden to IBM
blue gene? that beat kasparov? that machine is still running. it was also the last IBM machine purchased by LLNL, because IBM cannot be relied upon to provide features for any sum of taxpayer money. they are completely incapable of it. the first exascale machine (possibly second, idgaf) is instead a completely modular supercomputer from consumer AMD chips. and spack enables the government to pay AMD a bazillion dollars for bragging rights without getting stuck into a software–hardware monopoly like IBM loves
-
blue gene? that beat kasparov? that machine is still running. it was also the last IBM machine purchased by LLNL, because IBM cannot be relied upon to provide features for any sum of taxpayer money. they are completely incapable of it. the first exascale machine (possibly second, idgaf) is instead a completely modular supercomputer from consumer AMD chips. and spack enables the government to pay AMD a bazillion dollars for bragging rights without getting stuck into a software–hardware monopoly like IBM loves
open source, in the informal sense, is something todd gamblin (spack creator, one of my favorite people in the whole world) convinced the DOE to invest in (like i did at twitter), and he was immediately proven right, because a ridiculous number of other labs are now using spack.
this also means the power relations spack enables (there are always power relations in any org) are now available elsewhere. spack in this view is an institutional and political tool—just like every other package manager.
-
open source, in the informal sense, is something todd gamblin (spack creator, one of my favorite people in the whole world) convinced the DOE to invest in (like i did at twitter), and he was immediately proven right, because a ridiculous number of other labs are now using spack.
this also means the power relations spack enables (there are always power relations in any org) are now available elsewhere. spack in this view is an institutional and political tool—just like every other package manager.
you can and should argue that "b2b vs b2c" codifies deeply corporate-centric profit relations, especially those advanced by VCs and tech startups. in that way, it's a harmful analogy, and it exposes my preconceptual bias from employment (twitter in US tech, and LLNL in US government/academia). i would love to hear ways the "customer" terminology inculcates harmful assumptions we should steer away from. please dunk on me. write screeds of my misdeeds.
-
you can and should argue that "b2b vs b2c" codifies deeply corporate-centric profit relations, especially those advanced by VCs and tech startups. in that way, it's a harmful analogy, and it exposes my preconceptual bias from employment (twitter in US tech, and LLNL in US government/academia). i would love to hear ways the "customer" terminology inculcates harmful assumptions we should steer away from. please dunk on me. write screeds of my misdeeds.
but i like the word "customer" because it's missing from "open source". when maintainers (consider xz-utils) are overwhelmed, they're serving microsoft customers. but those customers can't reach them, and certainly can't pay them. microsoft, and google, and facebook, and certainly amazon, are serving as middlemen and doing a shit fucking job at it
-
but i like the word "customer" because it's missing from "open source". when maintainers (consider xz-utils) are overwhelmed, they're serving microsoft customers. but those customers can't reach them, and certainly can't pay them. microsoft, and google, and facebook, and certainly amazon, are serving as middlemen and doing a shit fucking job at it
i especially seek to destroy google for vengeance, but the entire cloud apparatus runs on not just gnu/linux, but every fucking maintainer in between. more and more it's musl/linux because glibc is monopolistic, yet somehow linus torvalds is the only one getting rich off this shit
