just learned about this project erjic that does seccomp+bwrap sandboxing https://codeberg.org/prisixia/erjic the interface is really nicely designed and i believe will fit the needs of my build system perfectly
-
i have just made up an analogy to summarize why json profiles are so important and i refer to it as "b2b vs b2c" software interfaces
"b2b" is a tongue-in-cheek name for tool-to-tool communication (e.g. build tool => sandbox configuration). this is why the package metadata format i spent years negotiating with the python community to develop https://pip.pypa.io/en/latest/reference/installation-report/ uses JSON, because JSON is extremely stable as a protocol and can be deterministically checksummed. this is why it's suitable for lockfiles
-
@hipsterelectron b2c is "exploit them as much as possible, never give them their means of independence that binds them to your product" and b2b is "only exploit them a reasonable, marketable amount, because their sales people aren't stupid" right
@alina @hipsterelectron don't forget that b2c also has "don't document anything because nobody will read it anyway" while b2b has "don't document anything otherwise they may not need your support contract" -
"b2b" is a tongue-in-cheek name for tool-to-tool communication (e.g. build tool => sandbox configuration). this is why the package metadata format i spent years negotiating with the python community to develop https://pip.pypa.io/en/latest/reference/installation-report/ uses JSON, because JSON is extremely stable as a protocol and can be deterministically checksummed. this is why it's suitable for lockfiles
@hipsterelectron is there anything of constant size that cannot be deterministically checksummed - ohhh wait there are formats without "canonical" form (minus whitespace) i remember that people still have bad ideas sometimes
-
"b2b" is a tongue-in-cheek name for tool-to-tool communication (e.g. build tool => sandbox configuration). this is why the package metadata format i spent years negotiating with the python community to develop https://pip.pypa.io/en/latest/reference/installation-report/ uses JSON, because JSON is extremely stable as a protocol and can be deterministically checksummed. this is why it's suitable for lockfiles
when brett cannon (the maintainer of the
packaginglibrary who ensures pypi uses the backdooredMETADATAformat) wrote PEP 751 in collusion with astral (the startup that stole my zip file work without credit and sold to openai), he made sure to erase this work, because PEP 751 is a TOML format (and this is the crux of the distinction i'm making) -
"b2b" is a tongue-in-cheek name for tool-to-tool communication (e.g. build tool => sandbox configuration). this is why the package metadata format i spent years negotiating with the python community to develop https://pip.pypa.io/en/latest/reference/installation-report/ uses JSON, because JSON is extremely stable as a protocol and can be deterministically checksummed. this is why it's suitable for lockfiles
@hipsterelectron and then some use toml instead even though it isn't supposed to be read by someone
-
they mentioned they use it for running games which is so great to hear bc that's exactly the kind of end-user application that typical sandboxing mechanisms don't care about whatsoever and which (like compilers) have very strong dependencies upon which files they can access
@hipsterelectron
speaking of games, it should come with a script that sets up a sandboxed Wine desktop
(might be out of scope but still) -
when brett cannon (the maintainer of the
packaginglibrary who ensures pypi uses the backdooredMETADATAformat) wrote PEP 751 in collusion with astral (the startup that stole my zip file work without credit and sold to openai), he made sure to erase this work, because PEP 751 is a TOML format (and this is the crux of the distinction i'm making)TOML is not stable and not widely supported, but let's ignore that. the important quality of TOML is that it has multiple equivalent representations and absolutely no normalization mechanism. this means you can't reliably checksum it.
consider for a moment if being able to determine whether a lockfile has been changed might be useful in any way.
-
TOML is not stable and not widely supported, but let's ignore that. the important quality of TOML is that it has multiple equivalent representations and absolutely no normalization mechanism. this means you can't reliably checksum it.
consider for a moment if being able to determine whether a lockfile has been changed might be useful in any way.
PEP 751 is a useful case study in conflation of b2b and b2c because it describes mutually contradictory goals. first off, i claim a lockfile is not user-facing: it is the output of a process invocation (pip, uv, poetry, etc) and intended to be consumed by another process. brett cannon instead courageously fights for user empowerment and describes the lockfile as something a user is expected to audit by hand (so it's their fault if they didn't read it closely enough and get hacked). this motivates the TOML format.
-
PEP 751 is a useful case study in conflation of b2b and b2c because it describes mutually contradictory goals. first off, i claim a lockfile is not user-facing: it is the output of a process invocation (pip, uv, poetry, etc) and intended to be consumed by another process. brett cannon instead courageously fights for user empowerment and describes the lockfile as something a user is expected to audit by hand (so it's their fault if they didn't read it closely enough and get hacked). this motivates the TOML format.
@hipsterelectron lmao
-
TOML is not stable and not widely supported, but let's ignore that. the important quality of TOML is that it has multiple equivalent representations and absolutely no normalization mechanism. this means you can't reliably checksum it.
consider for a moment if being able to determine whether a lockfile has been changed might be useful in any way.
@hipsterelectron if you're already using a structured data format you might as well destructure it into its dictionary representation for doing the comparison right
-
@hipsterelectron if you're already using a structured data format you might as well destructure it into its dictionary representation for doing the comparison right
@hipsterelectron i also wouldnt want my lockfiles to use the canonical / stripped json version but to be pretty-printed instead, of which the formatting can vary
-
PEP 751 is a useful case study in conflation of b2b and b2c because it describes mutually contradictory goals. first off, i claim a lockfile is not user-facing: it is the output of a process invocation (pip, uv, poetry, etc) and intended to be consumed by another process. brett cannon instead courageously fights for user empowerment and describes the lockfile as something a user is expected to audit by hand (so it's their fault if they didn't read it closely enough and get hacked). this motivates the TOML format.
in fact, the TOML format is appropriate for exactly that b2c case! we use it in pants: https://www.pantsbuild.org/stable/docs/getting-started/initial-configuration users of pants have a
pants.tomlat the repo root for this reason.what about subprojects? so pants is a monorepo build tool, and per-directory BUILD files cover project-specific config. BUILD files are restricted python code, because we need to support loops and other logic. this is one major advantage over autoconf, in which key-value configuration parameters are determined by user-provided shell scripts. this makes autoconf-based builds impossible to introspect or interop with other tooling
-
in fact, the TOML format is appropriate for exactly that b2c case! we use it in pants: https://www.pantsbuild.org/stable/docs/getting-started/initial-configuration users of pants have a
pants.tomlat the repo root for this reason.what about subprojects? so pants is a monorepo build tool, and per-directory BUILD files cover project-specific config. BUILD files are restricted python code, because we need to support loops and other logic. this is one major advantage over autoconf, in which key-value configuration parameters are determined by user-provided shell scripts. this makes autoconf-based builds impossible to introspect or interop with other tooling
pants does maintain the same distinction as autoconf in its separation of "maintainer-defined" vs "packager-defined" configuration. in my automake C projects (see e.g. https://codeberg.org/cosmicexplorer/delulu), you'll see i build the
configurescript myself and track it in the repo, while a packager can invoke my prebuilt script from a tarball or git checkout.correspondingly, the
pants.tomldefines configuration values i control, but the packager can override or extend these values via environment variables or cli args. see https://www.pantsbuild.org/stable/docs/using-pants/key-concepts/options for more -
pants does maintain the same distinction as autoconf in its separation of "maintainer-defined" vs "packager-defined" configuration. in my automake C projects (see e.g. https://codeberg.org/cosmicexplorer/delulu), you'll see i build the
configurescript myself and track it in the repo, while a packager can invoke my prebuilt script from a tarball or git checkout.correspondingly, the
pants.tomldefines configuration values i control, but the packager can override or extend these values via environment variables or cli args. see https://www.pantsbuild.org/stable/docs/using-pants/key-concepts/options for morethese handoffs between release artifacts (dist tarball, git checkout) correspond to handoffs between interacting groups of human beings (codebase maintainers "upstream" to "downstream" packagers). "upstream" and "downstream" are often used informally to mean two distinct handoffs:
- between codebases that depend upon upstream,
- distro packagers which consume a whole dependency graph.
i have frequently castigated and derided cargo for failing to distinguish between these two.
-
these handoffs between release artifacts (dist tarball, git checkout) correspond to handoffs between interacting groups of human beings (codebase maintainers "upstream" to "downstream" packagers). "upstream" and "downstream" are often used informally to mean two distinct handoffs:
- between codebases that depend upon upstream,
- distro packagers which consume a whole dependency graph.
i have frequently castigated and derided cargo for failing to distinguish between these two.
previously (in conversation with junyer, the late RE2 maintainer) i had arrived upon the motto "build xor pkg" (see e.g. https://circumstances.run/@hipsterelectron/115241855909691763) to codify what (imho) pants and spack do right: separating packaging concerns from build concerns:
- a package manager (pip, spack, npm, poetry, and every linux distro) describes relationships across codebases (and therefore necessarily defines an interdependent ecosystem).
- a build tool covers one specific codebase. it is a tool for maintainers to describe their code, to develop the code, and to generate release artifacts (which are definitionally consumed by package managers).
-
when brett cannon (the maintainer of the
packaginglibrary who ensures pypi uses the backdooredMETADATAformat) wrote PEP 751 in collusion with astral (the startup that stole my zip file work without credit and sold to openai), he made sure to erase this work, because PEP 751 is a TOML format (and this is the crux of the distinction i'm making)@hipsterelectron I guess they have no issue with bare python not being able to write toml files if the tooling use rust.
-
previously (in conversation with junyer, the late RE2 maintainer) i had arrived upon the motto "build xor pkg" (see e.g. https://circumstances.run/@hipsterelectron/115241855909691763) to codify what (imho) pants and spack do right: separating packaging concerns from build concerns:
- a package manager (pip, spack, npm, poetry, and every linux distro) describes relationships across codebases (and therefore necessarily defines an interdependent ecosystem).
- a build tool covers one specific codebase. it is a tool for maintainers to describe their code, to develop the code, and to generate release artifacts (which are definitionally consumed by package managers).
you might wonder whether a shared library or an executable is consumed by package managers in the same way as a python wheel. in fact, a shared library is a packaging format defined by the dynamic linker, and the dynamic linker is in fact a package manager defined by by the libc. an executable is in (typically) ELF or mach-o format, and the OS executable interpreter is the package manager consuming that output.
-
you might wonder whether a shared library or an executable is consumed by package managers in the same way as a python wheel. in fact, a shared library is a packaging format defined by the dynamic linker, and the dynamic linker is in fact a package manager defined by by the libc. an executable is in (typically) ELF or mach-o format, and the OS executable interpreter is the package manager consuming that output.
@hipsterelectron static executable is a distro? :galaxybrain:
-
you might wonder whether a shared library or an executable is consumed by package managers in the same way as a python wheel. in fact, a shared library is a packaging format defined by the dynamic linker, and the dynamic linker is in fact a package manager defined by by the libc. an executable is in (typically) ELF or mach-o format, and the OS executable interpreter is the package manager consuming that output.
these are not the kind of analogies introduced in high school physics, which describe ballistic trajectories without air resistance or gravitational potential difference or the curvature of the earth. the OS executable interpreter is literally a package manager and the libc dynamic linker is also literally a manage manager.
the OS in particular is notable because it maps a binary executable into a format suitable for the MMU and instruction stream required by the CPU. the compiler is of course complicit in this—the ELF format is translated by the kernel into an instruction stream for the CPU. during this process the data is read from the filesystem into RAM, because the CPU operates upon RAM.
-
just learned about this project erjic that does seccomp+bwrap sandboxing https://codeberg.org/prisixia/erjic the interface is really nicely designed and i believe will fit the needs of my build system perfectly
@hipsterelectron Have you seen https://gitlab.exherbo.org/sydbox/sydbox ? xD
