@hipsterelectron also spent a whole year moving from Java 8 to 11 to 17 to 21, Scala 2.10 to 2.11 to 2.12 to 2.13 and Spark 2.1 to 2.4 to 3.x across tens (hundreds?) of projects. All updates intertwined like that because none could indepently jump to the end state.

Most aggravating was the lack of cross scala version support in Gradle. I had to patch so many projects to make it a variable to begin with, even on Java services consuming libraries, unaware of Scala.

@hipsterelectron on top of that, you want ML in batch processing, in stream processing and in services so every very heavy context needs to be compatible. The freedom we had meant embarrassing things like Spark and Flink used incompatible Avro and Iceberg versions, different JDK version, different Java source version, different Scala versions.

@hipsterelectron at Netflix in the ML Platform the freedom of dependencies was an absolute nightmare in JVM world. Everything was service based but implying client libraries with heavy dependency trees. In ML you want every input you can get your hands on, so bring the entire universe together. Then you want to apply ML everywhere, thus take the whole universe everywhere else. Every single project with so much gradle boilerplate to resolve same issue all the time.

@hipsterelectron was the single version really such a downside at Twitter? Want to hear you perspective from EE side.

From Data Platform I helped many times with the typical suspects, jackson, guava and other libraries messing everything up. We had the main issue of Hadoop being outside of the monorepo and having surprise runtime dependencies and failures and during my time we switched every to classpath isolation. I felt we were the “pain sponges” and that was good.