I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools.

[gabrielesvelto@mas.to]

@csepp several fancy IDEs already have extremely sophisticate refactoring tools that understand the language syntax, e.g.: https://www.jetbrains.com/help/idea/refactoring-source-code.html

[elrohir@mastodon.gal]

@gabrielesvelto This parallels the discourse that says "LLMs are useful to automatically send template emails dozens of times a day". My Brother In Ohm, email templates have existed for decades at a fraction of the cost of a single token. You are just Dunning-Krügering yourself into thinking that the idea has just been solved today because you had never paid any attention to this problem until the day you wanted to search for an use case for a toy that you want to have an excuse for buying.

[aurisc4@floss.social]

@gabrielesvelto I've been presented a case where changes were quite trivial across many repos, but making those changes still required taking context into account. LLM was helpful.
But...
that same presentation showed logs of tool admitting of doing forced push when it was from the start specifically instructed not to do forced pushes.

Feels like we need sandboxed dev environments where these tools could not do dangerous things, as they themselves are bad at this.

[randamumaki@mstdn.social]

@dain @gabrielesvelto https://regex-generator.olafneumann.org

[csepp@merveilles.town]

@gabrielesvelto Yup, those are also pretty great.
Personally, I needed to refactor some C++ code that didn't fit any simple regex, so I ended up writing a Lua script to do it and did the rest of it by hand.
The only way I could find to reliably automate it would have been to write a custom clang-tidy pass, which didn't seem worth the effort.
I still wouldn't use an LLM for it, but I do wish there was an easier way to load the code model in a scripting language. To automate the refactor I did I would have needed to track arguments that are passed through variables or that come from function parameters, access non-C++ files (move strings to YAML), rewrite various forms of string concatenation to format strings, etc.

[gabrielesvelto@mas.to]

@aurisc4 I've done a mix of grep and send a lot of times to add context. If more sophisticate refactoring is needed there are tools that understand the syntax of practically any language in existence and can be used for direct manipulation of the ASTs. Every problem where the input is machine-readable can be solved in a faster, cheaper and more reliable way using tools that process the data directly rather than passing through a (very large) neural network.

[apgarcia@fosstodon.org]

@gabrielesvelto fun fact: the speed of sed comes from the fact that it leverages FSM-based matching under the hood. So, I say: FSM FTW!

[fourlastor@androiddev.social]

@gabrielesvelto I don't think the comparison is entirely fair tho. Both sed and syntax tree based editing are really powerful (and I use both when it makes sense), but if you need to do a one off migration you might be spending hours trying to figure out how to make it work right, while an llm will do a good enough job where you need to review the changes and fix a few mistakes, usually at the first try, without you having to actively spend time on it.

[gabrielesvelto@mas.to]

@fourlastor what about the time spent setting up the LLM, sandboxing it and then reviewing all the changes? What about the risk of the code containing prompt-injections that might be designed to introduce vulnerabilities or simply take over your machine or credentials for a state-sponsored attacker to use? What about the reliance on a single closed-source paid-for commercial tool? Those are a lot of disadvantages to make up for.

[dain@hachyderm.io]

@RandamuMaki @gabrielesvelto oh, that expression builder in the second step is really nice! wish it would then do match testing on more lines in further steps like how regex101 does

[fourlastor@androiddev.social]

@gabrielesvelto answering in order:

>what about the time spent setting up the LLM, sandboxing it and then reviewing all the changes?

This for what I'm working on is usually between 30 and 40 minutes, start to end (minus the time that the LLM takes to do its own work in its own git subtree, while I do other stuff). For context, claude doesn't commit, I review the changes locally (git is blacklisted). In my case this is been pretty stable on 100-150 tasks where I did the same kind of migration

[fourlastor@androiddev.social]

@gabrielesvelto prompt-injections

The project is closed source, and we don't have places where we randomly include text files, if someone IN THE COMPANY manages to introduce malicious code, imho they'd just infect gradle instead of hoping on someone running an LLM to trigger something (other than devs having access to only what they need). State sponsored hackers specifically are really not in my list of things I can defend from, be it from LLMs or whatever introduced attacks

[fourlastor@androiddev.social]

@gabrielesvelto What about the reliance on a single closed-source paid-for commercial tool

On this I 100% agree, you shouldn't RELY on it. I am confident that I can make the same changes myself (in some cases I did because it was clearly less time consuming than making an LLM do that), if tomorrow these tools disappear I am sure I will be comfortable working without them (as I do for example for my OSS/hobby work, where I can't really justify paying for the subscription)

[gundersen@mastodon.social]

@gabrielesvelto the developers of TypeScript have decided not to implement refactoring tools because the refactoring can be done by LLMs...

[hipsterelectron@circumstances.run]

@gabrielesvelto i spent the last week using sed to produce an entire module system for a prototype. lovely piece of software that expands the meaning of structured data. not at all perfect but if we're comparing it to statistical approaches it at least has the benefit of determinism

[fourlastor@androiddev.social]

@gabrielesvelto a counter example: one migration I needed to make was to migrate java serializable to parcelable. That was a GREAT candidate to be worked on by modifying the syntax tree. I created a small throw away plugin in intellij which did the work, removed the extension, added the annotation and ran on thousands of files in a few seconds.Imho trying to find the most appropiate tool for the task at hand is important, and having an all-or-nothing mentality (on either side) isn't constructive

[adingbatponder@fosstodon.org]

@gabrielesvelto For fun I tried writing rust code with claude code. The code took an age to compile when it worked (do we call it build?). The project took months and so the code got large & was slow to build. Claude was able to refactor it (after it worked) to build 10 times faster. That is not mechanical as you mention... but was really challenging. Mechanical refactors it does 100 times better still of course, because it seds too yes, but it can check the new syntax & test build each change.

[keithpjolley@discuss.systems]

@csepp @gabrielesvelto tbf, in all likelyhood it wouldn't be `sed` that fails. it would be the inputs to `sed` that failed - garbage in, garbage out.

[patricus@gts.posix.live]

@gabrielesvelto not really, it is not on my computer.

[crazyeddie@mastodon.social]

@fourlastor @gabrielesvelto It's not a use sed or use LLM scenario here.

Sed isn't a refactoring tool. There are plenty of actual refactoring tools that don't use LLMs. I was using them before LLMs were invented and no, fucking sed isn't the same thing. I'm rather hoping that wasn't actually a serious comparison

Mechanical refactors are deterministic algorithms. If the conversation is about sticking AI in that it's probably nonsense and you can leave without fearing you'll miss anything