Post by @veronica@mastodon.online
This analyst's report from Apollo about AI uses the term "S&P 493" to refer to the "S&P 500 minus the Mag 7" - because if you take out the AI companies, the S&P 500 is tanking. https://www.apollo.com/wealth/insights-news/insights/daily-spark/no-signs-of-ai-payoff-outside-the-tech-sector
@pluralistic At the tech conference I was at last week, which was of course 2/3 AI, it was often taken as self-evident that AI had given us huge gains. I should look up some of those people and send them this ...
@pluralistic One talk, that was quite interesting, was from a company with 180 developers that did a slow and careful rollout of AI, mainly due to high demand. They did a self-assessment, and developers claimed increased productivity. They said it was too early to say if it was actually true though, and promised to come back with results (presumably another year) if they had any.
It's the same where I work. Developers *feel* more productive, but I have yet to see any numbers backing it up.
@veronica @pluralistic I have looked at two AI tools, one is a simple line completion model that guesses at what I'm about to type. It's a simple model that runs locally and is right about 80% of the time. Overall I think it offers a benefit.
The other is a big model that tries to figure out the next few lines, and even after letting it at a code base for some time, the guesses are wildly inaccurate, and mess with keystrokes, so I wind up having that inaccurate code injected, which means I then need to remove it before typing what I was already planning to type. This has lowered my productivity. I have used this model to generate test cases; about 80% of the cases made sense, the rest needed minor tweaks, so there's a potential use case there.
I will try getting an advanced model to build a small application, but at this point I expect I'll spend as much time fixing the generated code as it would have taken to write it in the first place.
One place where it might be useful is generating code in a language I am familiar with but haven't mastered. At that point the logic would still be auditable, and I could use that review to raise my level of expertise.
@alan I find the VSCode auto-complete feature highly annoying and I mostly just keep it off. Some of the stuff I work on is pretty obscure, like parsing streaming binary subtitle data from DVD/Bluray, and it keeps suggesting code that is obviously stolen from somewhere else (and also doesn't fit my use case). I've also had it echo back my own code too, when it's relatively rare stuff. Funnily, it came with a license warning last time. I cannot recall them asking to ingest my code without credit!
@veronica The sort of stuff my line completion (well, line plus a bit) is I type, in a loop, "if variable is empty" and it suggests completing the if with a continue statement, so a tab saves 10-11 keystrokes. I can live with that.