Post by @alan@mindly.social
This analyst's report from Apollo about AI uses the term "S&P 493" to refer to the "S&P 500 minus the Mag 7" - because if you take out the AI companies, the S&P 500 is tanking. https://www.apollo.com/wealth/insights-news/insights/daily-spark/no-signs-of-ai-payoff-outside-the-tech-sector
@pluralistic At the tech conference I was at last week, which was of course 2/3 AI, it was often taken as self-evident that AI had given us huge gains. I should look up some of those people and send them this ...
@pluralistic One talk, that was quite interesting, was from a company with 180 developers that did a slow and careful rollout of AI, mainly due to high demand. They did a self-assessment, and developers claimed increased productivity. They said it was too early to say if it was actually true though, and promised to come back with results (presumably another year) if they had any.
It's the same where I work. Developers *feel* more productive, but I have yet to see any numbers backing it up.
@veronica @pluralistic I have looked at two AI tools, one is a simple line completion model that guesses at what I'm about to type. It's a simple model that runs locally and is right about 80% of the time. Overall I think it offers a benefit.
The other is a big model that tries to figure out the next few lines, and even after letting it at a code base for some time, the guesses are wildly inaccurate, and mess with keystrokes, so I wind up having that inaccurate code injected, which means I then need to remove it before typing what I was already planning to type. This has lowered my productivity. I have used this model to generate test cases; about 80% of the cases made sense, the rest needed minor tweaks, so there's a potential use case there.
I will try getting an advanced model to build a small application, but at this point I expect I'll spend as much time fixing the generated code as it would have taken to write it in the first place.
One place where it might be useful is generating code in a language I am familiar with but haven't mastered. At that point the logic would still be auditable, and I could use that review to raise my level of expertise.
We learn by doing. Reviewing isnβt doing, or you could just read existing code to learn a language.
Evidence I have seen so far is that AI will stop development of expertise.
@ThreeSigma @veronica @pluralistic with a few exceptions (eg. APL, Lisp) a new language is just syntax and a handful of different operators. I say this as someone with experience with maybe two dozen programming languages, many obscure.
@ThreeSigma @veronica @pluralistic That said, your concern is valid for people just entering the field. There's little market for junior developers if an AI can fulfill that role cheaper and faster.
How we educate developers is going to have to undergo radical change.
@veronica @pluralistic @alan
Then why need the AI at all? Youβre being inconsistent. Either the language is trivial and you could just do it yourself, or itβs tricky and the AI saves you time (that I claim is valuable learning).
Iβm also an old hand at this but Iβm still learning or I get bored.
@ThreeSigma @veronica @pluralistic It's a case of learn by doing. As an example, I might ask for an implementation of something I've written in one language to be rewritten in Zig. Then I'm going to get a big pile of code that is a rough approximation of the port. I could go burning more credits or tokens or whatever to have the AI refine it into something with a bunch of subtle and difficult to find bugs, or I could use it as a baseline and start to explore, rewrite, and debug from that first pass.
That's my preferred modality of learning. Reading a language reference manual is a little dry, but now I'll be digging into it repeatedly to get an understanding of the nuances of the language. Instead of tutorials that circumnavigate quirks and edge cases, I'll run into something I don't understand and go digging until I do. At the same time I'll gain familiarity with build and debugging tools, assorted libraries, the ecosystem in general.
And if I'm lucky, in a reasonable amount of time I'll also have my application implemented in an environment more suited to the job. Most importantly, I'll actually know how that application works.