How well can LLMs write COBOL?(bloop.ai) |
How well can LLMs write COBOL?(bloop.ai) |
It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases.
I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains. In any case, I'll try out this benchmark and see how it goes.
There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that)
Unless GPT-4's training data includes non-public code bases (I doubt it), it likely has rather little COBOL code in it
There’s definitely a lack of training data and ability (but unfortunately not confidence) in less widespread languages. It’s quite bad at pinescript, confusing versions of the language and producing unrunnable code, and being unable to correct it when given feedback
It's kind if interesting how far this could be taken with all kinds of valuable information from people and companies, especially code and business logic. Everyone wants to say AI this and AI that to keep up with the times and they all keep dumping all of this valuable data in for free.
Then consider the authentication method used tied to this data collection and if you are using a company emial, etc, it helps them weed out the garbage too.
I guess we can't fight progress...
Other people will try to build on OpenAI stuff, find that it's not quite good enough and OpenAI doesn't care to really make it good enough because it's a lot of work, and it won't be until we get competitors that take less sexy model tech and take the time to make it REALLY GOOD at certain things that AI really makes good on its promise. I'm guessing that will be driven by people taking open source tools that are ~80% of the way there and really building a system and domain logic around it to make it excellent.
> Gemini:
> JSON: In JSON, double quotes are used to delimit strings. If a double quote needs to be included within the string itself, it's escaped by doubling it (e.g., `"This is a string with a "quote" in it."`).
(And, just an editorial note: there are no backslashes in Gemin's response / HN is displaying it correctly. Gemini's response is nonsense — wrong, but also not even internally consistent.)
> Me:
> Does your JSON example parse as valid JSON?
> Gemini:
> Yes, the JSON example I provided ("This is a string with a "quote" in it.") is actually valid JSON.
> The reason it's valid is because the doubled quote within the string is interpreted as an escaped character, representing a single literal quote within the string itself. The JSON parser recognizes the double quotes at the beginning and end as delimiters for the entire string, not the escaped quote within.
(Its example, is, of course, quite malformed.)
I'd say I has mixed results, definitely chatGPT knows the language and can give examples but I've also had a lot of frustrating things it wasn't able to resolve.
It got a lot better with Fortran 95 and newer, but in the old world of e.g. lahey compilers and custom commands only available there, any LLM has failed me consistently to stick to these intricacies. I can't even blame them, when asking humans questions about these topics, you'll get all sorts of answers that are equally close to a correct solution, but almost never precisely correct.
On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems.
Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.
Edit: Well, maybe. With the example from the article it wasn't as successful.
I am from the C64 demo scene and in this regard ChatGPT is pretty useless. VIC tricks, raster timing - nothing a LLM can help with at the moment judging from my experience with ChatGPT 4.0 so far.
Same goes for Amiga and simple blitter access for scrolling.
LLMs will be very limited here unless they receive sensory feedback repeatedly.
Pure algorithms like sorting for example may be doable by GPT, but the mentioned machines are very creatively used to come up with effects.
Here are techniques reused, not so much code itself, which gets modified and optimized for every demo and often times recombined with other techniques.
Most techniques for the VIC are pretty well documented, but the timing as well as recombining them is the heavy lifting.
The real problem is the entire ecosystem around those systems. Remember, a lot of COBOL software dates back to a time before things like relational databases. You'll be working with flat files that might, if you're very lucky, have column and record separators and useful names/documentation explaining what they are. If you're unlucky you'll have to figure out field widths from the code and infer what the fields are based on their actual usage. Oh and if you get it wrong you just messed up something related to payroll or financial compliance; enjoy the punishing fines.
That kind of stuff, more than the language, is the reason nobody wants to touch old COBOL systems.
That could be a test of reasoning and reading comprehension
You'll be able to see the code multiplying a dollar value by 0.03 but not necessarily know that this is because of some statutory requirement of some Minnesota tax code that only applies to industries producing both left and right handed monkey wrenches but only if the company was incorporated before 1975. That obscure law isn't referenced in any documentation but was found by an accountant in 1982. The change was made to the code but only referenced in a paper memo with a small distribution list but all of those memos were shredded after being archived for ten years.
ChatGPT can't really help document code that's decades old and doesn't have any references to the why of the code. The how is straightforward but rarely as important as the why.
It's just intellectually dishonest to talk this way.
They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.
> They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.
It's not intellectually dishonest at all. It's an issue of conditioning. There's a class of developers that blindly copy and paste code from StackOverflow or the first hit on Google. They're the same class that will uncritically copy and paste ChatGPT answers.
ChatGPT is worse than SO because it's adaptive. If someone pastes in a SO answer and it doesn't immediately work the developer has to at least engage with the code. ChatGPT can be asked to refine its hallucination until it parses/compiles.
The class of developer blindly copying and pasting answers will not have the expertise to spot hallucinations or likely even fix the inevitable bugs they introduce. Additionally ChatGPT by its nature elides the source of its answers. At the very least a SO answer has some provenance. Not only the poster but some social signally through votes that the answer is legitimate.
ChatGPT answers don't have any of that. It will also happily hallucinate references.
Conditioning junior developers and learners to rely on and trust AI coding is setting them up to fail. It's also going to stunt their growth as developers because they'll never gain any domain knowledge. In the meantime they'll be unknowingly sabotaging products with legit looking but broken code.
Whether that makes a difference depends. If you are dealing with mainstream modern COBOL (like recent versions of MicroFocus or IBM Enterprise), it probably won't. If you are dealing with some obscure legacy COBOL dialect, odds are high there is some very helpful printed book which nobody has scanned.
However, for languages like Python, Java, C, C++, JavaScript, Go, etc, it also contains untold millions of lines of code slurped from places like GitHub. Whereas, I really doubt it contains anywhere remotely near as much COBOL code, just because you look for COBOL code on GitHub public repos, you will find very little – the vast majority of COBOL code is in-house or vendor business software, and few seem to want to make that stuff public – and what COBOL code GitHub has is mostly toy exercises or ancient stuff, not examples of significant contemporary production code. The only way OpenAI is going to get a substantial quantity of that is if multiple private parties (such as banks) give them access to their COBOL code bases – not impossible, but absent some public info saying it has happened, it seems more likely it hasn't.
I expect GPT-4 (or any LLM) is not going to perform as well on complicated programming tasks for COBOL compared to other languages. For more mainstream languages, it has millions of examples to help it do a better job, for COBOL it likely doesn't.
Also, nobody needs to do complicated coffee talks with cobol, it wasn't meant for it. What we do need a lot of is translating cobol to python or Java.
What you are saying may well all be true, but it doesn't contradict what I'm saying.
What people don’t get about cobol is that it is a language you can learn in a few days. It is meant to be readable for non-technical users so Cobol itself is not the issue. Once you get past the meaning of sections, and the implicit defaults that are different from modern programming languages, it is way easier than, say, C.
Its really not bad for them either. Verbose, but not bad...