So you’d basically install Ollama, download one of the versions of this model off HuggingFace, create a Modelfile since this isn’t in the default Ollama repo, and then Ollama can answer prompts with the model. Modelfiles are very simple, based on Dockerfiles. It takes like 15 seconds to make one if you aren’t messing with the various parameters.
Once it’s in Ollama, just get one of the various GPT plugins for VSCode and give it the Ollama URL (http://localhost:11434 by default). I use continue.dev but there are many.
Continue takes over the tab autocomplete with the LLM, and has a chat window on the right where you can use keyboard shortcuts to copy code into the prompt and ask it to edit/generate code or ask questions about existing code.
the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
And you can search for any GGUF on huggingface
Where would I start if I wanted to use a model programmatically ? Like let's say I am building a chat bot. I have a large data set of replies I want the model to mimic, and I'd want to do this in Python. Of course, I'd probably use a different model than Granite.
> Our process to prepare code pretraining data involves several stages. First, we collect a combination of publicly available datasets (e.g., GitHub Code Clean, Starcoder data), public code repositories, and issues from GitHub
Citation needed
All I've seen from them in my professional experience is actually legacy mainframe maintenance.. Not shovelware, but very far from hardcore tech.
They've been doing "AI" for ages. Notably Watson over the last couple of decades or so.
I've not seen any proper evaluations for Granite against, say, Llama or Mistral.
Until we do it's probably too early to say they can't compete, at least in some areas where others perform poorly.
Previous Granite models were on the level of first llama in my benchmarks.
I’m expecting this version to be roughly comparable to llama 2
Did you even read the benchmarks they post on that link? Assuming they're not outright lying, their 8B model is superior to Llama/Mistral models of the same size for coding tasks.
Before doing that, I would start basic. Pull llama3 and see what it does with your prompts. You may be surprised how much is already in there and just not need to involve your own data at all. If that doesn’t work, check HuggingFace to see if someone has already made a model/fine tune/LoRA for what you’re trying to do. There are many, eg I found a Magic The Gathering rules model the other day.
If those fails, or you just want to play with your own data, you’ll need to figure out what “mimic” means.
If the model does okay with generating content but the content is factually wrong or missing background, you may be able to just do RAG (retrieval augmented generation). Basically running your documents through an AI that converts them to embeddings (some kind of vector, I don’t understand how they work). Then when you run a query, you can search for related embeddings and pass them to the model so that it “knows” the content that was in the document. This is the easiest; open-webui (the Ollama web chat interface) has some RAG support. Danswer is open source and built from the ground up to do RAG, and has built in support for ingesting from Slack, Drive, etc, etc. OpenAI also has embedding as a service.
A step up from that is making a LoRA. To my novice eyes, LoRA’s are basically a diff of the models parameters or weights. So rather than training a whole new model, you just add deltas to an existing one. These let you “teach” the model something while preserving the base generation capabilities of the underlying model. Ie you won’t have to worry about making sure you feed it enough data that it can speak English properly, because it gets that from the base model, you only have to give it enough data to speak about whatever you’re training it on.
If that doesn’t make any sense, go check CivitAI for Stable Diffusion (image model) LoRAs. The effects are way more obvious on image AIs.
Anyways, LoRAs are trained so you’re into training there. I think HuggingFace has tools that make this easy, but I don’t know enough to say anything with confidence.
The last option, which you almost certainly don’t want, is to train a new base model like llama3. You’re starting from 0 there; you have no existing model so you will have to teach it everything. It will take a ton of data, it will take forever to train, and it will likely be much worse than even randomly clicking models on HuggingFace. Meta has spent who knows how much on Llama and it still hallucinates.
If you end up training, you’ll probably end up doing it in the cloud unless you have tons of VRAM doing nothing. Prices are pretty reasonable, I think A100s are around $2/hr. I don’t know how to gauge how long it needs to train, but I believe it’s related to the amount of data you’re training on. I believe it’s pretty reasonable for LoRAs though, I’m guesstimating in the $20-ish range?
Edit: oh, and I’m not affiliated in any way, but I found out last night that Fireworks’ new function calling model is free while it’s in beta, which is a neat/fun thing to play with. https://fireworks.ai/blog/firefunction-v1-gpt-4-level-functi... it’s also open weights if you want to run it locally, but it’s a 40B model so I can’t on my 3060
Their customers aren’t going to build their own RAG and agent frameworks, vector DBs, data ingest pipelines, finetunes, high scale inference serving solutions, etc, etc.
There’s an incredible amount of stuff to buy.
PALO ALTO, Calif. – IBM defined at (trade show ed.) Hot Chips a new interface for the 2020 version of its Power 9 CPUs. The Open Memory Interface (OMI) will enable packing on a server more main memory at higher bandwidth than DDR, and as a potential Jedec standard could rival GenZ and Intel’s CLX.
OMI basically removes the memory controller from the host, relying instead on a controller on a relatively small DIMM card. Microchip’s Microsemi division already has a DDR controller running on cards in IBM’s labs. The approach promises to deliver up to 4TBytes memory on a server at about 320GBytes/second or 512GB at up to 650GB/s sustained rates.
https://research.ibm.com/blog/albany-semiconductor-research-... etc
IBM doesn't have fabs, but they still do R&D into semiconductors that very much target future commercial processes. They do a fair bit on quantum computing too, to name just a couple of things.
If IBM split off half of their mainframe division and let some competition get going I think the segment could actually be something to contend with.
The basic idea of the IBM mainframe is almost perfect for what a lot of companies actually need (massively reliable hardware to support lots of middling software; most work is shunting data around) but everyone knows they're going to get locked into IBM.
There really aren't a lot of companies out there that can claim to do similar (and of course besides s390x, an ancient and venerable CISC, IBM also has Power, so they are doing this 2x over). You'll find a lot of IBM employees contributing to what I'd consider "hardcore" tech like LLVM and the Linux kernel as a result, because they genuinely have a large amount of expertise in those and similar areas. And here I'm not even really including Red Hat, but if you include them then they are even more overweight in the hardcore tech category.
If anything, a lot of the rest of the tech industry has left "hardcore tech" behind due to efficiency concerns as a result of a longrunning industry wide process of consolidation and commodification that IBM has resisted for obvious reasons. IBM is hardcore to a fault if anything.
TLDR: I actually think IBM punches above their weight in the "hardcore tech" area so long as our definition is sufficiently low level rather than say, cloud services, in which case fair enough you can probably fairly say they suck at that.
Here I've also chosen to entirely ignore IBM research.
That data issue is important enough for some companies to pick mediocre model over llama or mistral.
IBM and other big players are vigilant about these things, and this is what companies pay for.
Their software may not be better in some metrics, but they're cleaner in some and their support contracts allows people to sleep tight at night.
This is what money buys. Peace of mind and continuity.
There will be market for their services. Maybe a different one, but there will be.
The gist is still current, but you need to fill in AWS as the current uncontroversial choice.
And more importantly, IBM will guarantee it in the case that they're wrong. _That's_ what companies pay for.
So will OpenAI, according to Sam Altman. Can they be trusted?
IBM has proven itself in various ways over the years, OpenAI hasn't.
While IBM is a behemoth of a money making machine, they put money where their mouth is. OpenAI does not.
So I'll trust IBM, but not OpenAI.
This support contract stuff: what are you talking about? You download these models, you use them. What would you pay for? It’s not clean data, they say it’s clean: why would I pay liars? Let’s game out the indemnity idea. I pay $10k/mo for 12 months. Then OpenAI loses v. NYTimes, ruled LLM training is not fair use, need express permission. IBM pulls the models. What the hell did I pay $120k for? And by the way, you can pay a law student 1 beer to tell you OpenAI is going to lose because of Warhol v Goldsmith. You can do whatever you want with your money, but I personally would not waste it on worthless indemnity.
I know the Stack is not clean, because they included my fork of GDM's greeter, which is GPL licensed.
My words about IBM was in general. I can't tell anything about their models, because I didn't see mention of "The Stack", and I don't know what their models are based on.
On the other hand, IBM doesn't like risks from my experience, so they would play it way safer than other companies.
If their data is not clean to begin with, then shame on them, and hope their AI efforts burn to the ground.
BTW, LLM training is not fair use. For start, Fair Use's definition automatically excludes "for profit" usage. Just because OpenAI has a non-profit part and training done here doesn't make them immune to consequences of for profit operations.
