Next-generation AI models(tabnine.com) |
Next-generation AI models(tabnine.com) |
I use Copilot for its multi-line code generation. It's vastly superior. (Sorry Tabnine). Gotta give credit to Tabnine for kicking off this space -- they had a multi-year headstart on Copilot and it was pretty cool when it first launched, but it just doesn't come close to Copilot. OpenAI is kicking butt here.
Having said that, I still find value out of Tabnine -- I keep it installed it powers the pop-up autocomplete (eg Intellisense replacement). The Tabnine suggestions are immediate (as opposed to Copilot which takes a few seconds), and in a lot of cases it's super helpful for completing a variable name after 1 character, etc.
If I ESC out of the Tabnine suggestion, then Copilot takes over and a few seconds later I get a full line/multi-line suggestion.
I dunno - it's been working really well for me. Peaceful coexistence :)
...just kinda joking: for me, one major use case of this thing is a faster alternative to googling some syntax or method name.
Copilot uses a huge model and a ton of GPU's (and an unlimited budget) to focus on really long completions. This is a very specific choice to go for long completions.
We have worked hard to solve this problem differently and with the new generation you will see longer multi-line completions while also giving developers the options to use shorter suggestions (perhaps together with Copilot)
But almost an year has passed since I used this so I don't know if they have had a major update and the results are improved, but previously 1 out of 10 tabnine suggestions were barely useful, compared to 9 out of 10 with copilot.
Since it's a paid offering now I really hope they've upped their game.
Tabnine CEO Dror Weiss, wrote this detailed thread explaining some of the key differences (both in technology and user-approach) between Tabnine and Copilot: https://twitter.com/drorwe/status/1539293063117516801
At some point I started using Language Server (mostly Typescript and Python) suggestions alongside Tabnine and it started to feel like it was just getting in the way. In the end I got rid of Tabnine. I haven't tried Co-Pilot yet but may give it a go when I find some time.
I gave it a fair chance, let it index my private personal projects, ran it on my personal machine to see how it worked. The results are... not great. I'm using IntelliJ, and Tabnine is arrogant enough to believe that their autocomplete is superior to the libraries autocomplete, even when completely wrong or it has no idea of its autocomplete. These are opensource libraries.
Their inline suggestions (per what Github Copilot) isn't great either. I've tried their different strength levels (their UI for their product is fantastic) and it's not great.
Github copilot when I first started was pretty good, over the last few months it's become amazing. I'm going to be giving my money to Github over Tabnine, but would prefer to give it to Tabnine.
I haven’t tried Copilot for editing Java, because I’m no longer working in any Java code bases, but I wonder if it does a better job.
I have yet to compare to copilot.
File fileName =
File fileName = new File(path);
That story gets even weirder with the 3rd link on that page whose license is also "other" but this time the "view source" link goes to https://www.tabnine.com/web/assistant/code/rs/5c781237e70f87... . I find that weird for at least two reasons: (a) it clearly says "This snippet was taken from github" and has a GitHub style "org/repo" nomenclature, but doesn't link to the actual repo (b) at the very top of that file is the boilerplate Apache 2.0 license header
Finally, one should be very cautious about ever linking to "master" URLs, since the branch can get nuked if the repo owner decides to go with the "master to main" rename, it can lead the user to a copy of the file that is almost guaranteed not to be the same sha as the one Tabnine indexed, and related to that the repo can undergo a license change (FOSS to BSL is a very common one) leading to some complicated discussions
- Tabnine's local per-repository model shines best when used on big monolithic repos with lots of similar or boilerplate code. By learning on the same code it's used for it IMO does much better job at detecting local patterns. Also local-model means pretty "safe" from security POV for corporate use (I can't imagine big corps like Google and Amazon allowing use of Copilot for proprietary code).
- Back in they free tier of Tabnine was pretty useless compared to Pro because it severely limited model size. If it's still the case now I hope they have a long evaluation period.
- From usability POV, unfortunately it doesn't provide non-intrusive experience out of the box, which is very annoying and I believe is a reason for many users abandoning it pretty quickly. I had to modify both VSCode extension and VSCode itself (!) so that Tabnine's completion suggestions won't interfere with others.
It was posted by the VP of Business Development of Tabnine. Is this really organically ending up on the front page or are some bot/vote rings in control?
@dang shouldn't the title at least be changed to "Announcing our Next-generation AI models"?
What they're doing is impactful, impressive - and so eclipsed by an unreasonably well-resourced competitor that they don't even get given much credit.
"Not as good as CoPilot" really doesn't say enough about something that's still very impressive.
Personally I thought it was cool but not game-changing, and moved on when I got into CoPilot. I can't comment on the next-gen models.
But this is one thing that worries me about A.I speaking as an indiehacker.
Being scrappy won't soon cut it anymore. Right now you can bootstrap a solid product for $15K, at least a MVP. Soon with the amount of GPU and other resources required to train these models you'll need a min $1M budget and no amount of jugaad will be able to cut it.
Last time I checked(end of last year) Tabnine was a far inferior product.
If software is doing it, it can't be bad, I guess.
I don't when was the last time you've tried coding with Tabnine in the background, but I sure hope that now, with the release of our next-generation abilities you'll give it another go :)
Is it possible to use TabNine without using the random executable from https://update.tabnine.com/3.2.28/x86_64-unknown-linux-musl/... (Or is there the source available?)
Are there any instructions on to make it work with any other editor (GNOME-Builder in my case)? Is there Vala support?
Edit: Found this: https://github.com/codota/TabNine/blob/master/HowToWriteACli...
Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model, and only Microsoft can host the model due to the enormous amount of computing resources required for training and inference.
Tabnine, after comprehensively evaluating models of different sizes, favors individualized language models working in concert. Why? Because code prediction is, in fact, a set of distinct sub-problems which doesn't lend itself to the monolithic model approach. For instance: generating the full code of a function in Python based on name and generating the suffix of a line of code in Rust are two problems Tabnine solves well, but the AI model that best fits every such task is different. We found that a combination of specialized models dramatically increases the precision and length of suggestions for our 1M+ users.
A big advantage of Tabnine’s approach is that it can use the right tool for any code prediction task, and for most purposes, our smaller models give great predictions quickly and efficiently. Better yet, most of our models can be run with inexpensive hardware.
Now that we understand the principal difference between Microsoft’s huge monolith and Tabnine’s multitude of smaller models, we can explore the differences between the products:
First, kind of code suggestions. Copilot queries the model relatively infrequently and suggests a snippet or a full line of code. Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose. Similarly, Tabnine Pro also suggests full snippets or lines of code, but since Tabnine also uses smaller and highly efficient AI models, it queries the model while typing. As a user, it means the AI flows with you, even when you deviate from the code it originally suggested.
Second, ability to train the model. Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on. Tabnine can train a private AI model on the specific code from customers’ GitLab/GitHub/BitBucket repositories and thus adjust the suggestions to the project-specific code and infrastructure. Training on customer code is possible because Tabnine is modular, enabling the creation of private customized copies.
Third, Code security and privacy. There are a few aspects of this. Users cannot train or run the Copilot model. The single model is always hosted by Microsoft. Every Copilot user is sending their code to Microsoft; not some of the code, and not obfuscated - all of it. With Tabnine, users can choose where to run the model: on the Tabnine cloud, locally on the developer machine, or on a self-hosted server. This is possible because Tabnine has AI models that can run efficiently with moderate hardware requirements.
In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model. We don’t send to our servers any information about the code that the user writes and the suggestions they’re receiving or accepting.
Fourth, commercial terms. Microsoft currently offers Copilot only as a commercial product for developers, without a free plan (beyond a free trial) or organizational purchase. Tabnine has a great free plan and charges for premium features such as longer code completions and private models trained on customers’ code.
For any coders who might think this article will help them understand AI, the actual title is:
Announcing our Next-generation AI models
Perhaps it has changed since being posted but it's just a plug.
I actually really liked Tabnine but sometimes the tons of suggestions got annoying. I upgraded my IDE and forgot to add copilot but didn't feel like I needed to. Maybe that's a mistake... but sometimes it felt like it was getting in the way and I did encounter a few times it would create "bugs" which I would have to spend time going back and fixing.
Not trying to knock these though. They're both very impressive tools.
I have no idea how Tabnine builds their models today, and how they perform compared to Copilot. I guess one advantage could be lower latency in suggestions that they claim come from training smaller more specific models. But the way I find Copilot working for me is that it thinks about the same time I need to think and then it makes a suggestion for a good chunk of code. If my thinking and Copilot's thinking match up then I can save myself a good bit of typing.
https://docs.github.com/en/repositories/managing-your-reposi...
https://github.com/Vedenin/useful-java-links
When master branch is renamed to main, GitHub redirects any old links. https://github.com/github/renaming#renaming-existing-branche...
1. Initially Tabnine's auto-completion was triggered on any character, which best leveraged Tabnine's power but also had inherent problem: when Tabnine was triggered on non-letter character it sometimes prevented Vscode from showing suggestions from other completion sources (LSPs, snippets). There is a discussion in https://github.com/codota/tabnine-vscode/issues/6 with me explaining that the only viable solution is to reduce set of trigger characters to letters only. In the end a fix was pushed that reduced the set of trigger characters, which made the problem less likely but still not solved. The are numerous duplicates of this issue on Github.
2. Another problem is when Vscode has auto-completion suggestions from Tabnine and other sources (LSP, snippets), it frequently puts Tabnine's at the top of list. This is a big no-go for me because most of the time just want to complete the identifier (class field, method etc.). Modifying just the extension code didn't help so in the end I had add a small patch to Vscode itself, which gives lowest score to Tabnine's candidates: https://github.com/sergei-dyshel/vscode/commit/ee73034b9ec6c....
I must admit that both problems can be practically solved by new "inline auto-completion" mechanism in vscode which looks very promising for AI-based completion in general. I'm looking forward to evaluate it.
> Copilot queries the model relatively infrequently
Huh? It queries whenever you stop typing.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose
This is at best false, at worse disingenuous. Yes, it won't insert a completion in the middle of a line, but this is trivially solved by inserting a newline in the middle of the line and then triggering a completion at the end of the line you just created.
> In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model.
Copilot has this option too...
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
I mean, anyone who has used Copilot for more than 5 seconds would know that this isn't true. Copilot does a fantastic job at reading the current file and providing suggestions relevant to the file, rather than some hypothetical "average of GitHub"
That's relatively infrequently - TabNine offered me accurate completion while still writing my code, whereas with Copilot I not only have to wait for it to return the answer, I also have to hope it knows an answer. If it doesn't, too bad, lost time for nothing.
I didn't use Tabnine but anecdotally heard it is not as good to guess whole methods the way Copilot is. Now I understand why.
The only suggestion I would make, maybe lower the price, MS is bigger player, they pretty much set the standard now, if they have lower price even symbolically, it would hurt you more.
I personally am happy with Copilot, had some really wonderful moments and savings in time searching for proper method syntax is where I get a lot of benefit. I should spend time to get to know Tabnine as well so not to be ignorant on this new technology.
> Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model
It's true that only Microsoft can train the model—for now. But GPT-3 is a monolithic model, and OpenAI allows fine-tuning. There's nothing about the architecture that prohibits Microsoft from offering customers the ability to fine-tune the model.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose.
Here's a screenshot of GitHub Copilot suggesting code in the middle of a line:
https://imgur.com/gallery/Jp2qg7X
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
If the model is big enough that seems to be an advantage, not a disadvantage. At least, when I use GPT-3 to create a 4chan greentext, or an X-Files script, or a presidential speech, or an article by Hunter S. Thompson (or a presidential speech by Hunter S. Thompson) it doesn't particularly feel like it's averaging things out.
What I see is Copilot suggesting an entire line. Or does the suggestion start on the Err line with the "{" ?
All though in the past I found the limited suggestions offered with the Tabnine free plan too limiting to be of much help.
I actually installed a third-party extension which implements their autocomplete differently.
So weird that they didn't hear such major/consistent feedback. (Maybe they didn't want to tank their 'suggestions taken' number?)
Ok, but how is anyone supposed to know that? There's no verbiage anywhere on those search screens saying that
I thought using the code search with its auto-complete widgets would be a "try it in the browser" version of the product, but what I'm hearing is no, it's just some separate toy trying to be beat Sourcegraph or something?
I would find it a more compelling pitch to say tabnine will help me and my teammates follow more general best practices, though I understand how subjective that is.
Do you models incorporate any sort of time-based weighting, where more recent code is weighted more heavily? Seems like that would also help this issue, as more recent code tends to be closer to what we hope future code looks like. But then how do you let best practices evolve over time?
If nothing else the HN mantra is not "That's how we've always done it."
I've watched people use co-pilot but and I'm not 100% convinced it's a good thing but I'm open to it. The problem I've seen is that in some cases it generates some really bad code, code that could've been replaced by usage of a standard library.
What I think it's really good at is helping to learn new languages. Getting idiomatic code generated, even if you throw it out eventually is a great way to get a feel for a new language without having to jump back and forth between the editor and the browser.
I'm not against the idea of ai coding tools on premise, but it did strike me as kind of funny to imagine someone using two such tools at the same time.
I'm more of a minimalist, and would generally prefer to have lean tools which get out of my way. The idea that every keystroke involves a back-and fourth with a server, and that my basic coding workflow would need an internet connection is not something which is super appealing to me personally.
Any tool that can meet that ends should be considered.
Put another way, users don't care what anyone (i.e., in engineering) does or does not know. They're not going to wake up and say "I only want to use products that don't use Copilot." Nobody cares about any ethos, etc.