Using aligned word vectors for instant translations with Python and Rust

Using aligned word vectors for instant translations with Python and Rust(instantdomainsearch.com)

112 points by beau 5 years ago | 35 comments

beau 5 years ago |

We've released the underlying Rust implementation here: https://github.com/InstantDomain/instant-distance with Python bindings at https://pypi.org/project/instant-distance — feedback welcome!

habibur 5 years ago | |

For Linux, in the Makefile change the copy command to

cp target/release/libinstant_distance.so instant-distance-py/test/instant_distance.so

and it works. Built and running. The main tree was MacOS only.

Here's resource consumption in a sample run.

Time: 4.49s, Memory: 1552 mb.

Single word. Three langs including en.

planckscnst 5 years ago | | |

How did you figure this out? I've done lots of Linux software build troubleshooting as a result of using Gentoo, BuildRoot, and pacaur, but this doesn't ring any bells for a common issue

arbol 5 years ago | |

Did you try spacy's most similar method? It's written in cython so is presumably quite fast as well. Thanks for the rust implementation though, I will most likely use this.

Fiahil 5 years ago | |

I’ve not much to say on the actual lib, it seems great! However, don’t feel compelled to put all your rust code into a single lib.rs. You can split your work into several files and use ‘pub use’ and ‘mod’ in lib.rs to re-export your functions & types into a public API of your choosing.

cargo check and format time might also slightly improve!

ibraheemdev 5 years ago | | |

Funny, I often say the opposite. Don't feel compelled to split up your lib.rs. It's really refreshing to see a nice, compact library in one or two files. Much easier to follow, especially over "type per file". Of course, there are limits, but for a small lib like this, I personally would keep it in a single, or maybe two files.

dochtman 5 years ago | | |

I have a fair bit of experience writing Rust code and the current status is totally deliberate. I find module file sizes of about 400-800 lines of code optimal in terms of my ability to find things vs the unnecessary complexity of having to skip around files when changing something that touches an API boundary.

maeln 5 years ago |

This webpage use a significant amount of CPU constantly for no apparent reason (as far as I can see it is mostly a static webpage). What the hell ? Is it mining crypto in the background ?

maybevain 5 years ago | |

At a quick glance it seems like some React component is constantly re-rending.

Quick glance in this case: took a couple second snapshot on the Performance tab and saw a lot of React related calls.

beau 5 years ago | |

Sorry, this page had a useEffect/setState render loop. We are running react@experimental with concurrent mode, and missed the error. Rolling out a fix now. Thanks!

chakkepolja 5 years ago | |

Compiling rust.

denysvitali 5 years ago |

> For example, here are the results of translating the English word "hello":

> Language: fr, Translation: bonjours

> Language: fr, Translation: bonsoir

> Language: fr, Translation: salutations

> Language: it, Translation: buongiorno

> Language: it, Translation: buonanotte

> Language: fr, Translation: rebonjour

> Language: it, Translation: auguri

> Language: fr, Translation: bonjour,

> Language: it, Translation: buonasera

> Language: it, Translation: chiamatemi

Is it just me or these machine translations are worse than ... Google Translate?

beau 5 years ago | |

These results are less accurate than Google Translate. But they are far faster to get, and far less expensive to generate: https://cloud.google.com/translate/pricing — our goal is here is speed. We want to search through many possibilities as quickly as possible.

The word vectors have been aligned in multiple languages. Using an approximate nearest neighbor search we are able to find the nearest vector to the input in multiple languages very quickly.

To keep the example simple, we did not try to filter the data through hand-built language dictionaries. In fact, we simply drop words in other languages that also appear in the English .vec file. Words like "ciao" appear frequently enough in otherwise English sentences that the example code drops it from Italian, and so is not shown in the results:

% curl -s "https://dl.fbaipublicfiles.com/fasttext/vectors-aligned/wiki..." | grep -n ciao 50393:ciao 0.0120 ...

One improvement would be to filter out any words that do not appear in a hand-curated dictionary instead of filtering out words that already appear in English. We decided not to show how to do this because we'd already introduced a few concepts, like aligned word vectors, approximate nearest neighbour searches, and wanted to keep the example as simple as possible.

toxik 5 years ago | |

Google Translate is state of the art, so I’m not sure why that would be surprising. That said, is there something wrong with the translations offered?

fulafel 5 years ago |

Was disappointed this can't translate from Python to Rust.

dukeofdoom 5 years ago |

Can something like this be done to compare/translate subsequences COVID genetic code to SARS and other virus genetic codes. Would be interesting how much overlap there is. And would further the research into where it came from.

Full genome of COVID-19 is available:

https://www.snapgene.com/resources/coronavirus-resources/?re...

nestorD 5 years ago | |

Bioinformaticists have been able to do that with traditional algorithms for years (dynamic programming gets you a long way to compute an edit distance for example).

It is probably the first thing that was done once the COVID-19 genome was made public. A quick googling gave me that summary of the results: https://www.news-medical.net/health/How-Does-the-SARS-Virus-...

dukeofdoom 5 years ago | | |

Great article, thank you

mattkrause 5 years ago | |

It sounds like you're thinking of "sequence alignment", which is a pretty standard bioinformatics tool.

BLAST (=Basic Local Alignment Search Tool) is one common version, and the NIH'S NCBI has a variety of nice online tools here: https://blast.ncbi.nlm.nih.gov/Blast.cgi

Note that it does take a little bit of background knowledge to interpret:some motifs are just really common, others are shared.

PaulHoule 5 years ago |

Nice example.

The short text and that fact that your application would tolerate or celebrate catchy neologisms plays to fasttext's strengths.

beau 5 years ago | |

Thank you!

shakow 5 years ago |

> fast translates to vite in French

Only as an adverb, it should be "rapide" otherwise.

aitk 5 years ago |

At first glance at the title, I thought it was translating Python code to Rust code.

adsharma 5 years ago | |

It may not be a bad candidate for writing the rust part in python and then running it through py2many to generate the rust.