Node.js and Python Interoperability

Node.js and Python Interoperability(fridgerator.github.io)

94 points by fridgamarator 7 years ago | 21 comments

swordfeng 7 years ago |

I was once trying to do something even crazier: exposing full python accessibility to node.js, using proxies. So that you can use python objects and functions as native javascript objects and functions. It is feasible because the languages and runtimes share a number of common designs.

There were several tough things however. The first was error handling. Yes you can catch exceptions in native code and convert it to exceptions in the other language, but it's not straightforward to keep stacktraces. The second was circular references between runtimes. Since references across boundaries are global, garbage collector on either side could not reclaim circularly referenced objects. Although this could be resolved by manually breaking up the circle, it could be better to have weak references. (Or maybe other utilities, idk what would be more elegant.) The third was that js has no operator overloading, so I had to use .__add__() for example to call the python add operator.

One line example: https://github.com/swordfeng/pyjs/blob/master/test/jsobject.... It's a toy project I did years ago and not even compiling now. Also I was wondering if anyone really need to do things in this way, given there are bunch of popular and stable RPC libraries. But I was happy to learn something about underlying cpython and v8 from it.

masklinn 7 years ago | |

> The third was that js has no operator overloading, so I had to use .__add__() for example to call the python add operator.

I expect this wouldn't have worked in the long run as these methods are often just part of the protocol e.g. even `a == b` will try `type(a).__eq__(a, b)` then fall back to `type(b).__eq__(a)` (~~and then it'll do some weird stuff with type names IIRC~~[0]).

And most operators are not considered symmetric so the fallback is not the same as the initial (even `+` has `__add__` and `__radd__`, also `__radd__` might be called first depending on the relationship between type(a) and type(b)).

And then there's the "operations" which fallback to entirely different protocols e.g. `in` will first try to use `__contains__`, if that doesn't exist it uses `iter()` which tries to use `__iter__` but if that doesn't exist it falls back to calling `__getitem__` with non-negative sequential integer.

Which is why sometimes you define `__getitem__` for a pseudo-mapping convenience and then you get weird blowups that it's been called with `0` (you only ever expected string-keys). Because someone somewhere used `in` on your object and you hadn't defined `__iter__` let alone `__contains__`.

Good times.

[0] I misremembered: it's for ordering (not equality) in Python 2[1] `a < b` will first invoke `type(a).__lt__(a, b)`, then if that's not implemented fall back to `type(b).__ge__(a)`, and if that's not implemented either it'll fall back to a few hard-coded cases (e.g. None is smaller than everything) and finally to `(type(a).__name__, id(a)) < (type(b).__name__, id(b))`. That is the order of independent types with no ordering defined is the lexicographic order of their type names, and if they're of the same type it's their position in memory

[1] where there's always an ordering relationship between two objects, one of the things I'm most graceful Python 3 removed even if it's sometimes inconvenient

silviogutierrez 7 years ago | |

This is great. I wanted to do something similar so that I could render Django templates in native JSX.

Any idea which of those RPC libraries could help achieve this? I'd love a Django model instance that could be used seamlessly in a JSX template.

rkeene2 7 years ago |

I recently did something similar with Tcl (instead of Python) and Duktape (instead of V8): https://rkeene.dev/js-repl/

All the Tcl runtime is in the "runtime" object, so like "runtime.puts('Hello World')" or "runtime.expr('2128')", etc

It was a lot of fun

haolez 7 years ago | |

Tcl is one of those languages that feel special once you get to know it. In a good way! It’s like a command prompt for actual code. I don’t know how to explain it any better.

rcfox 7 years ago | | |

I'm not sure I got to know Tcl properly. I needed to use it to interact with an electronics design tool in order to extract information about a design, and so most of what I learned was from the tool's documentation about how to use its API, and some random Stack Overflow for specific questions.

Anyway, Tcl felt very much like an 80's interpreted language to me. It seemed like every data structure was built on top of strings, and every statement was being run through eval().

lmeyerov 7 years ago |

We needed to solve this as part of using pydata/GPU service calls from our node app. While we do have a couple of native modules, they're a PITA. Our solution for more generic code is async HTTP service calls passing typed Apache Arrow tables ( https://arrow.apache.org/docs/js/ ), which gives a cleaner path to maintenance, observability, packaging, distribution, low overhead etc.

The current trick we're looking at is making this zero-copy when same-node, esp for GPU code, so happy to chat with folks about that!

snek 7 years ago |

Very cool idea! For all of you thinking of writing native modules for node, please remember to use napi, not nan. napi is fully abi stable, while nan is not!

https://nodejs.org/api/n-api.html

We also have a c++ wrapper: https://github.com/nodejs/node-addon-api

santa_boy 7 years ago |

I'm having the requirement to work with Node and Python now. I write my webapps in node but have data analytics scripts written in python to be invoked. I'm planning to use [python-shell - npm](https://www.npmjs.com/package/python-shell)

fermigier 7 years ago |

Also: https://github.com/bobpepin/pyduktape (interop between the Duktape JS interpreter - https://duktape.org/ - and Python).

rcfox 7 years ago |

I've worked the opposite way: calling Javascript from Python, using PyV8. It worked okay enough, but it felt super fragile. We only used it to run a tiny bit of client-side code on the server. I wouldn't want to have a large project where Javascript and Python interact heavily.

true_religion 7 years ago | |

I have just taken to running server and talking over http whenever I want two languages to interface. It’s slower but it’s pretty much bulletproof.