Instagram Makes a Smooth Move to Python 3

Instagram Makes a Smooth Move to Python 3(thenewstack.io)

218 points by fjordan 8 years ago | 61 comments

dmalvarado 8 years ago |

“Yeah, Python is great in so many ways, too bad it’s not really scalable.”

I'm not even sure what this means anymore. I guess I'm just not sure how any language, when used correctly, could be inherently unscalable. My guess is statements like this came from a time when monoliths were the application design of choice? Now, assuming Instagram has just 1,000 photo handling servers, each one is only responsible for 95,000 photos a day.

Of course, that's not to say that Instagram doesn't have CAP issues. It does, especially in the "C" area, but again, not a problem inherent in the language.

mahyarm 8 years ago | |

The threading story isn't great. It's not statically typed, which can make working on a large codebase with more people more painful. It also forces you to write more unit testing code coverage to make up for the lack of a static compiler checking things for you. Raw performance is not good compared to golang or java.

When you start getting to a certain scale, developers are cheaper than your server costs in some cases. That is when something being performant is more worth it.

ubernostrum 8 years ago | | |

All of these are decades-old talking points which have been largely shown to be factually incorrect. Why are you repeating them without justification?

bpicolo 8 years ago | | |

And at that scale you can make additional microservices for the critical path where it matters.

crdoconnor 8 years ago | | |

"It's not statically typed, which can make working on a large codebase with more people more painful. It also forces you to write more unit testing code coverage to make up for the lack of a static compiler checking things for you."

I never missed static typing on large code bases but I had numerous bugs caused by python's implicit type conversions - string to iterable of characters ["h", "e", "l", "l", "o"], None to False, string "no" to True, 0 to False, "" to False, etc.

A lot of this mirrors C's infuriating implicit type conversions.

jeeyoungk 8 years ago | |

See issues like https://engineering.instagram.com/dismissing-python-garbage-...

If you have to turn off a crucial language feature to increase performance, I'm not sure whether a language is considered "scalable".

nawitus 8 years ago | |

You can "brute force" scale pretty much anything just by throwing up more computing resources at the problem - but that means the language/framework/library is less scalable than something else which requires less resources. If the different in scalability is large enough, I suppose one can claim that one of the alternatives is "not really scalable".

I suppose that quote could use better wording.

adventured 8 years ago | |

As one example, PHP 5.x as-is (without caching) isn't a language that scales well at all. It becomes a horrible bottleneck under heavy load.

thehardsphere 8 years ago | | |

Even with caching it's quite poor, especially if you are using frameworks that aren’t written to properly take into account its unique means of execution (which the vast majority do not).

threeseed 8 years ago | |

It's old but still useful: https://www.techempower.com/benchmarks/

Almost every language can handle typical performance requirements. But when people say slow or unscalable it's almost always in relation to other languages.

Dowwie 8 years ago |

Members of the Py3 transition team (the authors of that article) gave a talk about the project at PyCon 2017: https://www.youtube.com/watch?v=66XoCk79kjM

bifrost 8 years ago |

I can't say I think this has ever been really true: "Performance speed is no longer the primary worry. Time to market speed is."

Performance has been a concern but programatic loadbalancing has been around for decades. When I worked at MSN/Linkexchange back in the late 90's we never really worried heavily about the performance of the language we used (Perl) because we could scale out servers. Perl isn't that speedy but it sure was easy to develop in. We served a billion and a half clicks per month with 8-10 machines from a single datacenter before I left, with Perl.

mixmastamyk 8 years ago |

Right, you hear a lot of griping about moving from Python 2 to 3 but I personally didn't have as much trouble as expected. Some of my projects just worked. One small tip I don't think they mentioned. Start using the logging module instead of print and it will eliminate one class of potential issues.

vhost- 8 years ago | |

I've heard people complain about print being a function now more than a hand full of times and my response is always "are you really using print that much in your code base?"

MatthewWilkes 8 years ago | | |

You still use it in debuggers and in the REPL. Using it seldom makes it harder to relearn the muscle memory, and only hitting it when you're debugging makes it more likely that you're already frustrated when it happens.

dangayle 8 years ago | |

This is what I encourage all of our team to do, it just seems like a no brainer. Having stray "testing" strings pop up in our logs without any source is hella annoying.

passive 8 years ago |

I've been writing mostly small python projects for 15 years, starting with 2.2.

I've had no issues in the migration to 3.

I did some building with it around 3.3, starting to commit around 3.4, and with 3.5 I build everything in it.

I don't have the performance challenges Instagram has, but my experience with application development in general is that 98% of performance challenges can be solved with (not-too) clever engineering. This applies to projects in every language.

There are a vanishingly small number of scenarios where the performance of your runtime actually dictates your performance limits.

If you're working on something and are worried about Python's performance, or which Python to use, don't. Use 3, optimize later.

MatthewWilkes 8 years ago |

I watched the PyCon keynote on this topic, and while it's nice to hear they've moved to Python3 their approach probably shouldn't be copied.

For example, in their codebase they had ambiguity between bytestrings and unicode strings. As Python3 tries to prevent you doing this, to resolve a big footgun from Python2.

The right fix here is to be consistent in your use of strings. Sometimes that is tricky because of how third party libraries have decided to implement their 2/3 compatibility, but it helps prevent shooting yourself in the foot with unicode bugs down the line.

Instagram did not do this. They created utility functions to force their data into the format they wanted at the point it is used. In other places they used tuple() to make sure that map calls that had side effects were fully iterated over.

In short, they had bad Python2 code and now have had Python3 code. Sometimes, at large scale, it's your only choice. But to smaller companies looking at this it's a bad idea. You're setting a precedent in your code that it's okay to make the same mistakes that Python3 tried to prevent.

richard_todd 8 years ago |

So their server needs are growing faster than their user-base to the extent that they considered switching languages. PHP didn't seem to perform much better, so they stuck with python and got a ~12% CPU usage improvement by moving to python3. It doesn't seem like a 12% one-time improvement actually solves the original problem, though. Perhaps pypy would have been better?

williamstein 8 years ago | |

Instagram also uses Cython a lot (so I've heard from a talk), so switching away from CPython might not provide as much of a speed up as one might otherwise expect, as one can get C levels of speed (and concurrency) with significant effort using Cython. Also, Cython and PyPy might not play so well together...

eggie5 8 years ago |

types in python 3.5?! I had no idea -- that's exciting.

nyangosling 8 years ago |

>It made sense that, if we were going to stay on Python for the next ten years, we should invest in the latest version of the language.

I don't know this stands out to me in particular, but the 10-year commitment is definitely a big decision. I suppose I've never had to make a similar decision so perhaps this is more common than I think.

treve 8 years ago | |

Most successful application I've worked on have either already been around for 10 years, or have lived onto become 10 years old. The ones that didn't, were usually not successful.

It's a good reason to be careful to avoid the latest and greatest, but go for tech that has (some) track record of being maintained and used for a few years.