Ask HN: Would you use Python as the main lang in a potentially large project?

22 points by chuckus 9 years ago | 15 comments

tl;dr Would you advocate for using Python as the main language in a potentially large-scale project and long running standalone project (not a library)? Such as if you are a startup aiming to build a sustainable SaaS that becomes your full time occupation, or for enterprise projects?

I've been using Python for over 4 years from writing web applications for enterprise to startups in Django and Flask, and to RF test frameworks leveraging GNURadio. Python was a suitable choice for many problems I have encountered as because the ecosystem is extremely broad and it is very easy to get started, not just because the language is simple, but because of the popular "battery-included" frameworks. This means I can deliver value must faster from the beginning.

But having worked in a company which develops many different production systems developed over a period of 10 years, some running on platforms such as Debian etch, I can't help to think that had an enterprise language and ecosystem such as Java or .NET been chosen, maintenance and new features would have been significantly easier, because you hit performance issues much later and understanding legacy code bases are easier even if poorly written because of the static typing, and migration between different runtime versions is first-class.

I see Python (or similarly, Ruby) being used at large-scale at consumer-facing businesses such as Reddit, Digg, Uber, Dropbox, etc. but many eventually integrate to a JVM based language into their production system.

Would you advocate for using Python as the main language in a potentially large-scale project and long running standalone project? For people who have had the experience, it is worth using Python at the beginning and bearing the cost of adding another language and ecosystem in the mix when to support/replace Python when you hit scaling issues?

bjourne 9 years ago |

> Python as the main language in a potentially large-scale project

This is the flaw in your thinking, the word Potentially. Good engineering means you worry about actual problems and not potential problems (within reason ofc). Your actual problem is that you haven't written the web application, so you do that first. For doing that, Python is an excellent choice.

"But what about web-scale?" Well, chances are your product will fail for some reason and never become successful therefore thinking about scaling problems now is a waste of time. In my experience, scaling problems comes much later than most people think. I've worked on an interactive site in Django that served 60k visitors per day without problems. It was the database that gave us the most performance problems not Python.

chuckus 9 years ago | |

Good point, that adjective changes everything. In regards to scaling, yes I haven't met performance scaling issues with Python web applications, even when needing long running asynchronous tasks by relying on the ever popular celery and redis, because I've never had somewhat even moderate traffic. In hindsight, I am guilty of over-engineering previous web applications "to deal with potential scaling issues", especially at the infrastructure level, leverage clusters of containers when I could of got away with Heroku.

Ashish_J_S 9 years ago | |

> It was the database that gave us the most performance problems not Python

I agree database has given the most pain to me as well may it be SQL or NSQL

Ashish_J_S 9 years ago | |

>I've been using Python for over 4 years from writing web applications for enterprise to startups in Django and Flask

He has used python to write web applications.

solomatov 9 years ago |

The main advantage of Python for you is that you know the language and the libraries, and when you start something new, it's critical that you work on the project, not on learning a new technology.

The main advantage of JVM and CLR for large scale development is support for static types. But, Python has a standard for type annotations (https://www.python.org/dev/peps/pep-0484/) which allow you to have all the language services which are available for static languages (refactoring, completion, etc) available for the Python. Not all libraries have these annotations, but you can provide it yourself, or separate code with annotations from code without annotations.

You should also take into account that JVM, and CLR have much better runtime performance than Python, almost on par with C++, so you should think about this if it's important for your app to use resources as efficiently as possible, but typically it's not as important for SaaS projects.

chuckus 9 years ago | |

True, I've been using type notations with all my Python 3.5+ projects (and works very well with PyCharm), though noting that this is only a recent addition, available (officially) for very recent Python releases and is not enforced out of the box. And yes, delivering on the project is critical, especially for startups at the very beginning where you're still trying to find product-market fit so sudden production changes need to be swift.

was_boring 9 years ago |

What is large scale? Does it just mean lines of code? I'm in charge of a Python app that has about 75,000 lines (not including comments, blank lines, imports, generated code or tests), and been doing it for 4 years.

Would I recommend python for something the size I do now? No. I would recommend any other language that has strongly enforces types and compile time checks (like Java). They add an extra layer of safety on unit tests. I have dreams of rewrites that are in C# or Java.

Just today I fixed 2 bugs because of unexpected types being used.

itamarst 9 years ago |

Scaling is highly dependent on what you're building. For many web applications the app server isn't really the bottleneck, it's the database; switching to JVM won't help with that.

For other applications Python is way too slow... although PyPy helps a lot.

As for things like static typing and maintainability, it's certainly possible to do with Python. You also have the power to write horrible unmaintainable code. Unit testing tends to be much more more important in Python. And you can add static typing to Python these days, via external tooling.

Some ideas and tools for building long-term maintainable Python, originally written as guide for Java developers: https://codewithoutrules.com/2016/10/30/python-for-java-deve...

usgroup 9 years ago |

JVM is fast and supports productive languages (including Jython) which can easily interop without service abstractions. You leave more options on the table.

But like others have said, it's tempting and typical for developers to obsess about details that at worse will be a good problem to have (... like having to super scale). Worth making sure you're not one of them.

iurisilvio 9 years ago |

Yes. Use what you already know and make your project fast.

Today, after some large projects, I miss strong typing but not enough to change everything. You need more tests, but I can live with it.

We integrated with a large retailer in a complex project with only two developers and I think Python was the reason to it. Too many open source things already done and fast iterations.

codegeek 9 years ago |

"startup aiming to build a sustainable SaaS "

This is the key piece of information based on which I will give you my advice. If you know Python and use it already, then go build it in Python.

If you do grow to that level of scale, python will be the least of your worries. You will figure it out then. Just build in python and release the damn kraken.

joeclark77 9 years ago | |

I want that last sentence on a coffee cup. Or tee-shirt I guess, but I have too many at the moment.

CC_ing 9 years ago |

I've the same problem, but instead of python is ruby(with rails) vs python/node.js(that perform a better than ruby)

chuckus 9 years ago | |

In your case, if you are just planning to de a web API with the majority of the workload being I/O (such as hitting other APIs), then node.js (outperform Python and Ruby out of the box in this case IMO). Otherwise, Python is a more versatile choice. All 3 are relatively easy to start, but given you like Rails, which is big on convention over configuration, the closest you'll get, that has a large community is Django, and a Node.JS counterpart would be sails.js. The biggest difference is in the languages itself, all 3 have some distinct differences in design.

But all 3 suffer from the same issues that dynamic languages have, and weren't designed for enterprise. Node.js is also being added to existing enterprise applications, but normally for anything web facing.

EDIT: Add opinion relating back to main question.

sontek 9 years ago |

Yes.