With those bullet points, looks like you didn't talk to actual engineers, but rather middle-layer management people.

Not even bad engineers try to pretend like this is true.

Why I use tensorflow:

- keras

- I used tensorflow yesterday

Despite the absolute nightmare of getting it installed and running on a gpu, I managed it and had a fantastic model. It was doing so well that the company wanted to expand the project and build out a multi-gpu rig as part of it. So I get building that environment and install the latest CUDA, cuDNN, nvidia driver and use tensorflow 2.0 aaaaaand it wouldn't work. I actually spent a long time hacking on it till on a forum I read that it was just a bug that hadn't been fixed yet.

At this point I decided to see what Pytorch was like. In literally one day I installed everything and migrated my project completely over to pytorch. Same speed, same accuracy, works perfectly on a multi-gpu rig when I set it to. It was like a breath of fresh air.

The next day I wrote some C++ to import a saved pytorch model so it could run in a deployment environment. The C++ api is also great. The docs are lacking a little bit, but an Facebook researcher mentioned to me on the forums that they're hoping to have it all done by next month.

It's unlikely that I'll be going back to tensorflow.

1. Extremely easy to debug and work with. Being able to debug effortlessly in PyCharm makes life very easy.

2. The API is quite clean and nice and fits in really well with Python and nothing feels hacky. I've developed my own Keras-like framework for experimentation, training and evaluating models quickly and easily and the entire experience has been really enjoyable.

3. The nicest thing though is that as the article points out, a huge percentage of researchers have moved to Pytorch and this allows us to more easily look at other researcher's code and experiment with things easily and incorporate ideas and cutting-edge research into our own work. Even for things that are released in TensorFlow, if it is an important publication that gains attention and traction in the community, you will likely have implementations in Pytorch pop up soon enough.

I do think that TensorFlow still has an edge on the deployment at scale/mobile side of things as pointed out by the article. But Pytorch is a lot younger and they are making a lot of progress with every release in that space.

There're maybe all of two "surprises" I've encountered in all my time using it, if even (1. Gradients are accumulated in state, 2. nn.Module does funky things with attributes, so use something like nn.ModuleDict if you're going to be dynamically setting modules). Everything else works like a dream, and works almost exactly how you expected.

Model parameters? .parameters() gives you a dict-friendly generator of tensors. Model state? .state_dict() is a dictionary. Loading model state? load_state_dict(state_dict)... just loads a dictionary. Reusing modules across different modules? Just assign them! Determining what parameters to optimize? Just ... give the list of parameters to the optimizer.

You can use all your using Python development and debugging tools, and it feels 100% natural. I can fit it into other Python workflows without making the whole program centered around TensorFlow.

TensorFlow is undoubtedly powerful, and if you have the time/resources to put into a static-ish TensorFlow-centric workflow, it could pay off many times over. But it definitely feels like learning an entirely new language, with an entirely different debugging pattern. And furthermore, a language that is constantly changing patterns and best practices, other than super-standard Keras examples.

To put into context, even running the official TensorFlow models repository has deprecation warnings. Whereas torchvision works like seamlessly and reads like a reference for writing PyTorch model code.

There is just a developer-centric focus to PyTorch that makes it a joy to use.

I am a bit saddened by all of this, because I really liked how easy it is to define a graph in Tensorflow in Python, serialize it, and then use its minimalistic C API to use the graph in Go, Rust, or wherever you need it.

How is your experience with PyTorch and backwards API compatibility (I know that they only reached 1.0 fairly recently)?

Lightgbm library has consistently performed well. I've been interested in how many colleagues instantly jump to neural nets when in my experience this often doesn't beat lightgbm on medium sized datasets not related to text/images.

Code: https://github.com/Chillee/pytorch-vs-tensorflow

Ablation of claims: https://thegradient.pub/p/cef6dd26-f952-4265-a2bc-f8bfb9eb1e...

JS interactive charts: https://chillee.github.io/pytorch-vs-tensorflow/

With fastai module that's built on Pytorch learning and developing Deep Learning solutions have become a lot easier. So there's a real game on now

It's always like this.

Think how Ubuntu took over the server market because amateurs were preferring it instead of Redhat/CentOS. And when they became professionals or were in a position to decide, they also put Ubuntu on the server because this is what they knew best.

A quick observation that may not be 100% accurate but still worth mentioning: in some ways TF feels like it was written to solve large scale issues on day one. For example, when I started playing with the new TF 2.0 distribution strategies and dataset pipeline, I quickly got the sense that this thing was meant to move and ingest bucketloads of data across hundreds/thousands of vm instances. In a way, I suppose it's a reflection of Google culture where there's a strong emphasis on not doing things that don't scale to Google Scale.

As a result of this, I sort of feel that you should start with PyTorch and eventually graduate to TF if/when the scale requires it. This is sort of like starting with Rails/Django/Node, and migrating to a Go/JVM/[Insert Your Favorite Static Language Here] stack when the traffic load warrants it.

https://www.youtube.com/watch?v=EqWsPO8DVXk

* Automatic differentiation of higher-order differentiation being important, and how there's clearly room to disrupt there

* Increasing hardware diversity seems to mean that both frameworks will run into a brick wall as-is

Exciting space. It'll be fascinating to see how dramatically, or not, things change in the coming years.

Back when we were using TensorFlow, whenever we wanted to try something new, sooner or later we would find ourselves wrestling with its computational graph abstraction, which is non-intuitive, especially for models with more complex control flow.

That said, we are keeping an eye on Swift + MLIR + TensorFlow. We think it could unseat PyTorch for R&D and eventually, production, due to (a) the promise of automatic creation of high-performance GPU/TPU kernels without hassle, (b) Swift's easy learning curve, and (c) Swift's fast performance and type safety. Jeremy Howard has a good post about this: https://www.fast.ai/2019/03/06/fastai-swift/

Perhaps it’s the nature of the game that changed with many new kinds of architectures and so on. But maybe Keras is already overengineered for someone who just wants to make thumbnail sized GAN stuff at home.

> Great API. Most researchers prefer PyTorch’s API to TensorFlow’s API. This is partially because PyTorch is better designed and partially because TensorFlow has handicapped itself by switching APIs so many times (e.g. ‘layers’ -> ‘slim’ -> ‘estimators’ -> ‘tf.keras’).

Arguably, one of the biggest issues Google had with Angular was the switch from 1.x to 2.x. You'd have thought they learned about how not to make major changes on OSS projects.

Facebook on React for instance do an amazing job here, they use prefixes to anything they don't want to support like "UNSTABLE_" and show warnings forever when they actually plan to make something small obsolete.

I tried to learn from both, so in some of my bigger personal OSS projects (amount of work involved) like npm's "server". I purposefully made some APIs a bit more limited than I could to have more flexibility later on if I didn't like the direction. Of course at a different level, I am a single dev doing OSS on my free time after all.

But I understand in a project of the size of e.g. Tensorflow it's not an individual dev learning, it's more about the company learning how to do things better.

I haven't had success doing so using frameworks such as Torch and TF, even if their toolkit is better to develop new solutions.

Also we get to write code in C++, which can be a big positive when developing machine learning SDKs. I personally still do most of the prototyping in Python though.

I'll be checking the link on the post that mentions that pytorch allows models to be converted to c++, looks promising actually.

[0] http://dlib.net/

And then the title is "PyTorch vs Tensorflow", but it never says whether the Y axis is unique mentions of PyTorch or Tensorflow? From the context I guess PyTorch, but come on!

The Y axis should be "Fraction mentioning PyTorch", and the title should be "Papers that only mention PyTorch or Tensorflow" (assuming I have understood this correctly).

Shame it was labelled so badly because it's an amazing graph otherwise!

I'll add that it was much easier to install PyTorch with GPU support than it was to install TensorFlow with GPU support - at least that's how it was around November of last year. The PyTorch install was painless, whereas we ended up having to build TF from source to work with out setup. Could be different now as I haven't looked at TF since then.

One area where I wonder if neural nets would be a more useful option is using something like an LSTM to predict defaults based on a sequence of data? I've tried this a handful of times and doing a bit of feature engineering to aggregate data in a handful of fixed buckets has usually been better and easier, but I'm far from an expert in that area.

I know Jeremy Howard has shown decent results with fastai/pytorch for tabular data and I've seen some Kaggle teams do well with neural nets for tabular data. I've also had decent results with gbdt/nn ensembles. But I think in most situations where you just have tabular data, you'll get better results with less effort if you use lightgbm or the like.

I call them “tricks” but really they’re just design decisions based on what current research indicates about certain problems. This is largely where the “art” part of neural networks comes from that many people refer. The search space is simply too big to try everything and hope for the best. Therefore, how a problem is approached and how solutions are narrowed and applied really matter. Even simple things like which optimizer you use, how you leverage learning rate schedules, how the loss function is formulated, how weights are updated, feature engineering (often neglected in neural networks), and architectural priors make a big difference on both sample efficiency and overall performance. Most people, if they’re not just fine-tuning an existing model, simply load up a neural network framework, stack some layers together and throw data at it expecting better results than other approaches. But there’s a huge spectrum from that naive approach to architecting a custom model.

This is why neural networks are so powerful and why we tend to favor it (though not for every problem). It’s much easier to design a model from the ground up with neural networks than it is for e.g. xgboost because not only are the components more easily composable thanks to the available frameworks but there’s a ton more research on the specific interactions between those components.

That doesn’t mean than every problem is appropriate for neural networks. I completely agree with you that no matter what the problem is you should never jump to an approach just because its popular. Neural networks are a tool and for many problems you need to be comfortable with every one of those decision points to get the best results and even if you’re comfortable it can take time and that isn’t always appropriate for every problem. My other point is that I wouldn’t draw too many conclusions about a particular algorithm being better or worse than another. I’m not saying that was the intention with your comment but I know many people in the ML industry tend to take a similar position. It really depends on current experience with the applied algorithms, not just experience with ML in general.

And if you have any kind of seasonality you a dataset with a large enough timeframe. (At least more than a year.)

Nonetheless, LightGBM and xgboost are also commonly used in the insurance sector.

They are still somewhat problematic for conversion rates in a highly dynamic market though.

common people do want to appreciate & adopt the things that seems fit to the knowledge sphere at present from the researchers. tensorflow approaches are better and respected each and everyone of the community as well in exchange enlightened us new ways of understanding of ml solutions. it have turned into a family “If you want to go fast, go alone. If you want to go far, go together.” and given the assets alphabet have a common man can turn into researchers! e.g. >> https://learn.grasshopper.app take this for example "Learn to code anywhere. Grasshopper is available on iOS, Android, and all web browsers. Your progress syncs seamlessly between devices." << this is the status quo ! it's a gift of a lifetime for generations !

Though you can already use very clean Pytorch style libraries like Flux and Knet or the Tensorflow bindings to leverage the benefits of Julia for high performance numerical processing on the adjacent tasks such as data preprocessing.

You can take a look at https://discourse.julialang.org/t/where-does-julia-provide-t... for some of my questions.

Essentially, the biggest advantage imo is that Julia offers a single cohesive language, where compilers can do anything at the language level. I don't think this will allow for a single killer application - almost anything Julia can do ca n be simulated by some combination of Python/C++.

However, what might be true is that using a single language allows for much faster development and iteration than a combination of Python/C++. I think the way that'll manifest is in more and more high quality libraries coming out for Julia that are higher quality than the Python ones.

Maybe wait 5 years, and we'll see what happens :)

[1] https://github.com/FluxML/Flux.jl/issues/625