Zeta: Functional Neural Networks in Ocaml

Zeta: Functional Neural Networks in Ocaml(github.com)

115 points by frjalex 6 years ago | 9 comments

Athas 6 years ago |

A quite bright bachelor's student of mine did something similar for his undergraduate thesis[0]. Conceptually, he represented neural networks as functions from input to output, and connecting two networks required that the output of the first matched the input of the second, and produced a new network. Now, to enable gradient descent or other kinds of optimisation, the networks were not literally functions, but rather pairs of functions for running the network forwards and backwards. While limited in some ways (e.g. no recurrent networks), it was quite convenient, and close in performance to TensorFlow on a single GPU.

[0]: https://futhark-lang.org/student-projects/duc-bsc-thesis.pdf

6177c40f 6 years ago |

I had been messing around with a similar concept not too long ago in OCaml- as it turns out, strong, static typing can be very nice to have in writing ML models (and code in general). I find it reduces some of the cognitive overhead that comes from having to keep track of how the functions interact in the program (or perhaps, it reduces/clarifies the potential ways the functions can interact).

I'll be very interested to see where this project goes.

dmit 6 years ago | |

It's always been baffling to me that certain problem domains with extremely slow feedback loops have standardized on using dynamically typed languages. Machine learning and Python. Video game scripting and Lua. A lot of scientific programming as well.

How did that happen? Did the trend start because the work was done by domain specialists who were not necessarily expert programmers, and C was too unergonomic for non-experts to use? Is that the reason a Skyrim modder can't get immediate feedback from a compiler in 2020 - because 25 years ago game devs thought that documenting a C API was harder than embedding a scripting language in the game engine? Or perhaps because evaluating scripts was more secure than loading third-party DLLs and exposing the game's innards to them?

Edit: also, if every CSV file came with a schema, that would be great. Even if it says that every column is of type Option<Any> - at least then I know what to expect.

75dvtwin 6 years ago | | |

I think it is more that 'expert programmers' where told to leave 'domain-specific' bits for power-users to deal with.

Therefore,

- game scene development in games, are in scripts. - Equity derivatives and swap contracts are in Excel - scientific modeling is in python

I also agree with your other though that, dynamic loading enables 're-use' of the core framework, so those frameworks/engines become marketable technology assets so to speak.

The programmers, then in some instances, created DSLs to then auto-generate performant framework-specific code from the DSL-written user specs.

But those, usually, are quite limited and also some power users migrated to be descent (not the expert engine/core-level, perhaps) programmers

andrepd 6 years ago |

ML in ML?

Jokes aside, see:

https://github.com/LaurentMazare/tensorflow-ocaml

Also

http://ryanrhymes.blogspot.com/2017/03/build-neural-network-...

for how you can very quickly write something from scratch with "Ocaml's numpy", owl.

allisterb 6 years ago |

I'm also working on something like this with F# and TensorFlow:

https://youtu.be/GUMTpTqzsJA

https://devpost.com/software/sylvester-tf

using typed natural number dimensions for arrays, vectors, matrices, tensors et.al

brianberns 6 years ago | |

This looks awesome. I’m an F# developer and I agree with your intro paragraph 100%:

“One of the ironies of today's computer programming landscape is that functional languages directly inspired by the declarative languages for expressing abstractions and equations of logic and mathematics, have been sidelined for mathematical and scientific computing in favor of imperative, dynamically-typed languages like Python and Julia.”

Sadly, I’ve found that every road leads back to Python when it comes to deep neural nets. I tried using TensorFlowSharp from F# but found it very frustrating. Only Python seems to have full bindings to the TF API. Would love to give your wrapper a try.

allisterb 6 years ago | | |

I've created a Jupyter notebook here: http://ec2-52-15-66-200.us-east-2.compute.amazonaws.com:6677...

that demos my ongoing progress with Sylvester and I'd love your feedback. It's also available on NuGet: https://www.nuget.org/packages/Sylvester.tf/0.2.3.4 with a native TF 2 package: https://www.nuget.org/packages/Sylvester.tf.Native.Win/.

It's still WIP but coming along pretty well.

akotulski 6 years ago |

It's great to hear that people are trying to use ocaml for ML :) To me it seems like owl is the most mature ocaml project in this area. I recall they even had some GPU support. Unfortunately, I didn't have a chance to try it out so I can't comment first hand on how "ready" it is.

https://github.com/owlbarn/owl