Srclib: A polyglot code analysis library

Srclib: A polyglot code analysis library(srclib.org)

76 points by anant 11 years ago | 32 comments

sqs 11 years ago |

Thanks for posting this! I'm one of the original creators. It's still alpha, so there will be bugs. But it can do some pretty cool and useful stuff already in your editor (and it powers https://sourcegraph.com/).

Anyone have feedback about how we're explaining the problem and solution, and the first-run experience of using the editor plugins and the "src" tool? We would LOVE to hear it so we can fix/improve it.

michaelmior 11 years ago | |

For a tool which is all about being language agnostic, I'm curious why you went with Go struct definitions to define the formats instead of something like JSON Schema[0].

[0] http://json-schema.org/

sqs 11 years ago | | |

Simplicity and lack of time. :) We'd love to move to something more language-agnostic for the schema definitions. If you're interested, we'd definitely accept a patch.

diggan 11 years ago | |

Hey! Quick question.

Are looking for the source code to see if you're working on a PHP implementation. Hard to find any links though. Then I started the think, maybe it's not open source at all?

Didn't find any information so, is it open source? In that case, where?

adamnemecek 11 years ago | | |

There's a Github link on the website right below the email signup

https://github.com/sourcegraph/srclib

adamnemecek 11 years ago | |

Did you guys consider hooking this up with llvm since it provides call graph generation for the supported languages? If so what was the conclusion? If not I might look into it, no promises though.

sqs 11 years ago | | |

We haven't looked at llvm yet. None of the tools for the existing languages we've targeted so far (Python, JavaScript, Go, and Ruby) had llvm-based tools that we chose to use. It would be AWESOME to hook it up to llvm. We'll help however we can, if you are interested. I'm sqs@sourcegraph.com.

joshschreuder 11 years ago | |

Looks really awesome! Love the Github and Sublime plugins.

Unfortunately I can't try it out as there doesn't seem to be a Windows version. Any plans for one?

sqs 11 years ago | | |

We would love to have Windows support in srclib, but we honestly have no idea what (or how much effort) it would take. It might work in Cygwin as-is. If you or anybody else wants to try it, or at least identify issues, we would help.

One related thing is a lot of folks want to add C# support. I assume that running Roslyn requires Windows, which'd require Windows support for srclib. https://github.com/sourcegraph/sourcegraph.com/issues/157

azmenthe 11 years ago | |

Awesome project! Any timeline on vim support?

sqs 11 years ago | | |

None of the original contributors use vim, so we wouldn't make a very good plugin for it. We'll help anyone who wants to make one, feature them on srclib.org (obviously), etc. Do you know anybody who'd be interested in making a vim plugin? A good place to start looking to see what's involved is in the code for emacs-sourcegraph-mode, sourcegraph-sublime, and sourcegraph-atom.

comex 11 years ago |

Based on reading some of the docs (can't check out the source on this phone), looks like this has great potential; for a long time I've wanted something better than cscope, and not limited to a single language, to use with vim.

My main concern is that any system based on scanning entire source files offline is unlikely to have the same interactivity as IDEs. That is, if I type a line of C++, I don't want clang to have to churn through the whole file and any header files before the navigation updates...

sqs 11 years ago | |

Yeah, truly interactive editing and incremental updates are something that srclib doesn't handle right now. We have some plans/prototypes around using presentation compilers, lenient parsers, and a persistent "src" process to provide the same kind of interactive editing support that IDEs have, but there's nothing in srclib or any of the language analysis toolchains yet. We are /pretty/ sure it'll be possible to get the kind of responsiveness you'd want.

If you have time to take a look at how we've implemented the existing toolchains and architecture, it'd be awesome if you wrote down any suggestions you have for how we could implement interactive code assist.

sitkack 11 years ago | |

Just serialize the continuation to the database and everyone can fork() the state language analyzer with their own queries.

tikhonj 11 years ago |

Wow, this looks like an awesome project. As a pretty fanatic Emacs user who doesn't much like IDEs, I've always wanted a nice editor-agnostic IDE ecosystem that I could just plug into Emacs. I've seen language-specific things, which are great but naturally linked to their language, and Emacs-specific monstrosities like CEDET, but never anything trying to be truly universal.

Very promising.

For some reason though, I can't help but mentally parse "Srclib" as "Scribd" :/.

sqs 11 years ago | |

Thanks for the kind words! Please send feedback/complaints.

Re: the name "srclib", I was on the phone with our domain registrar when we got srclib.org/com, and the rep referred to it as "Sr. Clib" (as in "Señor Clib"). So that's stuck in our heads now. :)

josephschmoe 11 years ago |

IIRC IntelliJ does some things like this:

1. It's a polyglot IDE and open-source.

2. Specialized tools for exporting to multiple languages/DSLs.

sqs 11 years ago | |

I wasn't able to find the source code for some of IntelliJ's dynamic language support, like Python and Ruby. If it is indeed open source, I would love to check it out.

IntelliJ is awesome. We're just trying to make something that isn't coupled to a specific editor and is super non-monolithic. Our goal is for you to be able to run a single command ("src") in a directory, or have your editor plugin (or other kind of tool) invoke something like "src api describe --file FILE --start-byte N", to tap into a fully analyzed database of the source code in a package.

loevborg 11 years ago | | |

That's a potentially very useful project, thanks so much for working on this!

jalanb 11 years ago |

A couple of points on the website, https://srclib.org/

1. Can't find an actual Download link, even in the section called "Try it now". Oh - turns out the download links are under "Docs", which is kinda strange.

2. Downloaded for Linux. Unzipped, ran, got

    $ ./src --help
    bash: ./src: cannot execute binary file

I wonder if the links are correct, as all three download links on the Docs page (https://srclib.org/gettingstarted/) include the phrase "...?os=darwin&..."

sqs 11 years ago | |

Thanks for posting this.

Just fixed the download links.

What do you think of the new installation section on the homepage, at https://srclib.org/#install?

padator 11 years ago |

related work: https://github.com/facebook/pfff which focused on languages srclib didn't. Also a polyglot source analysis library.

sqs 11 years ago | |

Yeah, pfff seems very cool. It seems that pfff has written parsers and type inferencers for a variety of languages in OCaml. That is an impressive achievement.

For srclib, we are way lazier. We try to use existing libraries (written in the languages they target) for parsing, type checking/inference, dependency resolution, etc. We hope that makes it easier for JavaScript hackers to hack on srclib's JavaScript support, Python hackers to work on srclib's Python support, etc.

padator 11 years ago | | |

Yes, lately I've also tried to reuse existing "parsers" ... for instance I now use existing libraries for the java bytecode, ocaml typed AST (.cmt) and clang ASTs.

kovrik 11 years ago |

Looks awesome!

Hope to see Vim plugin soon.