Testing LLVM

87 points by awalGarg 9 years ago | 34 comments

I always find committing to LLVM very nerve wrecking, because of the post-commit CI testing. LLVM has so many architectures that more often than not something I write will fail on one of them. And the only way for me to find out is to commit it, wait for the buildbot to fail (which can take a few hours, during which I really can't leave my computer lest I leave trunk broken on some buildbot, which is a big faux pas), revert it and then figure out what went wrong. I'm hoping that at some point this will be improved, such that I can run the whole buildbot army on my commit before putting it on trunk.

shanemhansen 9 years ago | |

That's odd. It would be nice if branches could be tested with the build bot.

gus_massa 9 years ago | | |

I agree. When I submit a commit for Racket, the branch is tested in Travis and AppVeyor. That catches a lot of error.

Anyway, Racket has an additional internal CI called DrDr and some very subtle errors are only detected there, after the commit is merged.

wyldfire 9 years ago | |

I get the feeling that there's parts of the community that feel the same way. I'm hoping that the planned move to github will naturally cascade into pre-commit checks.

Joky 9 years ago | | |

A big big problem is having enough hardware to have a CI throughput one or two order of magnitude what it is now. Unfortunately that's not an easy thing considering how "heavy" it is to build and test the full toolchain.

ezanmoto 9 years ago | | |

Do you know of solutions that exist for pre-commit checks for GitHub at the moment?

Locke1689 9 years ago | |

Most of the .NET repos are structured to use inner and outer loop testing with Jenkins. Most tests on most architectures are run in the inner loop, which are kicked off in parallel as soon as you make a pull request to one of the .NET repositories on Github.

Some repositories, like CoreCLR, have outer loop testing that runs on a separate schedule (nightly, I think), but those tests are far less likely to break and are more devoted to finding rare and difficult to compute edge cases.

SloopJon 9 years ago |

Interesting to see screenshots of LCOV. I'm hoping to get an intern to work on test coverage this summer, and I wondered whether LCOV is still current. Looks like the latest release is from December 2016.

Joky 9 years ago | |

The screenshots are from the "old" style coverage. We have much better views now: http://lab.llvm.org:8080/coverage/coverage-reports/opt/cover...

See for instance how having your cursor on top of each side of a condition tells you how many times each individual condition were evaluated!

bootload 9 years ago |

"Compilers are usually not networked, concurrent, or timing-dependent, and overall interact with the outside world only in very constrained ways."

Are there any parallel compliers?

tux3 9 years ago | |

I can say that for C and C++, the compilation is very often parallelized at the translation unit (file) level, by starting multiple instances of the compiler either locally or over a network with something like distcc. This is simple and effective enough that there wouldn't be much gain in parallelizing the compilers: all the cores are already busy most of the time.

Locke1689 9 years ago | |

It depends on what you mean by parallel.

Certainly the Roslyn C# compiler is highly parallel. All files are parsed in parallel, then all classes are bound (semantically analyzed) in parallel, then the IL serialization phase is sequential.

bootload 9 years ago | | |

"It depends on what you mean by parallel."

Across different machines, not cores on ^a^ chip?

enqk 9 years ago | |

Microsoft's CL.exe is, through the /MP option

boris 9 years ago | | |

Which only has effect if you pass multiple files to compile.

newsat13 9 years ago |

One quirk of llvm is that they don't have a pre-commit CI.

mp3geek 9 years ago |

I do find it strange that such a large project isn't using a better VCS. SVN seems to be very antiquated.

1. Git allows you to easily switch between multiple work items while keeping track of the work done in each item. 2. Git allows you to easily integrate with people who have significantly diverged clones of your tree without too much trouble. 3. Git allows you to easily work offline.