For work, I decided to look up some systems that allow programming assignments to be checked automatically. The following list is by no means complete, but should provide a general overview. I found most of these using some Google-fu; the rest are from a systematic review published in 2010  and a Master’s thesis from the same year  .
This system does not seem to have been updated for some six years, so approach with caution. Supports Java, though possibly extendable to other languages.
Web-CAT is an advanced automated grading system that can grade students on how well they test their own code. It is highly customizable and extensible, and supports virtually any model of program grading, assessment, and feedback generation. Web-CAT is implemented a web application with a plug-in-style architecture so that it also can serve as a platform for providing additional student support services to help students learn programming or software testing.
Plugins exist for Java, C++, Scheme, Prolog, Pascal and Stander ML. The system seems to be able to support any programming language, provided it can be installed on a web server.
Marmoset is a system for handling student programming project submission, testing and code review. We’ve been developing it at the University of Maryland for over 5 years, Last fall, we handled 74,000 submissions for a total of 172 different projects from 1,700 students in 27 different courses. It works in all different programming languages, and is designed to work well with both very small and very large projects, such as our OS course in which a submission consists of tens of thousands of lines of code.
This system was originally developed in 2006 as part of a PhD thesis, but seems to be maintained and updated to this day. Interestingly, Marmoset supports authentication via OpenID as well as LDAP.
Robocode is a programming game, where the goal is to develop a robot battle tank to battle against other tanks in Java or .NET. The robot battles are running in real-time and on-screen.
Not exactly like the other systems, but this could very well be used to gauge the programming skills of a student. There are several IBM developerWorks articles on how to program for Robocode.