Atari Flashback 2

Not strictly a retro console, but definitely a console to play retro games on, the Atari Flashback 2 is a great and cheap way to enjoy some classic Atari 2600 gaming. It comes with 40 built-in titles, 2 joysticks and can even be modded to play proper 2600 cartridges. You can grab one (either new or used) for a truly modest price over at Amazon.com

As for the included games, they do feature such gems as Adventure, Centipede, Asteroids, Combat, Yars' Revenge, the exclusive Yars' Return, Pitfall, Video Chess and Super Breakout. Oh, and you can easily connect the old paddles to it too.