Any of the three plugins on this list are great options if you're looking for a smooth, easy to use jQuery YouTube pop-up player plugin. They all feature lightbox effects and really clean transitions.
This plugin is easy to use and install, and is a great, lightweight option for embedding YouTube videos as popups dialog boxes on your site. You can customize the size and styles of the dialog box. Videos are embedded via iFrames.
This plugin features YouTube and Vimeo support. It also has a really nice option that allows you to choose whether or not you want the popups to be launched with or without autoplay. The popup is also fully responsive and the style is completely customizable with CSS.
This plugin is highly customizable and also offers support for both YouTube and Vimeo videos. Read the documentation to see the dozens upon dozens of transition and style options you can add to the code to customize the popup dialog box exactly to your liking.