In "mask edit mode" one could create and animate 2-D masks in the 3-D view, similar to 'grease pencil'.
A mask system could be linked to a specific camera, allowing one to create a scene with multiple cameras, each with their own mask systems.

*Newly created masks would appear in an ordered list, with their order signifying their z-position in the mask stack.

*A mask would be composed of a closed curve-like object, that would allow the user to change each handle/vertex between auto (curvy) and vector (sharp) like the current curve system.
This system should allow the user to key-frame each handle, but what's more, it is vital that the user should be able to add and delete handles on a per-frame basis so that a mask can grow or shrink in detail over time.

*Each mask would have its own modifier stack, that would allow one to use modifiers to change the behavior and input of individual masks.

*To use masks in compositing, a mask node representing an entire stack of masks could be used as input.

Use the UV/Image Editor to draw mask over the render result and display the whole thing in the backdrop of the node editor.

In the UV/Image Editor ad a datablock for masks as a pulldown menu. Use the fake user "f" for multiple masks.

In the F-curve editor there should be mask related keyframes.

In the composit node editor there should be a new mask related node where the datablock of the mask can be selected.

We need an editor of some sort that can generate animated spline masks for usage in the compositor/sequencer.

This editor's output should be accessible as a sequencer strip and a compositor node to be used as needed

This masks, each of them composed of any number of closed bspline surfaces, would ideally be stored on a datablock for easy sharing and linking

Edit: here's a design graph of a more detailed proposal

http://primatefx.com/salazar/RotoMasks.png