I too have though along similar lines – and have even spent a few moments sketching database structures for such an application – usually to screw em up and bin them LOL !!
Te great strength of Yahoo groups is the ability to use the the web and e-mail interfaces
If we have the right engine integrating web and email – its relatively easier to add the bells and whistles to the user interface you describe – but I think developing a robust engine is more of a challenge.
It would be loads easier to use some existing Open Source framework – however their seem to be a number of frameworks CM or forum frameworks could be built on to provide what we need – but we also need a strong e-mail interface list server.
maybe the approach followed by http://groupserver.org/groups (an open source project) might be useful?
mark