I wonder what is the optimal regulatory environment for utilities? I've always been suspicious of what seems to be a pretty messed up system (from what little I know).

Perhaps Gore's idea of the electranet is one of the ways to chip away at the system..

The answer to your question is unfortunately far too long and complicated for this format, but the simplistic answer is "more deregulation".  Unfortunately, the botched job we did at restructuring hasn't created much political appetite for that approach right now, but it's the only way to fix the problem in the long term.  I've got pieces in press now in a couple journals, and will provide links on Grist soon as they are live, but the general theme is that:

a) The problem is as much legal as technical, because so long as we don't change the rules, utilities can legally-accurately (but economically and environmentally disastrously) argue that any change of the existing rules is a violation of the 14th amendment of the constitution (no unlawful "takings" of property), on the basis that their monopoly is a contract with the state to provide guaranteed capital recovery and ergo - no matter how poorly considered that capital investment - they must be repaid.  You don't fix this all at once, but once you shift the rules, you gradually wash this out of the system.

b) The biggest opportunities for competitive markets are at the load, where you can use opportunity fuels and recover waste heat - which means that you can be way more efficient, and have much lower capex (since you don't need those expensive wires).  But that is also precisely the part of the grid that our paltry efforts at dereg have stayed away from.  And so we find ourselves in a situation where the ability to really lower costs and carbon are blocked by laws designed to protect utility shareholders at the expense of utility consumers.  (Thus, we need a lot more dereg.)  

Ultimately, the key is to make a decision: do we think that the electric system is inherently impossible to deregulate, in which case we ought to get rid of the profit burden that is simply a tax on the public, or do we think that we really like markets, in which case we expose utilities to truly competitive markets (and enforce antitrust, lest we get more Enron-esque manipulation).  I personally prefer the latter, but so long as we defer the choice, we suffer the consequences.  (One of the ideas I liked most comes from the Galvin Institute, who has suggested that utilities ought to be grid managers - which they have historically done very well - but should not be responsible for metering or billing functions.  In other words, we recognize that they have done a very good role as civil servants, but a terrible job as businesses.  So let's regulate them as what they are rather than what they claim to be, and then let competitive markets address generation both upstream and downstream of utility assets without having to get into a fight with utilities about whether clean, cheap energy is in the best interest of utility shareholders.

I wouldn't put any stock in what Tyson Slocum has to say about the ways to restructure markets.  Apropos of my comment about the propensity of the world to believe things that a $400 billion industry says, one of the worst offenders are the so-called "public advocates" who have come to drink far too much utility Kool Aid as it relates to their long-standing efforts to conflate the interest of a utility with their consumer.  (For example, an oft-used tactic of a utility to block the installation of cleaner power is "if you let them build that plant, it will lower our returns and we'll have to raise rates to other consumers.")  Utter BS, but it has essentially made many public advocates into unknowing shills for the utility industry.  Not all of them mind you, but Slocum is fairly notorious in this regard.  

The specific paper is riddled with errors and misrepresentations from the opening paragraph, so much so that it's not worth the time for a detailed response.  (For example, retail rates in deregulated states have actually risen more slowly since they deregulated than in those states which did not deregulate.  Claiming otherwise is simply false.  Check the DOE/EIA website and do the math yourself.  Yes, the overall rates are higher on an absolute basis, but they were also higher before deregulation, so the only relevant comparison is of the relative increase, which is fairly overwhelmingly in favor of dereg.)  

Notwithstanding the above though, I'd simply make the point that no one has deregulated - we've simply restructured.  This is perhaps a subject for a longer post, but claiming that we've learned anything about dereg based on our slapdash efforts to reorganize a couple players is like saying you know something about fine wine because your cousin once ran a moonshine still.  If Slocum and his ilk don't want to drink moonshine, that's their business.  But that experience doesn't give them any ability to tell us that a nice Bordeaux doesn't taste good.