How to Save a Life: The Inside Story of Grey’s Anatomy by Lynette Rice has the scoop on why he really left the show
executive producer James D. Parriott:
“There were HR issues. It wasn’t sexual in any way. He sort of was terrorizing the set. Some cast members had all sorts of PTSD with him. He had this hold on the set where he knew he could stop production and scare people. The network and studio came down, and we had sessions with them.”
“He didn’t like the inconvenience of coming in every day and working. He and Shonda were at each other’s throats.”
executive producer Jeannine Renshaw:
“There were times where Ellen was frustrated with Patrick, and she would get angry that he wasn’t working as much. She was very big on having things be fair. She just didn’t like that Patrick would complain that ‘I’m here too late’ or ‘I’ve been here too long’ when she had twice as many scenes in the episode as he did.
“When I brought it up to Patrick, I would say, ‘Look around you. These people have been here since 6:30 am.’ He would go, ‘Oh, yeah.’. “It’s just that actors tend to see things from their own perspective… He’s so high energy and would go, ‘What’s happening next?’ He literally goes out of his skin, sitting and waiting. He wants to be out driving his race car or doing something fun. He’s the kid in class who wants to go to recess.”
THR has the whole book excerpt