I was also going to mention Peopleware by deMarco and Lister. Written in 1987, it covers exactly this issue. Their solution was neither individual offices nor open plan. The best option was a "workteam sized" (8-10 people) space with reasonably high partitions and individual spaces within. As I remember (it has been some time) the partitioning was also flexible to allow different dynamics for different teams as projects come and go. This sounds very similar to what you have described above. The idea being that those people who work together can collaborate without interference from outside. And with the space each person had their own corner to retreat into.