The arcology, a single massive building that amounts to a city in its own right, has occupied the imagination of visionary architects and science fiction writers alike since the twentieth century. In part it's the extension of the twentieth century experience to its ultimate conclusion; after a period shaped by people moving from rural to urban or semi-urban areas, the idea that the vast, rambling city could be boiled down to its essence within just one set of walls seems like a natural endpoint.
In theory, arcologies are great. They're the absolute antithesis of sprawl, since by their very nature they must be efficient in order to prosper. While today a city can expand by annexing land from rural neighbors and throwing open the floodgates to suburban development - this is pretty much the way Mississauga went from a cluster of small towns to a city of 700,000 in fifty years - adding on to an existing arcology would be a major construction project, extremely taxing in time, effort, and money, and so rewards would naturally stem from working within its limits to the best possible degree. Arcologies would, by necessity, advance the frontiers of knowledge in sustainable living.
I've recently started reading Oath of Fealty, a 1981 novel by Larry Niven and Jerry Pournelle set in and around Todos Santos ("All Saints"), a massive, hulking, fortress-like arcology built upon a riot-ravaged residential neighborhood somewhere in Los Angeles ("Los Angeles") of the near future. Though I can't say which side the authors come down on, having only made it to page 125 out of 324, I can't say their speculations about what life in an arcology might be like are off the mark. Todos Santos is a society tightly controlled by custom, its inhabitants under the constant possibility of surveillance, and absolutely ruthless in dealing with what it perceives as threats to its security.
Part of Todos Santos' reason for being is described as practice for starships - in a world without the possibility of faster-than-light travel, starships would out of necessity need to function as self-reliant, sustainable, entirely self-contained societies. Other arcologies arose out of a belief that denser and denser populations would be the wave of the future - this is their primary function in SimCity 2000, for example, as they allow mayors to gain a large source of tax revenue with a small physical footprint, or as a means to combat overpopulation, as in Robert Silverberg's The World Inside.
Personally, I don't think either motivation will be particularly strong, should arcologies ever be built - particularly not for overpopulation, for exactly the same reason that space colonization would not solve that problem. What I think could spur arcologies is simple - the need for security in an environmentally unstable world. GURPS Terradyne follows this trajectory in some respects - set in 2120, on an Earth that is suffering from environmental degradation, many states operate "complexes" for the poor that amount to arcological slums that police will not enter.
Should environmental degradation continue into the future, I can see the concept of the arcology becoming more and more attractive as a means to potentially create islands of social stability. In Oath of Fealty, Todos Santos is portrayed as something of a vampiric parasite on Los Angeles, sucking whatever jobs and wealth from the city it can while constantly planning new methods to come out on top. It might be more appropriate to cast a twenty-first century Todos Santos as the castle of the feudal lord and Los Angeles as a village of serfs, with LA dependent upon the rigidly controlled, self-sufficient arcology for its own stability and survival. In other environments, social stratification and separation between arcologies and neighboring cities could result in rich, powerful arcologies with feral cities, metropolitan centers devoid of central authority or security - effectively, extending Mogadishu to its ultimate conclusion - just beyond their walls.
So far, no one has solved the multitudinous engineering and ecological problems that would constitute the foundation of a successful arcology. Pioneering efforts such as Biosphere 2 and Arcosanti have not necessarily lived up to their expectations. Nevertheless, no matter how much of a positive or negative effect an arcology would have on its surroundings or the people within it, the underlying technologies and concepts are still worthy of investigation and development. We've come to a point where, for good or ill, humanity is actively managing the planetary ecosystem. Knowing how one works in miniature would be a boon to understanding the whole of Earth.