Introduction
You face a frustrating reality when you build software: projects run late, budgets explode, and simple features demand endless months of labor. Software development defies traditional management and distorts time in ways that baffle outsiders.
Scott Rosenberg investigates this chronic crisis through the story of the Chandler personal information manager project. The central idea is that software fails not from technical complexity alone, but from the unpredictable and shifting desires of human beings.
The book traces the history of programming bottlenecks, examines the psychology of geeks, and explores the endless struggle to translate human thoughts into code.
You will find answers to several pressing questions as you read:
- Why do large software projects constantly fail?
- Can you manage programmers like assembly line workers?
- What happens when a visionary leader attempts to build the ultimate tool?
- How do abstraction layers save programmers from drowning in details?
1. Software Time and the Mythical Man-Month
Software development distorts time, causing developers to lose track of hours in a state of flow or become frozen for months by intractable problems. The temporal experience of programming differs vastly from normal clock time. When things go well, hours vanish; when things go wrong, progress halts completely as developers get stuck between dimensions without a clear path forward.
Scott Rosenberg experienced this firsthand while working on a software project at the online magazine Salon until 5:00 A.M., and decades earlier as a teenager coding the game Sumer in an NYU computer room. The project at Salon was finally published after an all-nighter at 9:00 A.M. in the year 2000. High school students experienced this same phenomenon in 1975 when playing the game Sumer.
The Scale of Software Dependence
Modern civilization relies completely on fragile computer code, making economic systems vulnerable to massive software failures and cost overruns. Software has quietly integrated into every aspect of daily life, from infrastructure and defense to finance and toys. Because software is inherently fragile and difficult to produce, these errors cost billions of dollars and lead to frequent delays and cancellations.
A 2002 study by the National Institute of Standards and Technology found that software errors cost the U.S. economy about $59.5 billion annually. Two out of three software projects come in significantly late, over budget, or canceled.
Brooks's Law and the Tar Pit
Software development distorts time and defies traditional management because adding manpower to a late project only makes it slower.
Frederick P. Brooks observed that scientific management concepts like the man-month fail in software because tasks require extensive communication and suffer from sequential constraints. Bringing a new worker up to speed drains veteran productivity, and individual programmers vary wildly in output.
Brooks drew on his experience managing the creation of the IBM System/360 operating system, where throwing waves of new programmers at the tar pit made things worse. He published these insights in The Mythical Man-Month in 1975.
Software projects typically break down across specific allocations:
- one-sixth of the time is spent writing code;
- half of a schedule is required for testing and fixing bugs;
- ten times represents the typical productivity multiplier of a great programmer over a mediocre one.
2. The Origins and Ambition of Chandler
Mitch Kapor launched the Chandler project to replace bloated commercial software with an open-source personal information manager.
You can trace the initial spark for the Chandler project back to Mitch Kapor's daily frustration with managing a heavy, expensive Microsoft Exchange server just to sync a few office calendars. Kapor found himself acting as the first responder to server problems in a closet outside his office. He viewed Microsoft's software as bloated, costly, and overly complex for small groups, requiring dedicated hardware, Windows licenses, Exchange software licenses, and outside consultants.
In 2001, a Microsoft Exchange server sat in a closet outside Mitch Kapor's office in downtown San Francisco, handling small-group scheduling for his office suite and his wife Freada Kapor Klein's nonprofit.
The Microsoft Exchange Frustration
Projects that attempt to blend open source collaboration with traditional startup execution can become trapped in a dispiriting limbo. Mitchell Kapor's Open Source Applications Foundation sought to build Chandler as an open source personal information manager while functioning like a classic startup with a core team of paid developers. Operating by consensus and tackling novel data approaches slowed down decision-making and prevented them from easily capturing the speed of traditional commercial releases.
Kapor founded OSAF with $5 million in funding, moving from Belmont to eco-friendly offices in downtown San Francisco, while development manager Michael Toy posted on his blog about the slow pace of consensus building.
From Lotus Agenda to Open Source
Lotus Agenda introduced a revolutionary approach to data organization, bridging rigid computer databases and free-form notes, though it proved too complex for the mainstream market. Developed by Kapor, Jerry Kaplan, and Edward J. Belove, Agenda allowed users to dump unstructured information on the fly and organize it later through automatic assignment and customizable views. Despite winning a cult following and rave reviews, it struggled because Lotus did not know how to market it and it ran on primitive DOS systems.
Reviewer James Fallows of The Atlantic wrote in 1992 that Agenda was far and away the most interesting and valuable computer program he had tried.
Kapor chose to make Chandler an open-source project to combine public contribution with his preferred software design work, despite his historical ties to proprietary software capitalism. After evaluating the open-source movement in the late nineties and consulting figures like Linus Torvalds, Kapor saw open source as both a maverick ideal and a pragmatic approach during a severe tech downturn. It freed him from old corporate conflicts over copyrights and allowed him to foster shared knowledge.
In the spring of 2001, Mitch Kapor established the Open Source Applications Foundation as a nonprofit organization and hired Morgen Sagen and Al Cho.
3. The Chronic Epidemic of Software Failure
Large software projects across government and private industry are marked by chronic failures, massive cost overruns, and missed deadlines.
Projects repeatedly suffer from moving targets, fluctuating goals, and unrealistic schedules. This dynamic generates a well-documented genre of disaster literature. Unlike bridge building, the software industry frequently covers up or fails to systematically learn from its past failures.
Metrics of Failure
The Standish Group's CHAOS Report revealed that the vast majority of software projects fail to meet their initial specifications. These endeavors remain consistently over budget and late.
The 1995 study surveyed 365 information technology managers at large, medium, and small companies. The results exposed stark realities:
- 16 percent of projects were successful;
- 31 percent were total failures;
- 53 percent were project challenged initiatives.
Subsequent updates showed slight improvements in total completions. Yet the proportion of challenged projects remained persistently high, with a 29 percent success rate and an 18 percent failure rate reported in Standish's 2004 update.
Historical Disasters
Government agencies and large enterprises frequently encounter massive structural barriers when building custom code. You can trace these difficulties through a series of expensive public cancellations.
The Federal Aviation Administration suffered through the Advanced Automation System debacle, which reached a peak cost of 1 million dollars a day. The Internal Revenue Service spent 2 billion dollars over ten years on a failed system upgrade that it finally canceled in 1995.
The Federal Bureau of Investigation attempted to build the Virtual Case File system as part of a 170 million dollar third phase of the Trilogy project. The agency ultimately scrapped it after a decade and 400 million dollars total spent across the initiative, failing to replace 30000 desktop computers.
Software developers are frequently motivated by a deep-seated idealism and the ambition to change the world. This emotional drive helps them endure the immense frustrations of coding.
Former Vice President Al Gore surprised the Chandler team at 543 Howard Street in San Francisco by walking into their weekly staff meeting. He praised their work as changing the world, echoing a sentiment famously used by Steve Jobs to recruit John Sculley. Rosenberg argues that this shared idealism is a direct psychological consequence of having to wrestle with daunting abstractions and bugs.
4. Prototyping, Python, and Abstraction Layers
Programmers manage complexity by building piles of abstractions and using interpreted scripting languages to boost productivity.
Software is built as an onion-like pile of abstraction layers. Each higher layer translates complex lower-level operations into simpler, more general units.
Programmers transitioned from early binary and assembly instructions like mov to high-level compilers and interpreters. You build piles of abstractions starting from assembly language up to high-level languages, combining multiple smaller operations into a single named unit to manage complexity.
The Power of Python
Guido van Rossum created Python as an interpreted language that translates source code line by line at run time. This allows code to be tighter and less verbose compared to lower-level languages like C or Java.
Interpreted scripting languages can significantly boost programmer productivity by abstracting complexity and avoiding upfront compilation delays. Python programs can accomplish tasks using three to ten times less code than C or C++ programs.
Andy Hertzfeld found he was three times more productive using Python than using Java, which was three times more productive than C.
Architectural Layers
When introducing new technology, teams must plan to discard their first implementation because they will not get it right the first time.
Andy Hertzfeld built the Vista prototype and Shimmer database for OSAF to test assumptions about user interaction and data management before planning to throw it away. Following Frederick Brooks's advice, Hertzfeld built Vista as a user-facing prototype stored in an RDF-based repository called Shimmer, designed solely to be milked for lessons and then discarded.
Morgen Sagen and Al Cho filled whiteboards before realizing their three-part format matched RDF standards already developed for the Semantic Web. Complex information can be structured using a simple three-part format of subject, verb, and object.
OSAF creators discovered that their independently designed data structure for sorting information mirrored the Resource Description Framework used in Tim Berners-Lee's Semantic Web vision. This allowed them to store all information in triples declaring relationships.
5. The Impossible Triangle and Geek Psychology
The Impossible Triangle
Software engineering is permanently constrained by the impossible triangle, forcing managers to choose between speed, cost, and quality. You can pick any two of these options at any given time, but you cannot have all three. Managers often try to push sliders on a mixing board to have it all, but reality forces constant trade-offs.
Software engineering is permanently constrained by the impossible triangle, forcing managers to choose between speed, cost, and quality.
Dan Shafer at Salon told the naive technical crew that he could make things fast, cheap, or good, but to pick any two.
Personality Profiles of Geeks
Information technology professionals possess distinct personality profiles characterized by heavy preferences for logical thinking and introversion. Abby Mackness of Booz Allen Hamilton presented results from the Human Dynamics of Information Technology Teams study evaluating 600 IT professionals across 77 teams.
The study revealed stark psychological patterns among technical workers:
- 77 percent of IT professionals reported a preference for thinking decision-making;
- 41 percent reported being introverted thinkers.
These preferences drive a heavy avoidance of collaborative efforts and interpersonal trust-building. You will notice that counting lines of code is a deeply flawed and deceptive metric for measuring software productivity in such teams. Tracking lines of code encourages sloppy, bloated, and broken code because it treats volume as productivity. True efficiency often involves writing fewer lines to make a program faster and smaller.
Bill Atkinson at Apple was asked to report his weekly lines of code for the Lisa computer and reported negative two thousand (-2000) because he had just rewritten a portion of Quickdraw to be shorter and more efficient.
Programmers frequently get derailed by recursive tool-building and preparation loops known as yak-shaving or axe-sharpening. Working on tools is often more fun than doing the actual job, leading to an infinite regress of secondary tasks required to support the primary task. Abraham Lincoln was once given 6 hours to chop down a tree and instructed to spend the first 4 sharpening the axe. James Gosling compared tool-building to Zeno's paradox, where sharpening an axe requires a sharpening stone, which requires finding the right stone, which requires going to Baffin Island, which requires building a dog sled.
6. Design, Specifications, and the Getting Things Done Workflow
Translating abstract human requirements into exhaustive specifications requires championing the ordinary user experience.
Mitch Kapor first delivered his Software Design Manifesto at the 1990 PC Forum gathering of computer industry luminaries. Computing professionals often ignore poor usability and blame the user. Design is not just applying attractive graphics. It acts as the fundamental creative act of imagining user needs and building appropriate software structures based on Vitruvius's architecture principles of firmness, commodity, and delight. Kapor has advocated for software design as a profession for two decades.
Specifications bridge human goals and machine precision. Visual specs detail screen layouts, while interaction specs dictate responses to user input. Because programmers are literal-minded, ambiguous specs lead directly to software failures and comical errors.
The book cites the movie This Is Spinal Tap, where a napkin sketch with a misread apostrophe turned an eighteen-foot Stonehenge monument into an eighteen-inch farce.
You must handle item mutability and polymorphism by creating undifferentiated notes that acquire specific functional attributes through stamping. Instead of forcing users to declare an item type upfront, Chandler allowed users to start typing a generic note and later stamp it as an event, task, or email. Donn Denman built a working detail view where a user could take a note titled "Fantastic Recipe", stamp it as a task, and automatically acquire requester and requestee fields.
The Software Design Manifesto
New UI designer Mimi Yin integrated David Allen's GTD productivity framework into Chandler. She designed a timeline-oriented dashboard inbox that organizing incoming items by past, present, and future to help users process open loops without feeling overwhelmed. Mimi Yin attended a daylong seminar by productivity coach David Allen and subsequently introduced GTD-inspired workflows to OSAF's design group in fall 2003.
Ambiguous terminology and overlapping word definitions across development and design teams cause severe communication friction in software projects. Developers and designers used the same words—such as item, data model, scheduled task, and event notification—to mean entirely different things, leading to namespace collisions. Brian Skinner won OSAF's conference room naming contest with imaginary place names like Avalon and Arcadia, though employees frequently confused the rooms because their names started with the same letter. Skinner won $150 in prize money and used it to fund a happy hour at Kate O’Brien’s bar.
Integrating Getting Things Done
You need to turn customer desires into clear rules. You can follow these specifications practices:
- Translate customer goals and desires into detailed marching orders for programmers.
- Exhaustively detail visual elements such as screens, button placement, menus, typefaces, and colors.
- Record interaction specs detailing how the program responds to clicks, drags, and typing.
You can manage items flexibly in your software repository. You should apply these structural steps:
- Allow users to enter unstructured information as a generic note first.
- Enable users to apply a specific stamp to transform the item into an event, task, or mail object.
- Dynamically attach new attribute sets to the item in the repository while maintaining its original ID.
7. The Pivot to Server Standards and Dogfooding
Adopting established Internet standards and forcing developers to rely on their own beta software accelerates bug discovery.
Software development organizations use the internal practice of dogfooding to accelerate bug discovery and drive quality by forcing developers to rely on their own beta software. At Microsoft and OSAF, developers run their daily systems like payroll, email, and calendars on their own unreleased software. The goal is to shove developers' noses into their products' flaws to find and fix bugs faster.
At OSAF, Mitch Kapor needed a calendar to stay synchronized with his assistant Esther Sun, making a shared calendar the primary use case for making Chandler dogfood-able.
Adopting established Internet standards like WebDAV allows a software project to leverage existing remote sharing and collaboration protocols rather than inventing them from scratch. Lisa Dusseault introduced OSAF to WebDAV, an extension of HTTP built into Windows and Mac OSX that manages collaborative editing, file locking, and conflict resolution on remote Web servers. This forced OSAF to reverse its stance and adopt a server-based approach.
Mitch Kapor announced a major shift in point of view at a June meeting, recognizing that his initial frontier idealism regarding server-free environments was impractical.
Planning and retrospectives
Using physical sticky notes organized on a whiteboard by release buckets provides managers with a tangible, granular view of project scope and scheduling reality. To map out future releases, OSAF managers used Post-it notes where each note represented a uniform block of a single developer's time, categorized by development groups and targeted dot releases.
Lisa Dusseault, Katie Parlante, Sheila Mooney, and Chao Lam used sticky notes on the office whiteboard, ultimately generating 138 notes that revealed a two-year timeline to reach a dogfoodable release.
You can organize release planning on a physical board by applying these practical steps:
- Clear a section of a whiteboard and draw vertical columns labeled for each dot release such as 0.4, 0.5, 0.6, and 0.7.
- Draw horizontal divider lines corresponding to each development group: apps, services, and repository.
- Write individual tasks on Post-it notes representing roughly even granularity of developer time.
- Affix the sticky notes to the appropriate grid cells on the whiteboard to visualize the workload.
10 Key Ideas
- Software development distorts time and defies traditional management because adding manpower to a late project only makes it slower.
- Mitchell Kapor's Open Source Applications Foundation sought to build Chandler as an open source personal information manager while functioning like a classic startup with a core team of paid developers.
- Large software projects across government and private industry are marked by chronic failures, massive cost overruns, and missed deadlines.
- Programmers manage complexity by building piles of abstractions and using interpreted scripting languages to boost productivity.
- Software engineering is permanently constrained by the impossible triangle, forcing managers to choose between speed, cost, and quality.
- Translating abstract human requirements into exhaustive specifications requires championing the ordinary user experience.
- New UI designer Mimi Yin integrated David Allen's GTD productivity framework into Chandler.
- Adopting established Internet standards and forcing developers to rely on their own beta software accelerates bug discovery.
- Software development organizations use the internal practice of dogfooding to accelerate bug discovery and drive quality by forcing developers to rely on their own beta software.
- Using physical sticky notes organized on a whiteboard by release buckets provides managers with a tangible, granular view of project scope and scheduling reality.