Get out of Git hell: preventing common pitfalls of Git

At Amazon, Release Engineering falls under what we call Operational Excellence: designing, implementing, maintaining, and releasing a scalable product. There is an even more basic component that is often ignored: source control. Good source control practices are necessary but not sufficient for delivering good software. Over the 25+ years source control has been used, each tool has come with its own set of pitfalls: CVS, subversion, mercurial, and most recently, git. For decades, the unwritten rule has been for each organization to identify and mitigate these pitfalls independently, with an expectation that the next innovation would remediate it. This approach scales neither for large organizations such as Amazon nor the software engineering community at large. The real source of this dysfunction—remote collaboration between software engineers—must be examined and ultimately fixed. In the interim, it is up to the engineering community to share practices independent of software process to make up the difference. At its core, source control is a fundamental tool of software engineers, expected to be easily understood and “just work;” this assumption is invalid on a number of dimensions. Neither Software Configuration Management (SCM) nor the tools used are intuitive to new practitioners, and must be taught. The changing landscape of newer tools misleads even expert users of past tools who are not screened for this critical skill. And finally, success is dependent upon synthesis of past experience and tuning a pre-determined process to both project goals and the team. Success, then, is stacked against the engineering team—so what happens when source control usage goes horribly wrong? The baseline and team end up in “Git Hell,” slowed down, or even blocked, by the very tool that facilitates collaboration and parallel development. “Git Hell” originates from various sources: poor tool design, misuse or misconfiguration of the command line interface, and lack of understanding of the “nuts and bolts” of the tool. For example, poor interface design or configuration, even with the command line interface, has widespread impact. A substantial flaw in the mechanics of git push caused substantial pain at multiple engineering firms. The interface was straightforward: a push sends all branches with updates to the server; adding the -f option forces an update; combining them proved disastrous, as an engineer with minimal knowledge of git could harm the integrity of the baseline without even realizing it. This prior version required each developer to add local configuration to his workstation, ensuring others in the future would repeat the mistake. These classes of issues are repeated at company after company, group after group—illustrating a systemic problem with git, its configuration, the instruction in its usage, and the interaction between collaborating engineers. To combat this, I generalized preventative measures as a workaround in a workshop entitled “Get out of Git Hell” that can be shared among engineers regardless of experience or process, at least until the root causes can be studied and remediated.