In a very detailed article, Slack engineers Ahmed Eid and Arpita Patel provide an interesting look at the processes they’ve adopted over the years to improve the developer experience in many different areas, and the tools they’ve used to that end.
The developer experience at Slack is delivered by a dedicated team of eight people, formed to answer increasing costs as the organization and its development team grow. In the most costly areas of the development process, Slack engineers focused on merge conflicts, long-running CI jobs, flaky tests, and CI infrastructure failures.
While developers can learn to fix some of these issues, the time and expense spent cannot be justified as the team grows. Having a dedicated team that can focus on these problem areas and identifying ways to make developer teams more efficient will ensure that developers can maintain intense product focus.
Costing an estimated $2,400,000 per year for 100 developers, merger conflicts are the costliest pain point. This resulted in Xcode project merge conflicts, parallel merges to main, and a lengthy request review process.
Xcode projects are notoriously difficult to merge without running into multiple conflicts. To solve this problem, Slack engineers used Xcodegen to generate .xcodeproj files from a YAML file, a more tolerant format.
Multiple concurrent merges increase the risk of merge conflicts, stopping the merging of additional PRs until the conflict is resolved. For this, Slack adopted Aviator to queue all PRs and process them one by one. Instead of merging a PR directly into main, Aviator tries to merge main into a developer branch. This breaks the main stage, rejects the PR and informs the author.
Finally, to speed up the pull request lifecycle, Slack engineers have found it useful to introduce timely alerts for PR assignments, comments, approvals, and direct messages for successful builds, including the option to merge PRs without leaving Slack. To accomplish all this, they created their own GitHub bot called MergeBot.
Mergebot helped shorten the pull request review process and keep developers in the flow. This is another example of how saving just five minutes of developer time saved a 100-developer team ~$240,000 in one year.
Fortunately, GitHub supports a similar feature, called scheduled reminders, although it doesn’t offer one-click merging from the message itself.
Action taken at Slack to improve the development experience, not just PR/merger process improvement. Another area of high cost is testing and failure in their CI infrastructure. In the first instance, the solution was parallel test execution with a strategy to run only the tests strictly necessary for a given PR based on the PR difference. In the second count, BuildKite proved to be effective in increasing CI-infrastructure reliability.
According to Slack, improving the developer experience has made developers happier and reduced overall development costs. If you’re interested in the full details on how Slack got it, don’t miss the original article.
