A GitLab-assisted DevOps Journey at Jaguar Land Rover

This time I have chosen to summarize a lecture from the recent DevOps Enterprise Summit in London. The conference was organized by Gene Kim, the author of a well-known book, The Phoenix Project. Lecture recordings were published a few days ago.
I have selected this specific lecture for a number of reasons:

• It’s an interesting story about DevOps implementation, with trials and errors. The speaker, a manager from Jaguar Land Rover, is candid about the things they tried which didn’t work out.
• Their system is delivered to hundreds of thousands of vehicles.
• The lecture discusses interesting usages of GitLab , AWS, Docker, GitLab CI/CD and more.

You can find the full recording of the lecture at the end of this article.

DevOps Journey at Jaguar

The speaker was Chris Hill, Head of System Engineering at Jaguar Land Rover in Portland, OR.

Chris manages a team of developers, who aim to improve the delivery process of their system’s software. That system contains 75 modules and is deployed to 750,000 vehicles already on the road.

Chris presented the Jaguar I-PACE, the first all-electric Jaguar’s vehicle, which will be launched next month. He told about their DevOps process and their “DevOps Journey”. That journey started two years ago and is still ongoing. As Chris said, this is a “continuous improvement” process.

Chris’ team is in charge of Infotainment, the interface screen which you see to the right of the driver (with the Jaguar logo in the picture above). Infotainment is one of 40 computerized devices in a car, all of which exchange data in real time.

The team develops embedded software for a device that must not distract the driver, and these are only a part of their constraints.  Chris mentioned the well-known tend in the automotive industry, of software becoming ever more important, as cars have become ever more technologically advanced (for example with the development of autonomous vehicles).

Jaguar Land Rover Company in numbers:

• Number of employees: 40,000
• Revenue: £24 billion
• Cars sold in 2017: 604,000
• Software developers:  5,000

Their DevOps Journey
In the beginning it looked like that (click on the picture to enlarge it):

They started their DevOps journey in 2016 with 2 employees in Portland, OR by reading books: The Phoenix Project (the factory icon at the end of the path is taken from that book’s jacket) and the Goal by Eliyahu Goldratt (about the Theory of Constraints).

(By the way, I have read both books and highly recommend them.)

The next step was to build a server:

They started with Linux, Git, GitLab and GitLab CI as Chris preferred open source software.

(In another article Chris explained that although the company in general uses Atlassian BitBucket and Bamboo, he went for GitLab (a reference is at the end). According to Chris, “Atlassian’s software is very good for managing parent-child relationships and collaboration with JIRAs. But sometimes vendors can start to get involved with other parts of the software development lifecycle that aren’t their core business, and customers get sold an entire package that they don’t necessarily want.”)

Back to the Journey

At that stage they viewed as progress the very fact that builds were running on a dedicated server rather than on users’ workstations.

Now they had three projects, and they discovered new problems:

• Only one person in a project knew how the build works and how to run it and how automatic tests work, and Chris’ team gradually moved to CI.
• Some builds consumed all of the server’s resources, so that sometimes version control stopped working – that’s because both the version control and CI ran on the same machine.

So, they bought more hardware and now had three servers and three CI slaves.

As the situation improved, and they added more projects to run in the same environment, they hit new bottlenecks on runners (machines that run GitLab CI), especially at peak hours, when everybody was running builds.

Chris said that at that point users began complaining and that they even preferred to go back to running builds on their workstations.

As a result, Chris and his team decided to change direction and instead of adding new hardware, to run GitLab CI in a family of Docker containers, short-lived virtual machines.

Code-based infrastructure

That was a significant change: for the first time their infrastructure was code-based (they used Packer by HashiCorp).

At that point running a build became self-service for the software engineers. They got the capability to run it by themselves.

That worked fine, the complaints stopped, and more projects and users jumped on the bandwagon.

And with growth, new challenge arrived… The new reality created a mismatch between capacity and resources. The most troubling issue was hardware maintenance (caused by power outages, network failures, CPUs overheating, etc). That was the point when they decided to move to the cloud:

Now instead of short-lived containers running on their own hardware, they had short-lived containers running on AWS EC2 machines. Each time they performed a git commit requiring a build, a temp machine was created. The build would run on that machine, and then the machine would be destroyed. Thus, they solved the problem of capacity shortage.

By that time 50-60 developers were using the system. More projects were added, especially larger projects, which were considered more critical for the company.

At that point Chris was asked to relocate to England, closer to the company’s headquarters, and to continue his work from there. That was the sign that from the company’s point of view, his project had become more important.

Their next challenge was to provide better and faster feedback.

They cut the time it took to get feedback on a new feature from 4-6 weeks to 30 minutes.!

They achieved that by automating each step as much as possible reducing the time it took to perform code reviews and pull/merge requests.

They simplified the build system and gave each developer the keys to run a build as a self-service by himself. Chris mentioned as an example that a JavaScript developer, who does not know Linux, now knows how to add his component to the build system and does it in 30 minutes.

According to Chris, by simplifying things they succeeded in speeding up project development and deployment. Now they are able to deliver their software automatically to 9 types of vehicles, as compared to zero automation a year and a half ago.

They are able to deploy their software directly to the cars in an hour! What used to take 6 weeks, takes now only one hour, from the build until the fix is running in the car. Moreover, the deployment is possible even when the car is moving (by maintaining dual environments — the “blue-green” deployment approach), even 700 times a day, including the capability of delivering from any branch.

Summing up – Chris’s Takeaways:

• Chris acknowledged that he learned things he did not know before in the world of software development (inspiration, persistence and continuous improvement).
• It is important to get an advantage over competitors, and implementation of DevOps can give you that advantage, as they have shown by the speedy delivery of I-PACE and overtaking other electric car manufacturers.
• It is important to ask “Why?” at each step, to challenge the team’s way of thinking, and thus make working processes better.

In the end, Chris said that the journey is not yet finished, and that this is a journey of continuous improvement.

[chimpy_form forms=”2″]