Few people realise how much software runs today’s cars, observes Les Hatton, emeritus professor of forensic software engineering at Kingston University, UK. “We sit at home and can’t work out why the printer isn’t working, and then we’ll get in a car where there’s probably more software underneath,” he says. “People should think twice about this.”
As vehicles become smarter and more connected, embedded software is likely to draw the eye of ‘digital vandals’ and extortionists. Alongside the prospect of hacking, automotive software brings the risk of serious digital error and subsequent safety recalls. Thus, in the automotive sector, the risk lies in on-board software as well as business software.
Experience suggests that modern cars contain a great many software flaws. Though most bugs cause only glitches, a few defects can create more serious outcomes and yet still lurk undiscovered. “Some errors require a relatively unusual set of circumstances before they’re triggered,” Hatton notes. “This is actually one of the longest-known empirical properties of software, discovered by engineer Ed Adams in 1984.”
Top tier suppliers like Continental have to take steps to minimise the risks posed by software in the many products they provide to OEMs around the world. In 2012, Continental established a Security & Privacy Competence Centre to take charge of this task, headed by Stefan Römmele (pictured). He leads a team of more than 30 cyber-security specialists located in Germany, Japan, Singapore and the US.
Making secure software is not simply about tools and technologies, Römmele cautions. “It depends on people and how they behave,” he observes. “Continental is developing its own processes for engineering in alignment with the upcoming international standard [ISO 21434], for incident response management and for testing.”
Software is embedded in a vast array of modern components, with today’s cars likely to have up to 100 electronic control units (ECUs) running every kind of automated system, from anti-lock braking to climate control.
Continental currently employs around 15,000 software developers and many more will be employed further back in its supply chain. As advanced driver assistance systems and autonomous technologies expand, software complexity is bound to grow.
“Continental is a tier one supplier, but we are also using components – both hardware and software – from tier twos,” says Römmele. “We have to make sure our suppliers create software with security by design, that it’s kept updated and also that they have incident response management in place, reporting to us if needed.”
Checks are made to ensure that suppliers actually use secure methods. “It’s a stepwise approach,” Römmele says. “The supplier has to say how it is managing and creating security and we carry out onsite audits.”
Römmele explains that all Continental products go through a threat analysis and risk assessment (TARA). “We have a method we’ve rolled out across Continental,” he adds. Internal guidelines spell out how to address risks and test fixes.
Once in the field, products have to be updated securely. Digital signatures and certificates are used to verify that new code is authentic, from a trusted source and unaltered in transit whether by USB, CD or over the air. Credentials are checked not just when updates are installed but whenever the ECU powers up, according to Römmele.
Continental’s security lab also carries out penetration testing, using a variety of hacking methods to try to breach its own defences. Risks evolve over time as new attacks come to light. Römmele concludes: “If we figure out after launch that specific new threats are important, we have to act.”
Today, bugs crop up in the same scattered fashion, but in shorter order. “If you send a million copies of a braking system out into the world, you’ll see defects much more quickly,” says Hatton. “So, one of the advantages of worldwide spreading of software is [that] you shrink the time required for unusual defects to appear. And they will appear. Nobody can test for all defects – it’s not feasible.”
Estimates from suppliers like Bosch and Continental suggest that today’s premium cars depend on as many as 100m lines of code. “Staying below one defect per 1,000 executable lines of code for the entire lifetime of the software is extremely difficult,” Hatton observes. “Less than 5% of all software genuinely falls into that category. The most reliable large application the human race has ever created is the Linux kernel, which is believed to have about one defect per 10,000 lines of code, in the entire lifecycle of the software.” Put simply, 100m lines of code implies tens or even hundreds of thousands of hidden defects.
Focusing on binariesAt the Detroit Motor Show in January, Canadian company BlackBerry launched a new tool to help battle these inevitable bugs, reducing the potential for both cyber-attacks and functional failures. Called BlackBerry Jarvis, it scans software for many different types of vulnerability and error.
While BlackBerry is known for smartphones, the company has become a force in trusted software. It owns the QNX operating system, which has underpinned critical applications from high-speed trains to internet routers for decades and is now employed by about 40 carmakers.
“The auto industry is starting to realise that if they want people to trust their cars, and if they want autonomous cars to become a mainstream reality, they are going to have to be very secure,” says Alex Manea, chief security officer at BlackBerry (see box, below).
Boulton: No tool can spot every errorAdam Boulton, the company’s business systems CTO and architect of BlackBerry Jarvis, concedes that no tool can spot every software error. “Jarvis doesn’t mean you can eradicate source code reviews,” he warns. “I never want to give people that hope – that’s not realistic.”
Source reviews look for mistakes or bad practice in code written by humans, whereas Jarvis analyses the binary executable generated from that code, looking for inconsistencies and errors. “There is an overlap of issues you would find doing a source code assessment and a static binary analysis,” Boulton explains. “Both techniques have advantages.”
Jarvis handles binaries for very practical reasons. “You don’t always have access to the source,” Boulton observes. “There is a benefit from having the source code, but that pretty much never happens. It’s particularly uncommon in automotive.”
As with hardware, there can be a complex supply chain in software and each link will aim to protect its intellectual property. Shipping binaries rather than source code protects investment. “It’s often very difficult to get hold of source code,” Boulton says. “And even if you do, testing it doesn’t scale particularly well.”
Testing the tech with JLRBoulton cites a pilot programme which BlackBerry carried out for Jaguar Land Rover, where Jarvis was pitted against a manual code review. “We actually fared extremely well, to the point where we found every issue like for like,” Boulton says. “JLR put their own comparison together; it was entirely independent. They were thrilled, because the source code assessment took about 30 days, whereas Jarvis took seven minutes.”
[mpu_ad]Jarvis works by decomposing binary files into a form where structural problems and vulnerabilities like buffer overflows can be spotted. Scans are initiated from a developer’s build environment and errors flagged up on a dashboard. Jarvis yields quick results partly because it runs in the cloud, with Amazon Web Services providing resources on demand.
Since it deals with binaries, errors spotted by Jarvis will often need to be reported back to the software’s supplier for a fix. “Jarvis doesn’t just tell you the issue, it tells you how to remediate it, where it is and gets everything ready for the supplier,” Boulton adds.
As a side benefit, Jarvis can also flag up sloppy coding. “You can check if software adheres to your own standards or coding standards like MISRA C or CERT C,” Boulton notes. “We do see some violations coming up because typically, you’re relying on trust in a software supply chain. You ask if people are following standards, they say yes. Now you can actually check. If they’ve been riding on the system thinking it’s going to remain a trust exercise forever, that has now changed.”
At the Geneva Motor Show in March, BlackBerry’s chief security officer, Alex Manea (pictured), gave Christopher Ludwig his views on how the tech industry can share software security insights with automotive manufacturers...
Can you give me some background about BlackBerry’s current activities?When people hear about BlackBerry, they mainly still think of the smartphones, but over the past few years we have shifted to becoming a software and security company. We don’t make smartphone hardware any more and we are fully focused on software and security. Ultimately, we want to help to secure the enterprise of things, which is the concept that these are all the endpoints that connect to an enterprise IT environment.
We’re really starting to see the rise of the IoT [Internet of Things] era – not only connected cars, but connected fridges, wearables, all of these types of things. On the one hand, the IoT has the same security concepts that we apply to laptops or desktops; we can apply them to a connected car as well. So, making sure you encrypt all the data, that you hash your passwords – these apply to cars as well.
What is interesting about cars is that they fundamentally change the threat level, because when you are protecting a computer or smartphone it is about protecting privacy; when someone hacks a car it is more about threatening safety, not only of the person in the car, but also the people around it. That is where we see a very new threat level.
The other thing about cars these days is the sheer complexity; a luxury car today can have about 100m lines of code.
And that figure is about ten times that of a modern aeroplane?It is ten times an aeroplane, but it is also ten times the Android OS [operating system]. A lot of people don’t realise how much code is in the car, but if you assume there is software vulnerability for every 10,000 lines of code, and you have 100m, that is a lot of vulnerability potentially in one car.
There are so many different ways that cars connect to the internet and to the outside world, everything from the OBD2 port to Bluetooth, to wi-fi, and to 3G, 4G, 5G. Hackers are always looking for the weakest link, so the more links you have, the more software you have, the more vulnerability you have. It is interesting because securing cars is not only important because of the safety element, but also because of the huge attack surface.
And so at BlackBerry, to help secure that, we have a fully owned subsidiary, called QnX, which is deeply embedded in the automotive industry, working with around 40 different automotive manufacturers to provide a core platform for security. They provide the QnX operating system, infotainment system, over-the-air software updates and a secure manufacturing mechanism that can actually inject keys within the ECUs and create a secure hardware route.
So ultimately, what we are doing with car manufacturers is creating a security platform that people can build on [for] autonomous vehicles. We recently announced partnerships with Nvidia and Aptiv, who are building self-driving car mechanisms and have selected QnX as the basic operating system.
What do you see as the biggest vulnerabilities for vehicles?I don’t know that I would look at any one point as the biggest vulnerability, but if I were a hacker and wanted to look at hacking a car, I would look at the OBD2 port because the OBD standard does not have many security standards built in, and by definition every car needs to have an OBD2 port.
At the same time, if I am looking at it from a broader standpoint and how I hack not just one car but a large series of vehicles, at that point I would look at hacking infotainment systems, and to get over not just to infotainment but core driving systems. OBD2 is easier, but internet-based hacks could affect a large number of cars.
Hacking the infotainment system could provide access to core driving systems across a large number of vehicles, warns ManeaSo then they would be looking to hack into the cloud?Yes. I would also look at hacking software updates. If people are sending software updates over the air, are those updates being properly authenticated, are they digitally signed, are they checking the signatures? If I can put my software on your car, it is not your car any more – it is my car.
Do vehicle manufacturers have a strong enough focus on cyber-security?They have become much more aware over the last five years because there have been so many high-profile car hacks. The auto industry is also starting to realise that if they want people to trust their cars, and if they want autonomous cars to become a mainstream reality, they are going to have to be very secure.
What we are seeing is a lot of auto manufacturers partnering with companies like BlackBerry because they have a legacy on the mechanical side of things, and they are partnering with tier ones and tech companies to really create and secure those software platforms.
Will we see new partnerships developing, for example between software specialists and their hardware counterparts?Yes, that is ultimately where it is going. Right now one of the things that scare me is the number of ECUs on the car, as cars can have dozens of them. If you go back to the concept of the weakest link, the more ECUs that you have, the more likely that one of them can have critical software vulnerabilities. So one of the things we are starting to see is a conglomeration of these ECUs; how do you reduce the number of ECUs and how do you start thinking about it in terms of security domains?
Is this an area BlackBerry and QnX are looking at?Yes – and another piece of technology we are offering that is getting very popular is hypervisor technology. The idea being that we want to separate safety-critical parts from the non-critical parts, so that [if] somebody hacks into the infotainment system from the cloud or the consumer downloads malware, that piece of malware can’t jump over to the driving systems and take over the steering wheel or the brakes.
Do we need to start upstream when thinking about how to build secure systems?Yes, we need to think about how we secure the supply chain and manufacturing. One thing that we offer to tier ones and OEMs is a solution by which we can do our own key injection to create a hardware route of trust within the ECU. And what this essentially means is that the ECU and different components can be manufactured physically anywhere in the world, and we can secure them remotely and create that hardware route of trust so that when you start putting the OS on top of that and start putting in applications, we can verify [them].
We are creating a secure ecosystem and putting that route of trust directly into the hardware, which is exactly what we have done with our phones.
Who do you think automotive manufacturers should learn from in terms of securing their products and supply chain?I think car manufacturers can learn a lot from the smartphone industry, because if you look at how the smartphone has evolved over the past ten to 15 years, you are going to see a similar evolution with cars – anyway, cars are mobile devices on four wheels. I do think that if you look at where smartphones started off, and where they are today, in the beginning security was not seen as a feature but was added later. That is never the way we look at it at BlackBerry, but the net result is that we have seen a lot of smartphones targeted by hackers.
Carmakers should look at the evolution of those industries and see that if you don’t put in the right tools from day one, and you find out ten years later that your devices are hackable, you are going to be in a lot of trouble – so you better put it in straightaway.
"What is interesting about cars is that they fundamentally change the threat level, because when you are protecting a computer or smartphone it is about protecting privacy; when someone hacks a car it is more about threatening safety, not only of the person in the car, but also the people around it. That is where we see a very new threat level." - Alex Manea, BlackBerry
