Éireann: Things are blowing up, in industrial systems, here in Germany, this
year! I had hoped that these things wouldn't happen. This kind of future
wouldn't be one that we are living in. But unfortunately it is. And I hope that we
can make that better, partly through the course of this talk. But more, I think, in
the future with your help and your work. So I'm sorry to begin this presentation
with such a dark thought but: This year's theme is a new dawn. And it's always
darkest just before the dawn. So we're going to go through some of that darkness
in industrial systems and SCADA-systems to get to a better place, right? Now with
that said no hacker really gets to be where they are without the help of other,
right? We stand on the shoulders of giants and part of the key is not stepping on
their toes, on the way up. So I would like to say thank you to a bunch of people who
are here and also some people who aren't here. Particularly the Oslo hackerspace
where I hang out. And these people have taught me a lot of things not just about
technology but about life and on "aprendo", which is how Goya signed some
of his last paintings and sketches - which basically means "I'm still learning". OK.
So with that said I hope that you will enjoy this talk with its darkness and its
humor all at the same time. I used to be in circus, as you may have guessed from
the mustache. So I encourage you not just to view this as a technical vulnerability
presentation but also as kind of live technical standup comedy. Instead of jokes
we have vulnerabilities. And I hope that you will enjoy them. So these
vulnerabilities are in switches. I chose to focus on switches and that will become
clear throughout the presentation, why I chose to do that for industrial systems.
And we are looking primarily at three different families of switches. Because I
don't want to pick on any one vendor. In fact, the whole idea of this talk is to
continue giving it. I have two other colleagues who couldn't be here with me
today, who have some vulnerabilities in some other switches. And they look forward
to presenting those vulnerabilities as part of this presentation in the future.
So every time we give this presentation we'd like to give some new vulnerabilities
and show that this is systemic and endemic risk. So the three switches we'll be
looking at today are the Siemens Scalance- family, the GE Multilin-family and the
Garrettcom Magnum family. These switches are usually not very big. They might be 8
ports, they might be 24 ports. And they're used in a variety of different locations.
So this talk is for you, if you work in a utility, if you test industrial Ethernet
switches, if you manage industrial Ethernet networking, if you're comfortable
at a Linux commandline and you play with web apps but you don't know as much about
reverse engineering. Don't worry, I'm exactly the same. I suck at reverse
engineering. But I care about this stuff. And so I'm learning. If you are a
developer of firmware then I think this talk is for you as well. I hope you learn
something from it. If you like vulnerabilities you'll enjoy this quite a
lot. I'm going to be sharing with you a little collection I have, you know. Some
people collect stamps or stories or jokes. I collect private keys. And I like to
share them with other enthusiasts such as yourself. If you happen to work for one of
the switch manufacturers you know I've spoken to before. Some of you I get on
with very well. We speak regularly. Some of you not yet - but I hope you'll come
and have a chat with me later. Ok, most SCADA or ICS presentations go a bit like
this: Pwn PLC, the RTU, the HMI - these are terms, you know, that all of us in
SCADA know. Maybe most of you know them by now, they're pretty popular. I hope you
do. But programmable logic controller, remote terminal unit or human machine
interface. And the basic idea of the presentation is if I pwn these things,
game over. Physical damage. I win. Isn't the world a scary place? And I encourage
you to demand better content. I certainly grew up with better content. I used to go
and see the presentations and the talks of a guy called Jason Larson. And he has a
fantastic example of this. I want all of you to try it, right now. Just think
about: If you had complete control over a paint factory. What would you do to damage
it? No one is going to get hurt. Everything's safe. It's a thought
experiment, right? What would you do to damage it? Most people can't answer this
question. And on certain types of processes I can't answer this question.
But other types I've worked with before and I can answer this question. And I
encourage you to to ask it. But if you like and you want to learn more go and see
Marmusha's talk - I think it's tomorrow. Think of my talk as a frame for her talk.
She's going to be talking about how to damage a chemical process. And what you
need to do as an engineer to do that. And the reason she's doing that is to build a
better process in the future. You have to break a few things to make them work a
little bit better. Okay. So what's the point in industrial control systems
security? It's not credit card data. It's not privacy. No disrespect to my privacy
friends in the room. I have the deepest love and respect for the work that you do.
But confidentially ... confidentiality is the lowest priority for us in industrial
systems. It would go: Availability, integrity, confidentiality. And you might
even swap integrity and availability in many cases. So, you have to protect the
sensor data or the control signals. Everything else is maybe a vulnerability
on the path to getting this. But it's not the most important thing that we're trying
to protect. So that's why I'm attacking switches. That's where the process is,
right? Now these may not be core switches. They're often a little bit further down in
the chain. They're field devices, right. So you might find them in any of these
locations. And this last example is not necessarily important be cause oil and gas
is important - but it's important because it gives you the general format of all
industrial systems. You have sensor network. And sensor data is traveling back
and forth. And you have control signal data. That's it, basically. You might have
different control signals on different protocols and you might have different
sensors on different protocols, giving you different values like pressure or heat or
whatever. But most processes follow basically this format. Okay. I don't do
SCADA 101. There are other people who do this. I'm trying to do a little bit, to
set the reference for this talk, but usually I avoid it. So basically there's
not much authentication or integrity in industrial systems protocols. There's not
much cryptography. You would expect there to be, maybe. I'm continually surprised
that I don't find any. And when I do find it, it's badly implemented and barely
works. So once you have compromised a switch or another part of the network you
can perform man-in-the-middle attacks on the process. Or you can create malicious
firmwares on these different switches. And that's what I'm trying to prevent. I'm
trying to find some of the different methods that people can use to produce
these firmwares - and then get the vendors to fix them, right. Okay. These are some
of the protocols. If you are new to this space, if you want to do some more work in
this area, but you don't know what to work on, take a picture of the slide or go and
find it later. And choose one of these protocols and go and work on it. We need
people to go to these different organizations. Some of them are
proprietary, some of them are open and complain that there is not enough
cryptography going on in this space. And yes you can use VPNs. But believe me, I
often don't find them. Okay. These are the switches, the specific versions of the
firmware, in case you're here for vulnerabilities instead of just me
waffling on about the basics. If you want to go and look these up, if you're a
penetration tester working in this space, you can go and find them all online. And
you can get a feeling for the kind of coding practices that go into these
different devices. Now I've tried to choose the vulnerabilities that I'm
presenting very carefully. To take you gently from web app vulnerabilities into a
little bit deeper into the firmware. So the first one we'll be looking at is
Siemens. And again, I'm not picking on any particular vendor. In fact I'm very proud
of Siemens. They're probably here again. They're here many years. And they fixed
these vulnerabilities within three months. And I think that was awesome - especially
in the space that I work in. The average patch-time in SCADA and ICS is 18 months.
So I think Siemens deserves a round of applause for getting these fixed.
Applaus So without further ado let's have some
fun, right. So MD5, you go to the web page for this switch. This is the management
page of a switch, right. And you interact with this webpage. And you have a look at
it. And on the client side they do MD5 of the password. Okay. That's fascinating. I
don't think that's particularly secure. But it's done in roughly the same format
as that Linux command. So I use the Linux command instead of the JavaScript just to
make it easier for everyone. You have the username at the beginning and the password
is in the middle. And then you have this nonce that's at the end, a number you use
once, right. I was surprised to see the nonce, and it's even called a nonce,
right. So somebody had done a little bit of homework on their cryptography. And
they understood that they wanted to use, you know, this number used once to prevent
replay of the hash every time. Okay, that's some pretty good work.
Unfortunately this is MD5 and this is protecting your electric utilities and
your water and your sewage systems. And you can brute force this in a few seconds,
if the passwords are less than eight characters. and if they're around 15 it
might take you 20 minutes or something. You can do this from PCAPs, from network
traffic captures. And then you have the cleartext password that you can use
forever after, with that switch. So, off to a bad start, in my opinion. So these
are the nonces that we're looking at. I'm glad to hear you laughing. It makes me, it
warms the heart, right. So you can see that they are incrementing and that they
are hex. Yeah. What else can you say about this? The last half is different than the
first half. Not only is it incrementing, it is sequential. If you pull them quickly
enough. For those of you who also do a bit of reverse engineering you might recognize
the first half as well. Anybody in the room see any patterns in the first half of
the of the nonces? No? Hmm? Very good, IP address. Mac address would have been a
good guess as well. I thought it was at first. And I got very confused when I went
to look for the IP address. Because I went to the switch itself. And the switches IP
address was not this in hex. It's the clientside address. Which I just couldn't
believe, right? Like, it seems like it makes a sort of sense if you're trying to
keep session IDs in state. And it's like oh I want a different session for every IP
address. And then I'll just use time, I use uptime in hex as the rest of my
session ID, right? You know, the entire IP space and time that can't be brute force.
It has a kind of crazy logic to it, right. Unfortunately it can be. And you can get
the uptime from the device using SNMP. And of course if you don't want to use SNMP
you can get old-school and use the TCP- sequence-ID numbers. So, not a lot of
entropy there, I guess, I would say. And I think their lawyers agreed when they put
out the comments on this. All right. Not only can you perform session hijacking.
And if you are attacking switches I'd like to point out that session hijacking is not
necessarily a great attack in this environment. Think about it like you would
at home, right. How often do you log into your router? In fact even more importantly
how often do you upgrade the firmware on your router? Everyone who has upgraded the
firmware on their router ever raise your hand. Just for an experiment. Thank
goodness, right. But wait, keep them up just for a minute. Everybody who's updated
it this year, keep your hand up. Everybody else put them down. Everybody who has
updated in the last six months ... okay ... So that gives you a sense of how long
these vulnerabilities can be in play on an industrial system's environment. If you
multiply that by about 10, right. Okay, so you can simply upload a firmware image to
a Siemens Scalance device with this version number without authentication. You
just need to know the URL. Cross-site request forgery, right. I just say CSRF
all the time. I don't even remember what it stands for. So you can upload or you
can download a logfile. Not that useful but you get a sense of what's going on on
the switch. You know what usernames might be present, whatever. Incidentally all of
these switches by default or at least this one only have two usernames, right. So
it's "admin" and "operator" I think on this switch. Or maybe it's not. But
anyway, there's two usernames "admin" and "manager"? I know I get them mixed up now.
But the configuration includes password hashes. I'm actually not even entirely
convinced they're hashes because when you increase the length of your password it
increases. But I'll leave that for future researchers to examine. You can download
the firmware image from the device, which is nice. So you just make a request. You
just post an HTTP-request to this device. And it gives you the firmware that it is
running back. That's not that big a deal, right. Because you're just viewing data on
the switch. But you can upload firmware and configuration to this device. Which is
an authentication bypass in and of itself. But it's also interesting because I can
take a configuration file from one of the devices that I have at home with a known
password. I can upload a new configuration file with a password that I know. I can
use the device to do whatever I want to do. And later I can re upload the old
configuration file that I got from the device, so no one ever even realizes what's
been changed, right. So. I think that's a disappointing state of affairs. And I
wrote a script to do this. So that you wouldn't have to when you are doing
penetration tests of these device. And I gave you a little ASCII menu because
sometimes I get bored. Cambridge is a small town and there's not much to do in
the evening. So feel free to go and examine my github-repository where I put
up some of this stuff. I'm Blackswanburst on Github, and on Twitter. So like I say,
Siemens are some of my favorite people. So I'm going to finish up with them. This is
old day, if you like all that you have just seen. But I want you to keep in mind
that these vulnerabilities will still be present in the wild for another two or
three years. And I encourage you to go and have a look at your systems, if you have
any of these devices. And check them out. And upgrade the firmware. I also hope this
encourages you that if you haven't done much in industrial systems and SCADA you
don't have to be intimidated by all of the engineering and the terminology, and the
verb beotch(?).. There is plenty for any of you in this room to do in the
industrial systems space. You need to spend a little time speaking to engineers
and translating your vulnerabilities into something meaningful for them. But that's
just a matter of spending more time with them and getting to know them. And I think
that's valuable too because they have a lot of experience. They care very deeply
about safety. And I've learned quite a lot of things from engineers. My general point
here is I'd like you to stop defending banks and websites and other stuff. We
need your help in industrial systems, in the utilities. We could really do with
living in a safer world rather than one where you're just protecting other
people's money. So we're gonna move on to the GE Multilin line. I worked on a GE
ML800 but these vulnerabilities affect seven of the nine switches in this family.
Seven because one of the other switches is an unmanaged switch. If you're a hardware
person maybe you want to go and play around with those but not so much my thing
and the other one uses a different firmware image but seven of the nine
switches use a similar firmware image GE offers a worldwide 10 year warranty. So
let's see if that includes fixing vulnerabilities. I think it should. What
do you think. No? Couple noes couple of yeses, undecided. All right. CCC is
undecided on something that's novel. Let's start with some new vulnerabilities. Cross
site scripting. Reflected, I grant you but still cross site scripting and I want you
to pay attention to the details. I'm not going to go slow for you and ask you to
think . I know it's morning, I know it's tough but I am going to ask you to think.
See flash up there flash.php and the third one. Yes, it runs flash in your browser.
So if you know something about Flash come and have a look at the switch some time. I
didn't go for active script attacks. There are so many attacks surface on this device. I
just I sometimes don't even know how I'm going to finish looking at all of them. So
I just work with the web interface to begin with. So you have this cross site
scripting times eight and I want you to notice in the last section there
arbitrarily supplied URL parameters. I don't know about you but I think that's
funny right. You can just make up parameters to stick your cross site
scripting in. laughs It's unbelievable right. Yeah. Anyways what does that look
like. It looks like that, they have an error data page. OK maybe I'm using a
browser that they don't approve or something but it deserves looking at. And
you can do quite a lot of things with javascript on the client side these days.
Disturbing. Anyways I'm not a big fan of XSS so I'm going to move on to things that
I think are worth my time. So if you fetch the initial web page of this switch before
you've even logged in you get this config. So this is pretty authentication. No
authentication, right. Now keep in mind that these switches are designed for process
data, right. It's not carrying traffic to images of cats. It's supposed to be for
engineering. So what happens if I add a nocache parameter and I make it say 500000
digits long. I should just be able to crash the web server. Right. Maybe maybe.
But you would not expect it to reboot the switch. And it takes a minute or so for
the switch to reboot which is actually really impressive comes up pretty quickly.
But you know obviously you can repeat this. So I wanted to examine that a lot
further. I wanted to know more about that that crash what was rebooting the switch.
But like I say I'm not a very good reverse engineer. So you're going to go on a
little journey with me where I learned a couple of things about reverse engineering
and I had to change my approach from looking at the webapp style loans to
moving into this other stuff. So why is why is it DoS even interesting. You'll
remember that I mentioned Misha's talk. So the reason I mention her talk, this is it
right. Denial of Service on a Website. Who cares it's tearing posters down as xkcd
once famously explained to us but in the industrial system's environment it's very
different. It can be very serious right. A simplistic example is you have an
application that has a heartbeat and if you stop that heartbeat it might go into
some sort of safety state it might for example scram a reactor. There is a famous
denial of service on PLCs that did scram a reactor in real life. Does anybody know
what H2S is? Any oil and gas engineers in the room? Okay so H2S alerts not reaching
their destinations is pretty serious business right. For those of you who are
not aware of H2S it's a byproduct of producing oil and gas and inhaled in very
very small amounts you can go unconscious and in sort of larger amounts. Respiratory
failure. So if you take CA safety seriously if you ever work on these rigs
in these environments you learn to care about the wind sock. Right one of these
alerts goes out. An alarm goes off. There are many different alarms you have to
memorize how they all sound on a rig and then react to them and when you hear the
H2S alert you look up at the wind sock to keep an eye on where the wind is and
trying to avoid being downwind of wherever the leak is. So a simple denial of service
that we would not care about in a web application environment in this
environment can be very serious. I'm not saying it always is. It just can be
right. So denial of service goes up in our list of problems especially when we're
looking at networking devices. Okay so that's that's it for the denial of
service. But like I say we're going to look at some other stuff. In fact the
story with the switch began with a concerned citizen about three or four
years ago I found 10000 industrial systems on the Internet as part of my master's
thesis and I was pretty uncomfortable with that. So I sent that data to various
computer emergency response teams around the world. I believe it was 52 of them
right. Not all of them were critical infrastructure. A lot of them were small
stuff but maybe 1 in 100. I was told or in one particular country when they got back
to me one in 20 were considered critical infrastructure. And after that you have a
sort of reputation among the computer emergency response teams of the world. So
people send you stuff you get anonymous e-mails from someone called Concerned
Citizen. Thank you very much. They sent me a firmware upgrade pcap of this particular
device. I suspect that they worked at one of the utilities and they wanted me to see
how upgrading the firmware of this GE switch was performed. So it all began with a pcap.
So I ran TCP trace to carve out all the files and see what was going on and you
could see instantly that there was an FTP session later looking at the switch I see
that you can also upgrade them over TFTP so the management of the switch happens in
HTTPs and is encrypted but the firmware upload goes across FTP right so you can
just carve the file out a little bit of network forensics I guess. So instantly I
could see that this one is complete and the ports on the end of the numbers give
me a clue of what's going on in the larger stream. This one seems interesting. Let's
have a look at it. So. I tried running file and binwalk I don't know about you
but I believe that hacking is a journey of understanding and facts hacking is
understanding a system better than it understands itself and nudging it to do
what you want right. And I also feel that I should understand my tools. I don't
really understand my tools until I know where they're going to fail me or they
have failed me in the past and in this particular case I think binwalk is a
fantastic tool and file is a fantastic tool. But they didn't tell me anything and
that was that was a journey of discovery for me. So that was nice. It was like OK
binwalk doesn't always give me everything. I think I was running an older version and
I think it would handle it now. But the point is after been walked didn't give me
anything just resort to the old school stuff right. Go strings and I found these
deflate and inflate copywrite strings and I could tell that a certain portion of the
file was compressed. This is just from the pcap. Remember this whole story. So I
tried to deflate the whole thing. That didn't work again. I just did something
simple get a python script that checks every byte to see which parts of the file
don't produce ZLIB errors when you try and decompress them and you figure out what
sectors of this file are compressed. So you go to your friend dd and you carve out
this section of the file right. So we have this larger firmware image with this
little compressed section and we have now cut this little compressed section out. I
suppose I could have loaded this up into python and use ZLIB to decompress it. But
at the time I was still trying to use command line tools and someone said I'll
just concatenate the gzip bytes on it. Gzip inherits from inflate and deflate. So
if you just concatenate the bytes it should still handle it. So I did that and
I got a decompressed binary. When you ran strings on that it started to make a lot
more sense and you could find the opcodes in it where previously it didn't make any
sense at all. So once you've got an image like that what do you do. Well if you're
me you just grep for bugs. I think I learned that from Ilija. If he's here in
the room thank you. Thank you very much. I asked him like a year or two ago. How do
you how do you find so many bugs. And he said: "Oh, I just, you know, I grep for
them, I use find." laughs And so I started thinking about firmware images.
Like if I was going to grep for a bug in a firmware image what would it be. And my
answer is hardcoded credentials and default keys because you find them every
single time so I have this command aliased on my machine and I just grep for it and I
find private keys and this is how you too can end up with a private key collection.
So, there you go.
Applause
Yeah they're hardcoded keys, but what are they for. It doesn't
stop there. You know you've got the keys, but what do they do, right? That was the
next step of the journey for me. Two of them you can see one sencrypted with a
password; we'll come back to that one later. Let's start with the one on the
left. If you load this key up into wireshark. and you use it to decrypt the
SSL you have a self decrypting pcap. Remember at the beginning it was using
HTTPS to manage the device and upload this firmware image. So if you happen to have
this firmware image you can decrypt all the traffic. No forward secrecy, right?
Now you don't have to be lucky and have concerned citizens send you an email. You
can download this image from the GE website and you can carve the keys out of the
image in the same way that I did and decrypt the SSL traffic of any pcap that
is sent to you. Now the passwords underneath that are in clear text. You can
see them highlighted down here. Password Manager and user manager. You can see them
up there as well and you can see that we've decrypted the SSL with that key. So
default keys, right? Is it a big deal? I believe the vendors in this case say you
can upload your own key to the device. For those of you who aren't used to working in
embedded it sometimes is difficult to generate a key on the device because you
don't have enough memory or you don't have enough entropy or you don't have enough
processing power. That's the usual excuses. And they're true I shouldn't say
excuses those those things are true. But you could of course generate it on the
client side and upload it to the device and that's what they allow you to do with
this switch which is great but where is your encrypted channel in which to upload
this key? laughs So you can use the serial device and make sure visually that there's no man
in the middle. But if you're doing this remotely – and I'd like you to keep in
mind that most substations are remote ­– if anyone here works in a utility are you
going to drive to every substation, plug in a serial cable to change the keys on
all these devices? It's the sort of thing you need to know in advance right? So the
problem with key management, particularly with SSL and the industrial systems
environment, is that you have to manage the keys. And these particular keys, well
the certificates are self signed so you can't revoke them. And besides industrial
systems are never connected to the Internet. So it wouldn't have made any
difference. So these are the kind of problems we're dealing with in this space.
And that's why I'm trying to encourage you. Whether you do crypto or privacy or
whatever spend a little time in the embedded space, just for bit: there's
plenty of easy work. OK. So what about the second key. It requires a password. I
didn't feel like brute forcing it. Maybe you do. I don't know. I tried all the
strings in the image. A classic technique, just in case someone had a hard coded the
password. I mean the hard coded credentials were there but not the hard
coded password. So I guess I gotta start reversing, and as I previously said I suck
at reversing. That's why I come to CCC, so I can learn, right? But I did find this
PowerPC ROM image. and I think its running eCos and redboot and I haven't even gotten
down to doing hardware stuff: taking it apart, having look at, it but I probably
will in the future. So there's the image I'm slowly starting to learn my way around
and figure out what's going on. So I had a look at the image and I figured out that
this key is used for SSH, right? Well it would be the other encrypted thing. But I
couldn't enable SSH on the device. I try and enable SSH on the device and I'm
logged in as manager by the way. which is highest level user on this particular
device, and I put it in the passwords that I know and a bunch of other passwords and
they don't work. Like I said, I tried all the strings in the image. So apparently to
enable ssh, I need a password for something. Now maybe I'm just
misunderstanding or I'm not so clear on what's going on but I don't know about
you. I kind of feel like if I buy a device that's supposed to be used for a safety
critical process I should be allowed to use SSH without having to call up the
vendor and get some special magic password. So considering I don't like that
approach. What if I patched my own key into the image right. I don't know the
password of their key but I know the password of a key I can generate. So I
just need to make sure it's roughly the right size and try and patch it in. Then
I've got some problems with compression because I've got to reverse the whole
process that I just described to you patch it into the larger binary. Will there be
any CRC or firmware signing? I don't know, right. So the uploaded image is not a
valid image for this device. That's correct: I messed with it. But I got this
error and it gave me a clue. It gave me a clue that I did indeed have some of my
CRCs wrong so when I altered the image again I got to this state. So you're
learning all the time by having a real device. Now some of my friends they do
static analysis and they don't buy these devices. I decided to buy this one. I
found one on eBay. It wasn't very expensive. I mean it depends on your range
for expensive. But if you're helping defend industrial systems I thought it was
worth the money. So I bought it and this enables me to try firmware images out and
I can slowly start to figure out what I need to patch on these firmware images to
do whatever I want. Luckily I just tried to patch mine to have SSH because I
thought people deserve to have SSH. So that's an Adler 32 up there on the left
and the other CRC is on the bottom so that Adler 32 and some adjustment of file
length although zeros in that line just above it eventually got me to the point
where it believes it's a corrupted binary. And then we have this CRC on the end that
we need to have a look at. Now I'm a big fan of suspense. I love suspense. I'm
going to leave that one is a cliffhanger and an exercise for you watching. So I
said I was going to talk about GE ML800 but I'm also going to talk about
Garrettcom. Luckily it's not very difficult. Garrettcom is the original
equipment manufacturer for the GE ML800 series. I noticed that because the
certificate I found attached to those private keys said Garrettcom in it and I
went and looked at their firmware images and they have similar CRC similar file
structures similar everything so I believe that they are affected by the cross site
scripting, the denial of service, and hardcoded keys. I understand from some
people that they have been in contact with GE to try and fix some of this stuff but
their response to GE was mainly "Sorry, this is the end of life on this device".
That's fine. I understand you're running a business but you're selling equipment to
people who manage utilities that we all depend on. If Sony goes bankrupt because
they get hacked that's one thing right. But you can't just dissolve a utility and
start again. As my friend Klaus points out regularly – fantastic insights into the
industrial system world, Klaus and Vanessa – you can't just dissolve the utility and
start again. You still have the same infrastructure you still have the same
workers. It doesn't work that way. You can't bail out utilities that we depend
on. So sorry. End of Life... I don't even understand why people buy these devices
and this code without code escrow. When you buy the code make sure you have the
code in perpetuity for these systems so that you can fix them when something like
this or something worse happens. If I'm your worst nightmare, you have real
problems because there are very dark people in the world actually damaging
furnaces in Germany. So me disclosing keys on stage is scary for you. You need to get
a grip. So, garrettcom? Here's your key too.
Applause
The strings come from the images. Developers are funny people really. I like
this. I just put them up because they're funny. Some people had some hard times, I
guess, writing some of this code. And my respect to them! They do great work but
you know, there's a couple of things we can improve on security in these devices.
So I once had the opportunity to stand in front of six different vendors at the same
time their computer emergency response teams at a conference and I said to them,
"Will any of you commit to an average patch time for vulnerabilities of three
months?" An average patch time, because it might take 8 months, as it so far has
taken in the case of GE and Garrettcom, to work on these issues. It might take a long
time in some cases but as an average patch time I think 3 months for things that we
all depend on is reasonable. So I asked these six different teams in the same
room. If any of them would commit to this and I heard silence for 30 seconds. So my
friend decided to call this the silence of the vendors right. And I think that's that
sums it up. I'd like to see better patch times. I'd like to see a computer
emergency response teams in each of these vendors and I'd like to see someone
responsible for security in each of these different utilities. I can dream, right? I
think that key management... the current practice industrial systems is to take
some insecure protocol and wrap it in SSL or TLS which is why we need the help of
you privacy people because TLS and SSL are not the be all and end all. They often
sort of go the wrong way, right. For example you can use TLS to do integrity
without encryption so you can verify that every message has reached its destination
intact but it is not encrypted. And this means that you can still do intrusion
detection analysis of the packets. That's really good. But nobody uses that in SSL
in other ways right. I'm a big fan of Shodan and use Shodan for a variety of
different things usually to get a sense of the Internet as a whole, right? Let me
back up a little bit. When I was at Cambridge I went to Darwin college and
because you're at Darwin college you read up a bit on Darwin and you think about how
Darwin thought and I think the Internet is kind of like that. When it was built by
the IETF and various people, who did fantastic work, they imagined it one way
and then we inherited it and it grew and it became an ecosystem and stuff happens
out there that you wouldn't expect. And so that's why I like Shodan. It's kind of
like being a natural scientist: what's a survey of the world, what kind of machines
are out there, what versions are they running, when do people update their SSL..
err, you know, their certificates do they do it before or after the certificate is
invalid. Do they always upgrade the algorithm. Do they increase the key size.
You know how do things change right you need to sort of study it as a whole and
that's my point when it comes to just taking SSL and slapping it over a
protocol. It's not quite that simple. So again we need your help. Where can we go
with these attacks. And you remember at the beginning I pointed out the underpants
gnome. The emperor wears no clothes. Altering switch configurations is a big
deal because you can exfiltrate process data. That gives you a map of the process
because industrial systems are bespoke. Each one of them is different. It does run
different traffic and we are lucky to work on security in this space because our
users are numerate and literate and they care about safety. They don't always
understand security but they do care about safety. So if you can make it a safety
concern they care. There are also engineers that many of these utilities who
look at the network 24/7. Not all of them but some of them. Can you imagine a home
network or something else with that kind of user base. We're lucky we should be
taking advantage of that user base. So getting back to the point you know denial
of service attacks to disrupt the process go and see Marmusha's talk. This will all
make a lot more sense when you go and see her talk. Basically any man in the middle
attack can disrupt alter or drop traffic at this point. If you can affect the
switches and the substation. And exfiltrating in the data gives you a map
of the process which leads towards further potential damage for the utilities. Now
it's not always that simple people will get up on stage and they will tell you I
am awesome and this is how it's done and it's easy to blow shit up. It's not true.
It takes a little bit of thought it takes a little bit of work. I am certainly not
awesome. I am just a quality assurance person from a former vendor. I just
decided to get into security and keep going with it. So you can't always perform
these man in the middle attacks. People will say you can. But the reason you can't
is real-time system constraints. Some systems will stop receiving traffic five
milliseconds or microseconds later and ignore anything. If a value doesn't arrive
in this time it doesn't care. So the idea that you can route the traffic out to some
other country and then back in and disrupt the process is bollocks. Sometimes you
have to alter the firmware to achieve that. That depends on the process but I'm
just trying to give you a sense of how performing actual attacks give you a sense
of what the limits are, what the logistical burdens are for the attacker
and that's important stuff for us to know. All right. Little bit of an overview.
Drunk session IDs. brute forcing MD5+NONCE, cross site request forgery for
firmware upload (of all things), reflected cross-site scripting (8 cases of
it) pre authentication denial of service, hardcoded keys times 2 in a firmware
image, SSL without forward secrecy, self signed certificates so there's no revoking
there's no managing of the keys on these devices right. Not to mention utility
workers are busy already. They may not have time to manage all of these devices
we might need to rethink that approach right. Clear text passwords under SSL
because well no one can break SSL unless you hard code the key in the firmware
that's downloadable from the internet. Enable ssh with a password and three
quarter of a year waiting for fixes for some of this stuff. I'm not happy with
that. I think that we could live in a much better, much safer world. And to do so we
need to talk very seriously about some of these issues. Don't take my opinion for
it. Listen to some other people. The best thing about doing industrial systems work
is the diversity of approach. You know I love that there are so many other people
doing SCADA and ICS. And I love that they're going different directions. So in
the future I plan to be on another stage with some friends and show you some more.
Thank you for listening mustache fans and as a parting thought. More tax money is
spent on surveillance than on defending common utilities.
Applaus
Herald: Thank you. It made me a scary Sunday morning. They got a utility *<<
guess, mostly incomprehensable* down the road. OK. We'll have some questions taken
please. As the session is recorded and streamed anything you say, say it into a
mic. Any questions up? Wow, it is Sunday morning.
Éireann: Number three, sure
Herald: everybody understood everything? You're kidding me.
Éireann: I've got one right here Herald: here is a question.
Question: Hey thanks I enjoyed your talk and I think it's very important to raise
awareness. But I think it's not to raise awareness. Not much in this community, but
within the engineering community and I see it a lot of times and many engineers
having lots of problems doing that for several reasons. There is maybe the
engineer who is thinking about this but has its miniatures in the back has to deal
with service personnel which know how to work a hammer and a screwdriver and on the
other side, engineers have to work with customers which more those lazy people.
And so that's how these things happen. And I think it's more important to raise
awareness of these kinds of things in the engineering community.
Éireann: So just to repeat a little bit for anybody else that couldn't hear it or
for the recording it's very important to work with the engineers some of the
engineers understand the problem. But typically management or lower level
service personnel don't always understand the problem. And it's not important to
raise the awareness in the hacker community. But more with the engineers is
what you were saying. Right. OK. Absolutely true. Completely agree with
you. I don't just come to these conferences and present to you guys. I go
and I present to the engineers too. And in fact a couple of engineers have come to
this conference because we did work at other conferences to see what the hacker
community is about and learn things from the hacker community because this is a
place where you can learn if you're just not afraid of getting pwned a couple of
times right. And it happens to me too right. I learned a lot from getting
compromised on my machine and watching someone do something. Anyways back to the
point I don't just work with engineers or hackers. I also work with C-level
executives so I'm on a sabbatical from IOActive at the moment. at the Cambridge
Center for Risk studies, and I'm working with the insurance people which has its
challenges shall we say. But some of them are very intelligent people and they want
to understand what's going on with hacking attacks and they want to approach this
from a slightly different angle. My stake in that is to be sure that when the
insurance people do get involved that they actually ask for fixes and improve stuff.
So yes I do my best to raise awareness wherever I can. And I'm not alone. You can
help me. Questioner: Thank you
applause
Herald: OK, there's another question here. Number two. Oh, and up there too, yes we
saw you. OK number two was first I think. Go ahead
Question: incomprehensible. So you mentioned a couple of things, err a couple
of vulnerabilities and I was wondering what you would think an ideal system would
look like. You mentioned key provisioning of course putting certificates. I assume
that they were different certificates for different devices rather than the same
certificate for all devices. Okay that's a bad thing. And and also sort of the way how
the software update management works. So how would you if you could give them some
advice how to design a system how would you do it?
Éireann: Okay. So first of all I wouldn't hard code the keys as you as you discussed
to be in every device the same. It's one thing to put in your documentation hey you
should update the keys but I mean if I can patch binary file with a key then there's
no reason you couldn't do that on the website where you download the firmware
image right. Just as an example as a thought experiment sort of makes that
clear. The upgrade path for these devices is download the firmware image from the
website to some machine and then carry it, because all these systems are airgapped.
to some other location and then upload it onto the switch right with hardcoded
credentials. So first off whenever you provision a switch initially you provision
all of the credentials for that device. That's standard practice of many routers
and other pieces of equipment today. And I would think less about defending and
securing the device than on being able to regularly check its integrity,
the integrity of the firmware that is running and the integrity of the
configuration. So I'd focus on that and I'd focus on being able to recover the switch
after it's been attacked. So you reverse your thinking. You assume that one day
someone is going to crack your firmware signing and crack this and crack that and
you focus on how can I quickly upload a new firmware image that is known to be
good and verify that the one that is uploaded is good to this device.
Questioner: Thank you. Herald: There was a question up there on
the balcony. Signal angel: Yes we have two questions
here on the net. So the first one is how would you solve the end of life issue.
Sometimes incomprehensible clients just gets really outdated.
Éireann: That's absolutely true and it is slightly unfair of me to be a hard on the
vendors. But it's my job to take the debate a little bit too far the other way.
So how would I solve the end of life issue is the question from the internet. I don't
know. I think that's not a technical problem it's a societal problem. Like when
we buy bridges they are bridges until they fall down. When we buy roads they stay
there until they go away. I mean there is probably some end of life issues in there
but it's almost more of a contractual legal issue and someone should study that.
There are people studying that but it's not my area of expertise but I'll try and
answer as best I can. I think code escrow is a good way to go when you buy some of
these devices you say I want the code for this device in the future. I want to have
access to it. If your company goes bankrupt I need you to give up the source
code for these devices when you go bankrupt or when you disappear or when
it's the end of life. There are a couple of manufacturers out there doing open
source switches. There's a company called Open gear who are awesome. They gave me a
switch to play with that I haven't had time to look at yet. I think that's amazing
right. And their code is open source and you can go and examine it. So you would
have the code anyway. Those are two different approaches. I think there are
others you can solve this problem technically or legally or socially but as
a society we depend on these utilities and that code should not just vanish when it's
difficult or costly to keep it upgraded. applause
Herald: There was a second question from the Internet.
Signal angel: Yes, so the second one is: what should a non-technical person in
the respect of incomprehensible set non- technical person sent to manage small town
utility do as best practice? Éireann: I think the first and most
important thing is to look for attacks. I'm sorry I should probably repeat that
question just to be sure. What should someone in a small town who manages
utility do to defend themselves and protect himself. So the first thing is
look for attacks. Even if you spend a few hours a week looking for something you
script something up or you hire some college kid to come in and script
something and look for things on your network and ask questions and yes they're
going to be a pain in the ass and is going to be difficult. But you're going to learn
things about your network and you might detect some attacks. The first problem in
utilities is no one is responsible for security. It's not my job. It's kind of
the mantra so for a small utility find someone whose job it is if you're a very
small utility there's probably some other small utilities near you and you can hire
a resource together to come and visit your different utilities and help you out. The
second one is watch your relationship with your vendor when you purchase this
equipment you spend a lot of money on it. Spend a little bit of time doing
penetration tests. Yes I like it when you hire me but you don't have to hire me.
There are plenty of other people you can hire who will have a look at the device
and find the simple vulnerabilities. So when you purchase something make sure you
test it for security purposes and that's very important because you can even put
into your contract if you fail the security tests we will pay you less money.
And the vendors are not going to react to security until you do that. So that's
the second answer. And I wish I had a third to make it very neat but I don't.
Herald: OK. There was one more question at mic 4 I think
Questioner: Yes hi thank you for your time.
Herald: Talk into the mike please. Thank you for your talk. Q Hi. I'm kind of a
newbie to the C3 community and I am not sure about the question I want to ask you.
Probably many people understand in this room but I don't know if I would like to
ask you what exactly do you mean by arbitrary firmware.
Éireann: No problem. So the question was What do you mean by arbitrary firmware. I
mean the firmware that I have altered that was not manufactured by the vendor to do
whatever I want. How do you trust that this switch sends all the packets that it
should send. What if it's, you know, my handle is BSB right. What if it drops
every packet that has BSB in the packet. Right. You can rewrite a firmware image to
do whatever the device can do and in some cases more things than the device usually
does to damage itself for example. So an arbitrary firmware is one in which anyone
writes the firmware and there is no checking to be sure that this is the image
that you want on this device whether it's provided by the vendor or the community
right. You still want checking that this is the correct code or the code that you
wanted anyway. Right. Herald: Okay thank you. Is that a question
here mic 1? OK go ahead. Questioner: Yes please. In your
hypothetical question, you asked what damage could I do in that paint factory.
But you can also reverse it. What kind of company secrets can I obtain for example,
your favorite recipe for your hot chocolate or the recipes of Coca-Cola.
They are vulnerable as well aren't they. Éireann: Yes. So the question just again
for everyone else. You don't just have to talk about damage in a paint factory or
any industrial system. You can also talk about intellectual property and protecting
the recipes that we use to bake cookies or make beer or whatever pharmaceuticals
whatever. And that's a fantastic question and I'm glad you brought it up a couple of
years ago when I was doing... well, more than a couple of years like eight years
ago, when I was doing industrial system security I realized I wasn't getting a lot
of traction. It was before stuxnet, I was a quality assurance guy. Everybody thought
I was fucking crazy right. Stuxnet, career. It's wrong. It's really wrong. But
the point is I tried to take that approach. I tried to say you have a
process in which you manufacture something and you make money by the fact that that
process is relatively secret and if you don't care about defending your workers
from being damaged then at least care about the intellectual property because
I'll get security in by some sort of back door right. I'm a little bit of a security
Machiavellian. I'll find a way to get security into the system somehow. So I
tried to say intellectual property you should be protected. And I found that they
didn't care so much. I mean maybe you'll have more luck maybe post-stuxnet that
that's a better argument. I hope you do. But it is an important question as well.
Right. It's not, it's not just potential for damage. I think there's a lot more
espionage going on on these networks than there is damage and sabotage. Herald: Okay
we'll take one more question on mike four. Questioner: Thank you okay. My question
concerns the concepts of software defined networking and open flow. So when I first
heard about software defined networking I thought well this is a huge security issue
and there may be huge vulnerabilities. After your joke I think this might
actually be a good idea to dumb down the switches and put the intelligence
somewhere locked up in a safe place. What's your opinion on that. Can they
actually improve security. Éireann: Yes. So the question is what role
could software defined networking play in these sorts of environments. And is it a
good idea from a security perspective. Anytime someone has a revolution in
computing we also have to update our security paradigm. So I think with
software defined networking it's not whether it's good or bad it's that you
defend that network differently than you defend one of these networks. So it's not
so much that as good as good or bad it's neutral if you know how to defend your
network. I don't care what it is. As long as someone is looking to defend it and
cares about how the flows are working. So I think software defined networking in
these environments could be a very good thing but the refresh rate on these
devices is not that high. So I don't think we'll see it there for a little while even
though it might be a good thing philosophically. It takes 5 10 15 20 years
to refresh these networks so it'll be a little while. But it's not good or bad. It's just
learn to defend what you got is the problem right.
Questioner: Okay thanks a lot. Herald: Okay okay let's give a big hand
for Éireann and thank you. Éireann: Thank you
applause