In episode 7 of Swimm Upstream, Host Tom Ahi Dror speaks with Assaf Cohen about his experiences as a network engineer and a training manager and finding the right balance in creating guardrails, protocols, and best practices for security breaches. Assaf is Training Delivery Manager at Amazon Web Services.

[00:08] Tom

Hello, everyone, and thanks for joining us on this episode of Swimm Upstream, a podcast devoted to engineering team leads and dev tool enthusiasts. I'm Tom, co-founder of Swimm, and today we're talking with Assaf Cohen. Assaf is a Senior Technical Instructor at Amazon Web Service. With years of experience as a network engineer and a training manager. We have about 15 minutes and lots of questions. Let's see what we can cover. Hi Assaf, we're excited to have you on thanks for coming to speak with me today.

[00:36] Assaf

Thank you for having me.

[00:38] Tom

So, Assaf, at Swimm, we take our coffee very seriously. Probably too much. So to start things off, I would love to hear how you take your coffee in the morning.

[00:51] Assaf

So I think I'm already on the losing team here because I don't really drink coffee. I don't really drink anything hot, for that matter. But I do realize that puts me at a massive social disadvantage because coffee is like the global welcoming, welcoming thing. And so what I normally do is I just grab the nearest cup of coffee I can find when I'm talking to customers or colleagues, and I just hold on to it until I can find my next bottle of water. So I apologize to all the coffee enthusiasts in the crowd. But we exist as well.

[01:24] Tom

We accept that apology - that's fine! And if you're looking for cold drinks, you can think about cold coffee, iced coffee, and so on. And that's all about coffee for this session. So I'd also like to add this past year and a half has been mostly focused on adjusting to the realities of COVID-19.

[01:47] Assaf


[01:48] Tom

And I'd love to hear what's one thing you learned to live without that surprised you since this pandemic began?

[01:55] Assaf

Well, I've been in the training industry for pretty much the last decade, and I've been mostly training overseas. So if you throw a dart at the map, and I have probably either been there or somewhere close. So I found myself on an airplane, probably anywhere between once or twice a month. And ever since this pandemic came in, I haven't been flying. So the last time it was actually overseas was during an AWS convention in January in Chicago, which is a story by its own right. But - so that's something that I, you know, I found myself that even though I've done this so many times, I don't really miss it. I'm good with staying here for now.

[02:39] Tom

Yeah, I think you're not alone in that. It's yet to be seen how many of those opportunities to fly will return and how many of those won't.

[02:51] Assaf


[02:52] Tom

Okay, so, Assaf, please, please describe what you do, and try doing that without using your title.

[02:58] Assaf

Whoo. Alright, so let's try and put it this way: I educate and teach people about use cases and best practices as they consume services that run on someone else's computer. How about that?

[03:17] Tom

Yeah. That sounds good. Okay, and what are the main things that you train them on, like focus areas for training?

[03:30] Assaf

So AWS has a training and certification division that covers a wide variety of subjects - anywhere from architecting through developing DevOps, security, containers, security, financial operations. So we cover a lot of ground, and every trainer, specialized every trainer has to have a solid base on most of these topics. But at the end of the day, most trainers pick their own specialty and train at those areas. So I cover quite a large area of those subjects. So I'm very diverse in that sense.

[04:11] Tom

So is it challenging? Apart from discussing or teaching skills and knowledge? Do you also have to educate developers on how to use the different tools and being aware of the tools that you focus on?

[04:32] Assaf

I would probably say most of my customers are developers. And I see that - I see that as a mission, right? Because developers today - I mean, just in general, everything is so fast tracked, right? Everything is - you have a problem, you stack overflow it until it bends to your will, and then you find a solution. You don't really care how someone got there or what was the journey that led to that solution.

[04:59] Assaf

And that's - it exactly where I see myself and where I see our team kind of, you know, weave our way, weave our way into this because it's our job to educate those developers to teach them more about what other possibilities there are there. You know, what are some of the use cases? What are some of the best practices? There are a lot of anti patterns out there. And a lot of companies, you know, it's everything - everything goes. I mean, once the developer does something - if a new developer joins, the senior developer is going to teach the new developer: this is how it's done; this is how you should do this. And that's, you know, a lot of the time, a lot of times, this really narrows down the point of view of a developer. So it's our job to - the way I see it is our job to empower the employee by showing them there are multiple ways of achieving something. It's not a question of, can I do this, but rather a question of what is the best way to do this? So there's a lot of material to be learned out there.

[06:04] Tom

And it are things different now? Do developers need to understand their infrastructure more than they did in the past?

[06:13] Assaf

Absolutely. So if in the past, organizations worked in a way, where, you know, some sort of senior management or mid to senior management used to decide on a product and just kind of give it to their developers, and here - this is what you need to use from now on. Now, it's kind of like shared responsibilities: things are being built from the ground up rather from the top to bottom. Developers today - and not just developers - I mean, developers, sure. But also DevOps engineers, and IT and security and QA, people in these positions, are now faced with a massive ecosystem of tools and services that they can use.

[06:55] Assaf

And they now have the freedom to choose, right? And it falls under their responsibility to use those tools. Now, one of the biggest advantages, specifically with AWS the way I see it, is that a lot of these services are what we call managed services, which means that they're purposely built to do something very specific. So just by using a managed service, you're already shielding yourself from misbehaving, right? You're already shielding yourself with doing something that you're not supposed to do because part of that shared responsibility model means that even though we're responsible for maintaining all the services and all the infrastructure and the data centers and everything else, you're still responsible as the customer. You're still responsible for configuring your services correctly, and engaging with them correctly, and creating an application that, at the end of the day, is considered secure from end to end.

[08:00] Tom

So it sounds like there are two things that help with changing the behavior of the people that you train, right? So there's the education part, right, understanding their responsibility. And because things are different than what they were before. And the other part is the tools themselves that help to enforce new best practices. Would you say that's correct?

[08:29] Assaf

Definitely. So right now, when I speak to developers and DevOps engineers, I always tell them, "Look, we can, you can add additional steps and additional security measures and additional testing to pretty much every step in your pipeline. Right? I can do whatever I want at this point. But that doesn't negate the fact that you still need to educate the people themselves, you know, to have protocols in place, to have a good understanding of what this product does, how it does it, what are we supposed to do if something bad happens?" I always challenge my customers. I tell them, you know, “Imagine, imagine if something happens in the middle of the night, right? Imagine you get a phone call in the middle of the night, and something crashes. What do you do? It's your job, you know, it's your job to fix it. You need to have at least some sort of answer to that question. Your responsibility doesn't end when you push your code into whatever code repository you're using.” So yeah, it's definitely about educating developers to have a wider range of view when it comes to deploying and operating their applications.

[09:40] Tom

Have you seen any interesting ways where the DevOps infrastructure, continuous integration tools, are actually used to enforce best practices where, you know, education is not enough?

[09:56] Assaf

Yeah, absolutely. I mean, we always say with AWS that, you know, security is job zero, right? So, if education is not enough, as a cloud provider - as a cloud platform, there are many tools that you can use now that are both preventive and, you know, detective, I would say. So from a preventive point of view, you now have the ability to automate the process of creating accounts for adding all kinds of policies that could prevent certain people from performing certain actions. So, for example, if I have an AWS account meant for developers who only deal with the test environment, then people in that account will never be able to deploy anything to the production environment.

[10:44] Assaf

But if that's not enough, we also have detective measures. So there are a lot of different tools in AWS, like AWS, config, and others, that are always scanning your account based on rules that you defined as the security officer or as the team leader, or whoever you are. And the findings are always reported, and they can also be automatically remediated. So just as an example, to kind of solidify the idea. So, for example, if I'm a security officer, and I decided that we are not allowed to have any unencrypted hard drives on any unencrypted EBS volumes to our EC2 instances, then that's something that I can catch online. And if that happens, I can make a decision of what I want to do with it. So I can either send a message or pop into the Slack channel of that developer. Or I can just take action, and you know, turn off or delete that volume and prevent this from going on too long. So yes, there are many - there's a whole ecosystem around dealing with these incidents. Absolutely.

[11:52] Tom

So it sounds like there's educating and there's putting limitations so users won't do things they're not allowed to do or are forced to do things that they must do. Are there any tools or dashboards that you see that help to increase a sense of responsibility? Like innate responsibility?

[12:20] Assaf

Yeah. There, I mean, again, it's divided between the education part and the practical part. So from an educational point of view, we always talk. There's something in AWS called the shared responsibility model. It's a model that kind of draws a line in the sand that dictates what is under the cloud provider responsibility - as we are responsible of the cloud, while customers are responsible in the cloud.

[12:45] Assaf

So it talks a lot, again, a lot of best practices and what you can do or cannot do, or how you should do things. But you know, in reality, we understand that sometimes things don't work, just just like they do in theory. So yeah, there are so many, there are so many tools that people can use, whether it's, as I said, whether it's reporting tools or detective tools or prevention tools, that will always - at different levels, right - will scan what's happening in your account, and then either report or automatically remediate it.

[13:21] Assaf

Whether it's from at the network level, the code level, the service level or the account level, so you can, but, you know, I have to say that at some point, you have to find balance, right? You have to find balance between allowing a developer or a DevOps engineer to do their job and also have enough guardrails. I would say to make sure that everything is safe. And I've learned over the years that this middle ground - this balance - is very different for a lot of companies which will be very different for a startup of five people and for an enterprise of 5000 people. So it's an evolution, right? It's an evolution that happens over time.

[14:07] Tom

Okay, so with that thought, thank you very much. That's all we have time for today. Assaf, thank you for coming on today.

[14:16] Assaf

Thank you for having me. Bye. Bye.

[14:18] Tom

Thank you, everyone, for tuning in. And I hope you'll join us for our next episode. To find additional episodes and full transcripts, or if you'd like to be a guest on our show, find us on our community page at Swimm with a double m dot io. Bye for now.