Joe Lust – Ubiquitous Quality Through Continuous Testing Pipelines

Next, I'd like to welcome Joe Lust with one of our partners, Mabel. Mabel focuses on intelligent testing, automation application, security testing isn't the only type of testing that software factories want to shift left. And Joe is going to show you how quality testing can also shift left by using a review app, kind of like what we do with dynamic application security testing.

Maybe his use get lab to shift quality testing earlier, delivering results to the developer before the code ever leaves their hands. We invite you to engage with Joe in the chat any time during his presentation or after. Thanks for joining our session today.

We'll be talking about ubiquitous quality and how you can achieve that with continuous testing. I'm just an engineer at Marvel, where we build testing software solutions. I've been building for the cloud for the last 10 years for the Web for about 20.

If you'd like to keep up with me, check me out at least Khoder on Twitter, but enough about me. What about our talk? We're going to talk about three ideas.

One, I'm going to convince you that you need continuous testing too. I'm going to convince you that you can achieve that. And three, I'm going to show you how to do that.

Let's jump in. So continuous testing, what is it? Continuous testing is testing at all levels if your code lifecycle, not just testing the code that went out to production, not just testing in Kuai or in your second pipeline, but testing at every Kimmet in every merger request.

Continuous testing is pervasive testing, and as we chase the idea of continuous testing, we have to understand why we haven't gotten there yet. To understand that we need to look at the pre dev ops era. Most of you here today, your dev ops experts, you can remember the way things used to be before dev ops.

We actually didn't test that off. You might have worked at companies where testing occurred monthly or quarterly. And oftentimes this testing was paper based.

It was manual. Somebody followed a script. And this meant we didn't really get a lot of coverage because it was tedious and costly to do that testing.

And the places we did that testing, those were sparse. Maybe we had a UAT environment or a QA environment, but there weren't many of them and we had to be careful not to break them. And the result of all this testing was often reams of log output that maybe we've talked with many testing professionals.

I remember one call with a Sara and Sara's story was about how her typical Monday was getting to work, getting her coffee, opening jenkins' and just looking at logs for several hours to figure out if the tests were good or bad. That's the way in the pre DevOps days we did testing. But now in the post DevOps days, harnessing the skills you have with service in the cloud, we can do continuous testing.

We can test instead of every quarter, every month. You can test every committee. That's thousands of times a month.

And we can use fancy new automation tools, selfie contests and into coverage, all of the cutting edge tools that are now available in these can harness, as I said, the cloud and the surveillance tools in your toolbox to give us unlimited test environments. And finally, with the new insights we get from an email, we can boil down Sarah's reams of server log cumquats and jenkins' and give you the developer just the correct insight you need to fix that book. As we move into that post devops period of testing, we move closer to the source of the book and I admittedly must say people like myself, developers, we are the source of books.

And so we need to shift left. We need to get closer to the development lifecycle, to the source of those books. We don't want to catch the books out of production, want to catch them where they're made.

We catch them before the merge. We get the greatest savings and I will get into the details. But as you probably already heard, there's tons of research that shows it further towards production.

You get many fold increases in the cost. And now when you and your developers are no longer fighting fires, you can actually get more work done and probably come to work on a Monday. You have that list of things you're going to achieve and you don't because all the fires you have to fight.

Well, we can get rid of this whole section of fires. And finally, you're killing a team when they're no longer chasing the silly bugs that I created, those regressions that have been broken four times in a row because we've got better automation that catches the source of each commit. They can actually focus on the needy, hard to solve, hard to test business logic problems that you've got all of this because we're shifting as far left as possible.

So that was continuous testing, how can you implement it again? Well. I'll show you.

First, we're really going to need two things, we're going to need a way to scale our application under test. We've got to deploy that code somewhere to test it. And we have hundreds or thousands of these applications to test for each commit, you name.

Then we need to do something highly skilled, for example, like get a few apps in second. Once we've got our test code out there, we need highly scalable test running solutions. And you can go with the SAS solution, white male, which we much prefer, or if you feel very proficient using tools like communities CONTAINERIZE and browser's things like that.

You can also use these modern tools like puppeteer and you can combine them in your own cloud skills and you can self host regardless. You still need a place to scale out and run all of those tests. So a quick word, what's Mabel?

Well, Mabel is an intelligent test automation platform. Our core tenet is we want to make everything easy, make authoring tests easy to use a chrome extension, you can click around and you've got your test and now can be replayed on many different browsers and very easily retrained, maintained. Many features come second, we make a unified platform because testing touches many areas, it's not just the tests that used to operate, but it's your pipeline, your source control management, your test track tools, your books.

And all of that can be brought together in one place, all its touch points with Mabel. And when you run those tests, you get tons of output, honestly. Meanwhile, we run one of your tests, we actually instrument everything and get almost gigabytes of data.

Well, I won't get into the details, but just check it out, you'll see it's all there. And finally, we want you create a customer centric tests. We're not just loading a silly little react button, clicking on it in a test harness.

We're actually running the entire application hosted on a server, going through your off, clicking on a real browser. That means that the buttons are coded or invisible or some crazy access is happening. You get the real world results.

That's what you can do. Now, you mentioned the second piece, you need a place to host that code, the code under test, we believe the solution for this ESKIL preview environments. What is a preview environment?

It's a OnDemand short lived environment. That means want to make it use it discarded. That's why we've got this new recycling icon here, because you might have worked at some large organizations like I have in the past where the environment was sacrosanct.

And if you broke it, 10 people told you you broke it. Well, with OnDemand preview environments, everybody gets their own environment. Thousands of environments.

Doesn't really matter how many you have. Finally, I would just note, if you haven't heard of previous environments, you might have heard them as or perhaps ephemeral, perhaps preview apps. There are many names, but they all mean the same thing.

That place that you're testing your. Let's look at an example of a real world prevue environment and emerge of ways to go out here, we can see that our famous 10 X developer, she's been busy and she's got merger request we need to review really quickly. So let's say it's really important and it's for a major customer need to do it ASAP so we can see here that it's only success.

So what could go wrong? I personally like you. I'm a professional.

I don't really know that much about success three. So I think I would be inclined, given how important this customer is to just go down here and click approve. But what if there was a better way?

What if I could really understand what these few lines really meant? What if I could show rather than be told what it is? Well, you can do that and get up using website.

So here it's the same merge request, but there is a review app here and we've been told it's been deployed to review this rule and we can actually click the button right here to go and look at that app. So let's do that. Oh, my goodness.

That that is not right. Now, you might not be familiar with the mobile home page, but we are not a candy store. We don't sell candy canes.

So this this is totally unexpected. And we might not have caught this with a typical Seleznyov test. And if somebody didn't go in and actually look at it, we probably wouldn't find out until our customers were tweeting pictures of it to our detriment.

So if we went back and we gave everybody this button, we would democratize and enable testing across our organization because you would not have to understand its three. You could actually just go in and experience the application as a user would. And that's something that almost everybody can do.

So with review apps and these sorts of buttons, we really empower everybody in your organization to contribute to Swashing books. So since your dev ops experts, you're probably thinking there are some challenges here. How are we actually going to operationalize all this?

For example, you might be saying all these servers, they're going to cost me a ton of money and all the networking that VPC, the firewall rules and all the DNS entries and domain changes I need to make. This is going to cost me a ton of money and time. However, if we use your now post office era cloud skills, we can use Cervalis to get rid of the servers.

There isn't a server problem and the networking problem. Let's just outsource that as surplus solution. They can take care of that for us.

We push that lamda out Amazon will route everything for you. What about all those domains and how do we up DNS? Don't worry, we can use path routing or a wild card.

Now you don't need to make those dangerous changes for centuries. And finally, cost isn't really concerned because these new surplus solutions, they all scale to zero. They cost micros cents per request.

It's a new way of thinking of things, but basically it means that cost is no longer a challenge. So how do we go about making such a service, prevue environment? Well, I want to show you a simple case of the typical single page application we see today where it was the sorts of applications you probably have in your organization, things like react or angular extension Jass.

All you need to do is take that static content, mostly HTML JavaScript, and drop it in a bucket, as we see here with these logos. It could be in any one of these cloud vendors. It could be in the buckets that you have an S3 and Google cloud storage.

The blob storage that you get at is your. And finally, you might have some dynamic pieces of your application, we can drop those in a container and we can run them on, for example, at about LAMDA or in cloud functions, Google or in your functions. Now, the dynamic parts in the static parts are hosted, but their own highly scalable solutions, for example, that may well, you've gone in and worked and seen places where Google cloud function might be scaled up to twenty five thousand parallel instances where it's not a problem, it doesn't cost much and just scales.

It's not something that you your divorce team has to worry about. So let me give you a detailed example of Prevue virus we use here, so every time we build one of our apps and maybe we take those HTML and JavaScript outputs, react compiles for us, we go and put them in a Google cloud storage bucket and that sits behind a CDN abandoned domain name. So all that routing and networking is taken care for us.

We take the dynamic pieces, maybe some path rewriting it has to happen. We place that in cloud run, which is a managed container runtime similar to a Stargate, and now we just keep scaling every time it gets deployed in the last quarter. So we've got a thousand preview environments for about a dollar.

Really? Is that easy? And you might be wondering the details of how do we relate to all of those different application versions?

Well, very simply, we can take our base URL. We can use path routing, so we give it a name, the pet store app, and then the hash from the comment you made. And now that will route to that specific version and that domain can have many thousands of different versions on it.

Similarly, you can use more complex mechanisms like wild card domains. We've got an article here for you. We actually go into the details.

We have code examples of doing this in Amazon and doing this in Google cloud. Really is that is. So now you've got preview environments.

You've made them scalable. We still need to make them work for you. As we mentioned earlier, we need to take those tests, those complex tests you've got that really test the entire swath of your application.

We need to point them at your previous environment, each of those permits, so we can get that instant feedback back to the developer was making that change. So she needs to pass fix that problem before it gets any further. Further, we need to have that in your Murgia quest, as we show in that example, to democratize that testing across your organization so that anybody in your company can go in and see what the change looks like.

It can be shown, not told what it looks like, and that really enables people across the board to contribute. And finally, you can use some tools you've got and have pipelines, for example, merge trains to further take this to the limit. You can take Maine for emergen and you can deploy that to another environment.

So you can be very sure when you merge to Maine or master that your tests, your code are all functional here. For example, we have what Mablethorpe does for us. And every time we make that new commit and we get oinks and says, here's the preview environment you can go and check out.

So you've gone through what this looks like and how you build it. Let me show you the brass tacks of building preview environments and scaling the Mount in GitLab. So, GitLab, you're probably using CI/CD pipelines and there's really two components you need to use here.

One, you need a review and the review app is that placeholder that tells the pipeline that you have deployed something somewhere. It gives it a name. It's the placeholder for that application that you just posted to you need to do is the dynamic environments.

Once you've to point that out there, that concept of the dynamic environment is going to what downstream consumers in your pipeline use that application you just made. It's going to let them test against it. If you would do anything else, they might need to.

And this is bypassing environments in the Urals that are passed into the environment. Excuse me. So let me show you the email, because as you all know, we are these days, you programmers.

I went to school for programming, but YAML is where it's at these days. So let's break this down here. We have a stage in the pipeline and we're just using Noguez because this is going to be a reactor.

And so we use the base note image. This is deployment stage and now we're just going to run the build command. We're going to make the artifact we would deploy for our application.

Now we're just going to use your tail because in this case, we're going to Google cloud storage and we're going to say make these files public Rabil directory and go and drop them into our bucket. And we're going to call it happening, whatever your applications name is. And we're going to use the shot as the unique identifier for it.

And here's the magic. We're going to define an environment. We're going to give it a name.

So now our environment is free to use our shop and we're going to tell it where that oil goes to. And now everywhere downstream from here can use the environment variable to get the environment. That's a mouthful.

And they can test whatever they need to by hitting this URL. And finally, what do we have here? Well, we only want to do this research requests for branches because that's what we're giving people.

That instant feedback we're going to exclude means you probably want to use that in a different way. You probably have other tools. So that, in a nutshell, is the.

One thing we are announcing today is our pipeline integration. We want to make it even easier for mobile customers on getting to use us as that SaaS provider to scale out the testing phase. And we're going to want to do is run all of our intelligent tests on the mobile cloud, which is going to help with scaling, because maybe you've got one test you recorded in Chrome, but now you want to run it in Chrome and Firefox and Safari and Internet Explorer.

And we like parameter size these maybe they're 10 different permutations. So 10 times for that's 40 tests. So we're going to instantly paralyze and launch all those containerised browsers.

Right. When you made that commit, we're going to get all the answers and then we're going to tell you this passed or didn't pass right there right away for you. So you don't have to worry about all the operational headaches that cost that test.

Status update is going to pop right into your merge request, as we see here, including links to the rich test outputs so you can rapidly drill down and say, why did it break? OK, here's the trace from chromes memory. I see the exact problem.

So I don't have to go and make my own test harness. I can now change that line of code. I barely broke my workflow.

I stayed in my flow state. Now, when I mentioned the live results, you can go to you, this is an example of what you'd get in the mobile app if you're using that is your service provider where we actually see all the deployments, how well they're running, and you can actually click on any of these runs, drill deeper, get the screenshots. As I said, Chrome traces you can dump the HTML from the page.

You can look at the actual network traffic that went back and forth, many, many different outputs in there. So you don't have to go and try to find a copy of Internet Explorer 11 and figure out what went wrong. Let me show you again in the Yamal how this would work here, we've got a build stage.

It's the marble test running stage, and we're just pulling in the premade marble docker image. And in our test image here, we're going to pass this argument, wine and all we're saying is A for the application ID. So this would be the ID of the application one test and then we're passing the URL and here's that tongue twister environment environment, the URL I mentioned earlier.

And this is just being pulled in because we created a dynamic environment for review. So this would be passed down through our pipeline, subsequent stages. And finally, we're using the built in wine to run the script.

Executable tests, which is actually going to call me, will go out to our API wants. As I mentioned, those 40 parallel browsers get all the results and it's going to be a good or bad 090 zip code. And it's going to do this for all of your requests, so.

What does this what you do? Well, it's going to let you not worry about how you scale those tests now, since you're all dev ops experts, we've got even more treats for you if you like. See lines, we've got the Maple Clie.

Once you roll up your sleeves and dive right in, anything you could do on our ovei. Excuse me, most things you could do on our UI, creating Westing, editing a lot of our entities. You can do that right here with a single line abash so you can stay in your most productive state, the command line, and tie this into all of your best scripts so you can easily inserted into your other pipeline workflows and using the supplies you can see here you can actually get the real time test outputs from those tests you're running.

Of course, you might be even more comfortable with Kerl some people, so we actually have an API. I. and you don't want to use pre-built integration, you can just hit the API directly for test results and creating deployment's, all those sorts of things.

So. Let's zoom out for a moment. Our core takeaway here is that you, as dev ops professionals, are critical enabler and your organization to bring continuous testing to the forefront.

And you can use your skills with serverless and with cloud to shift is far left as possible. Finding those bugs in your organization and empowering many people couldn't contribute before to contribute to the quality and the pace at which you are shooting product. If you've got any more questions, you can ask them below.

I'll be listening during this talk or you can see our references we've got here. We've got some blog posts that give you the details of how you can set up Cervalis environments on other vendors like Amazon or GCP. We've got the slide deck.

And if you'd like to learn more about our integration, it's right here. And finally, Mabel is actually free, as in no credit card. Go kick the tires, run a bunch of tests.

You can do a free trial there. If you'd like to keep up with us, you can check Mabel HQ or was or on Twitter. Thanks a lot.