Mastering the DevTestOps Symphony: Unleashing the Power of Collaboration | DevOps Experience 2023

Envision your code as your creative ally, forcing your desires and crafting a masterpiece of innovation. Today, I'm about to reveal the enchantment that brings this vision to life. Ladies and gentlemen, visionaries of technology and enthusiasts of the future.

Welcome to a moment that promises to transform your perspective and ignite your curiosity in this vibrant ambience of the DevOps experience. 2023 Conference When Knowledge converges with Aspiration. I am Shan Ani, and I'm honored to be your guide through the uncharted territories of mastering the dev test desktop sym symphony, unleashing the power of collaboration with ai.

So, dear friends, brace yourselves for a journey that promises to redefine the way we perceive technology and teamwork. Together we are about to step into a future where AI and human brilliance harmonize reshaping the landscape of dev test desktops with excitement in the air and promise of innovation on our loops. Let's explore how AI is nurturing collaboration and dismantling the barriers that have hindered progress, the future beckons, and as explorers of innovation, we stand ready to embrace it.

So let me introduce myself. I am Shan Ani, an avid tech enthusiast with over eight years of professional work experience. I've led a team of SSDs across 12 plus projects, stood as a keynote speaker at numerous tech conferences and taken up the mantle of being a brows stack champion, as well as the mome chapter lead.

I hold certifications across a number of them, as you can see them listed here. Um, but when I'm not an immersed in technology, I'm a passionate traveler, a dedicated foodie, and a tech advocate. As a blogger, broadcaster and podcaster and an open source contributor to WordPress, I've cultivated a network of around 13,000 followers on LinkedIn.

If you wish to know more about me or uh, wish to get in touch, feel free to re uh, reach out to me via my social media handles that understood. And if you wish to know more, you can always visit my, uh, website or just simply Google me up and, uh, you'll, you'll get to me. So without any further delay, let's dive into what I've covered as part of today's agenda.

Uh, well, it's time to embark on this exciting agenda and uncover the future of dev desktops together. So the very first point that I, uh, would like to put across is dev desktops, the modern path to agility. Well, you, we'll be, uh, exploring the captivating world of dev desktops, which is an approach, um, that is rewriting the rules of software development.

So here we'll discover how it azures in a new era of speed, agility, collaboration, and efficiency. Next, continuous integration and continuous delivery. Um, this is supposed to be the heart of modern software development, and here we learn how, uh, different practices reshape the development landscape, enabling rapid and reliable code delivery, third best practices and real world examples.

So drawing from real world examples, we navigate through certain best practices that we have, uh, seen across different organizations that have adopted dev test stops. Fourth myths and misconceptions about dev test desktops. That's what we'll bust fifth revolutionizing testing with Charge g bt, Bard and ai.

And lastly, we'll see what's next. So as we wrap up our journey, let's peer into the horizon and explore what lies ahead. The tech world as we know, never stand still, and we'll discuss the exciting possibilities that await in the future of dev desktops.

So let me begin with a quote by Aristotle. Quality is not an act, it is a habit. I know this is pretty self-explanatory, and what the quote generally says is quality is a habit rather than just an occasional activity.

In the context of software development and DevOps, it emphasizes the need to prioritize quality at every step of the process from development to deployment. Talking about software, most of us are already aware that the software development process starts with an idea that is eventually transformed into a concept and a functional product to ensure software quality testing is essential throughout the entire development process. Unfortunately, some people over, uh, overlook the importance of testing the idea in itself, thanks to Dan Ashby, as this image makes it easier for y'all to relate with what I'm trying to convey.

Testing the idea is critical for identifying information, refining and solidifying it. This can be achieved through various testing, integration and acceptance, uh, testing formats. These methods enable us to scrutinize different artifacts, identify their shortcomings, and refactor them accordingly.

Consequently, we can use these artifacts for all the activities in the software development life cycles, such as designing, planning, development, testing, and software maintenance. Furthermore, it is crucial to have a clear understanding of the software requirements and involve users in the entire testing process. In add in addition, having a robust quality assurance process in place, which includes code reviews, testing plans, and incident management is essential for developing high quality software after all quality software for one, according to Dan Ashby's DevOps model testing is a critical component of the DevOps workflow, and it plays a vital role in ensuring the quality and reliability of the software that is being delivered in a DevOps environment.

Testing is integrated into the development and delivery process, and it is performed at every single stage of the process. As you can see in the image here, one of the key principles of DevOps is C I C D, which we'll talk about shortly, but this means that code is integrated and tested frequently, and the software is released to production as soon as it's ready. Testing is an essential part of this entire process, and it's used to ensure that the code that is, uh, being integrated is of high quality and that is, um, uh, it doesn't break any kind of existing functionality.

Talking about automation, it is another key principle of DevOps and it's used to automate as much of the critical software delivery process as possible, which includes testing. Uh, automated testing allows organizations to run tests quickly and consistently, and it reduces the risk of human error in addition to integration and unit test testing. And DevOps also includes functional and non-functional areas of testing, such as performance, security and usability testing.

This ensures that software meets the requirements of the customer and that it's fit for purpose. And last, but not the least, um, it is also an important part of the feedback loop process within DevOps. So by measuring the performance of software delivery process and using that feedback to continuously improve upon it, that's how organizations can ensure the software they're delivering is of the highest quality and meets the needs of the custom.

So does that mean tests tests everywhere? As software professionals, we are familiar with the challenges when traditional testing struggles to match fast software releases, and that's where agile testing comes in. It's a practice that integrates testing through the software, uh, development lifecycle, ensuring high quality and efficiency.

So how does testing fit into the overall DevOps workflow? Well, that's where we talk about dev test desktops, where the key focus is on continuous testing. Now, unlike the agile manifesto, even dev test desktops has its own manifesto, which was coined in 2018.

And as per that, the goal of dev test ops is not to silo test from DevOps, but simply to raise the awareness and the visibility of testers and testing as integral parts of the DevOps quality culture dev test ops has set a culture where software developers, testers, and operation engineers work together, which adds value to the product, enhances quality, and helps in faster delivery. Moreover, dev desktops ensures continuous feedback about application issues from testers to developers throughout the stages of the product development. This indirectly reduces the business risk and the possibility of finding defects at later stages.

So how does it work? Well, uh, here are some, uh, five simple steps, um, that you can follow. The very first one being implementing bug free policies, which ensures bugs raised on production must be evaluated and closed in the minimal possible solution duration to build a successful CD pipeline, reduce the cost, time, and risk of delivering software changes.

That is by en enabling Incremental updates include more quality control steps or, uh, enable quality gates throughout the pipeline. Um, use test first approaches like test-driven development, acceptance, test-driven development, behavior driven development, et cetera, or run automation scripts to test out the new features. And last but not the least, store and reuse the suite of automation scripts for future use to assure stakeholders that testing is thorough and adequate.

Four types of testing that, uh, should be included in the dev desktops to work successfully are exploratory testing, manual testing, ad hoc testing, and automation testing. Now, like we spoke about the dev desktops manifesto, these are some of the five key principle guidelines that have been stated in the dev desktops manifesto. First, continuous testing over testing at the end.

Second, embracing all testing activities over only automated functional testing. Third, testing what gives value over testing everything. Fourth, testing across the team over testing and siloed testing departments.

And fifth product coverage over code coverage. Now these are pretty self-explanatory, so I won't dive into each of them in detail, but, um, heres, here's a quote for automation. And as you can see, automation may be a good thing, but don't forget that it began with Frankenstein.

So here as a takeaway from the same, just because you can automate doesn't imply that you should be automating it. In fact, automating the wrong things can trigger a, a ripple effect of damage. Automation is not a tool, it's a mindset change from who will do it to how will it get done.

Here are some strategies that ideally one can follow for DevOps, uh, with circle around continuous integration and continuous deployment. For that, we have tools like Jenkins, Travis, CI Circle, ci, bamboo, and so on. You have infrastructure as a a code like, um, chef puppet, Ansible, Terraform, and you have continuous testing, continuous monitoring, uh, performance monitoring, and various other tools listed out, for example, elastic, um, Splunk, Kibana new, uh, new Relic, Datadog, Grafana, et cetera.

Um, some of the key principles of dev test desktops, the very first thing I'd like to highlight is testing is everyone's responsibility. And as most of you already know about this test pyramid, which was coined by Martin Fowl and Mike Khan, software tests are basically bucketized into levels of different granularity, and that's how, uh, as many tests that are at the bottom of the pyramid need to be automated, and that's where the focus must lie. Another important principle to follow is, um, adopting an agile testing mindset.

To understand the agile testing mindset, we first need to determine what makes a team agile. To me, an agile team continually focuses on becoming self-organized and cross-functional to be able to complete any challenge they may face during a project. In addition, the team must put the customer first and ensure that the customer receives the best possible product at the end of each iteration.

Testing manifesto has, um, certain principles as well, like value testing throughout, over testing at the end value, preventing bugs over finding bugs, value testing, understanding over checking the functionality value, uh, building better features with quality than about how to break them. And lastly, value working as a team rather than working in siloed departments. Setting up a test center of excellence, which is a strategic framework that mostly circles around processes, tools, KPIs, and innovation plays a key role for any organization to shift its focus to a quality first approach.

Some stats around, uh, setting up a test center of excellence. While organizations that have adopted test center of excellence have reported that they were able to achieve 50 to 70% of automation in testing with an average of 30% reduction in test cycles and 2%, uh, reduction in the defect leakage. Here are certain tools that will help you, uh, in, in achieving the best possible dev desktops within your organization.

The first thing around automation testing tools, you can see a number of them listed out like robot framework test, Sigma Cent and so on. Uh, you have version control tools such as Good Automate, uh, auto Automatic Automatic Code Deployment tool allows, um, deploying code on daily basis. For that you have Jenkins Bamboo and so on configuration management tool like Chef Puppet.

Uh, then you have ticketing system tools or application lifecycle cycle management tools like Jira, Azure, DevOps, and so on. And lastly, you have monitoring and provisioning tools like Docker, Kubernetes, Datadog, new Relic, and so on. Uh, talking about accelerators, uh, the very first thing that comes to mind is service virtualization.

So that's where you do all the mocking and tubing, and you have various tools like Wire mark and overfly. Um, second thing is integrated test environments. So it's very important that you have different test environments to actually test your flows before they go live to your customers, right?

So you can enable a different set of environments like development, staging, pre-production, or cannery environments, and also maybe you make use of feature flags that can help, um, improve your overall quality. Next, we have continuous transaction monitoring, continuous infrastructure monitoring. That's where your Datadog and New Relic will come in.

Uh, self-serve test data portal and shift, right testing shift, uh, left is something that most of us, uh, talk about at various forums, but shift freight is also equally important when it comes to, um, maintenance and continuous monitoring of your, uh, production flows. Lastly, consumer driven contact testing. Um, it's, it's more of a contact which is agreed upon by your consumer and your partner.

And based on that, most of your tests are, um, proceeded. Lastly, test automation at scale. So when you are scaling your test automation, um, there could be various factors that need to be considered, like, um, modularizing your code or making it run on the cloud and, and various other factors.

You can also enable parallel testing and so on and so forth. Role of C I C D in-depth desktops. Um, as most of you know, C I C D stands for continuous integration and continuous delivery.

Uh, the, the major, uh, role of C I C D is as follows, uh, automate the test cycle by getting all possible critical tests automated work with the development team to set up automated unit tests. That is by adding quality gates like, uh, sonar Q or VERA code and various others, um, or, or maybe at at least, uh, enabling unit tests at the initial stages, uh, while you are promoting builds from development to test environments and onto production. Third, set up automated build and deploy processes.

Define and automate the handoffs between the development and the test teams. Create test environment profiles in preparation for automated environment provisioning, leverage cloud or virtual clusters to set up testing environments on demand. And lastly, integrate the automated testing capability within the development teams, um, continuous integration capability to establish continuous testing.

So as per this diagram, you can see I have, uh, bifurcated this into four different sections, which is code development, code, uh, continuous integration, continuous delivery, testing, and deployment. The very first part is where most of your developers write the code, push your changes onto your local branch, then onto feature and master branch. Um, people can use different, um, unit tests, frameworks like jest, oid, J, unit five, and so on.

And, um, finally, once your code is onto master, um, we, we make use of continuous integration tools like Jenkins. In this case. We have, uh, red Hat OpenShift also, um, where, where we are running these on-prem and we have, uh, JaCoCo integration, we have sonar cubes as quality checks.

And once your build passes, that's when we promote it onto the next set of environments. If it fails, uh, it needs to repeat the loop in itself. Um, once we come to the final stage, which is the delivery testing and deployment, we have our, um, A A W SS services in place.

Like you can see we are using ST buckets code build, uh, cloud formation and so on. And, uh, that's why we do the entire cycles of build, deploy, test and release. Uh, again, at the bottom right, you can see a number of manual and automated testing tools and frameworks that are in place.

And this is one such solution that you can use within, uh, your organization. Now talking about certain pros and cons about dev desktops, um, well, everything has its own pros and cons and, um, I think pros most of it are straightforward and simple, um, already listed out. But talking about the cons, let's, uh, deep dive into some of them a bit.

So the very first one is outsource infrastructure, need access to special development expertise. Now, if, if you have a DevOps guy within your team, it makes things simple, but yeah, uh, otherwise infrastructure management is a little bit of a concern, and hence it's highlighted as a con. Second, compatibility issues may appear while, uh, imitating the production environment.

And, um, that's, that's what we discussed already. Third, security is one of the most important concerns with dev desktops, and that's why you would've already heard of the term DevSecOps, and it's, uh, a lot prevalent already in this society. Some best practices around DevOps, which circle around testing, security and automation.

So, uh, with respect to testing, automate as much of the testing process as possible. Of course, the critical scenarios. First, uh, right test cases that cover all aspects of the software.

Incorporate testing early in the software development process, talking about security. Write secure code. Firstly, test for vulnerabilities.

Manage infrastructure securely and respond quickly to security incidents. Automation, automate as much of the software delivery process. Use tools that are easy to use and integrate with, monitor and analyze performance of soft software and infrastructure.

Uh, moving on to certain real world examples. I'm sure most of you have heard of Netflix, right? Netflix is one of the most well-known examples of a company that has successfully implemented dev desktops.

Well, they earlier worked in silo testing departments where, um, in fact each of the functions would do their own, um, respective function in, uh, irrespective of the other function. So if you see an architect would be busy designing the architecture, um, of a, of a product, the, uh, developer would be developing code tester would be testing, operation engineers would be busy with the support and release of the specific product. They later moved on to operate what you build model where, um, the developer takes the sole ownership and responsibility of everything, and he's also involved in the decision making process, um, has access to the tools and frameworks, and also, um, plays an equally important role as compared to the test and the operation engineers.

So what, what are some of the lessons that we learned from this while, uh, Netflix also adapted to, um, continuous testing, automation and, uh, following the C I C D approach, don't build systems that say no to your developers. Focus on giving freedom and responsibility to the engineers. Don't think about uptime at all costs.

Price the velocity of innovation. Eliminate a lot of processes and procedures. Practice context over control.

Don't do a lot of regular sta uh, required standards, but focus on enablement. Don't do silos. Walls and fences.

Adopt you build it, you run it. Culture, focus on data. Always put your customer's requirements first.

Don't do DevOps, but focus on the culture. And something similar is what we learned from Spotify. Um, here is one of the case studies, I'm sure again, Spotify, most of you know, is, um, an audio streaming platform with about four 50 million, uh, users.

And, uh, Spotify too followed the dev desktops approach at some point where they, um, focused on having proper C I C D approaches focused on continuous testing, continuous integration, and also maintain the feedback loop with their customers so they could, uh, deliver the best quality product. Uh, looking at some global statistics, uh, right from 2023 to 27, we can see according to a survey by a puppet, um, organizations that have implemented DevOps practices and automated their software delivery process, I've seen 60% reduction in the time it took them to deliver software to production. There was another software, uh, there was another survey by Gartner, uh, which noticed that there was 25% reduction in the number of defects in the software they deliver after adopting dev desktops.

Lastly, there was another survey by Accenture that showed, um, customers noticed 20% increase in their ability to deliver new features to um, production and 30% increase in their ability to respond to customer needs with a 50% reduction in the time it took to resolve their issues. We earlier discussed that we bust certain myths. So, uh, there are basically three myths that have been, uh, in and around dev test desktops for a while.

The first one being a hundred percent automation. Um, people say continuous testing requires a hundred percent automation, but as we all know, a hundred percent au automation of an application is not feasible. Hence, some scripts are run manually in the process of continuous testing.

Large organizations do not benefit from continuous testing as they have huge development and QA teams which make coordination difficult. Lastly, um, doing the cost benefit analysis. People say Dev test desktops is expensive.

Yes, it is a one-time investment, but once the team is equipped with functionality, a return on investment is tremendous due to better cold quality immediate feedback, uh, finding the root cause, easily fixing bugs as bugs are detected earlier, the cost of fixing also becomes much cheaper. Um, al almost no human mistake as the entire processes automated talking about the most interesting part of our session today. So that is revolutionizing testing with charge G B T Bard and ai.

As most of you know, uh, the evolution of software testing where the first software was basically made in 1948, and this is the current era where we are using AI to solve our software testing and development needs. So when we talk about testing in the cloud, it, it is a process where testing is done on a cloud-based infrastructure rather than, uh, traditional on-prem hardware. Cloud-based testing provides a flexible, scalable and cost-effective solution, uh, for ways to test software applications with chat DT borrowed and AI cloud-based testing can be further enhanced to improve testing accuracy and efficiency.

Um, it enables testers to create test cases using natural language, which in indirectly reduces errors. Um, AI can analyze data generated during testing to predict effects before they occur. Predictive measures can be taken to address effects before they impact the software charge.

G B T bar and AI can support continuous testing by providing real, uh, realtime feedback or test results and suggesting optimizations for test cases. It can be used to generate test cases automatically based on user's requirements. While AI can also be used to analyze test results and recommend optimizations for test cases, allowing test users, um, to continuously improve the quality of their tests.

Time for a quick dem. 5 model and you I will showcase, um, different examples like how I have used it for automated test case generation test execution and result analysis and effect prediction and prevention. So firstly, let's uh, take a look at the prompt I have given this, which is write a Python code to generate factorial of a number where the number must be taken as an input from the user and to come up with the most optimal code and urine test for the same.

So you can see it has come up with a pretty decent response considering tie FFL sketch blocks and also the accept and also unit tests are related to the same. So you can see it has considered positive numbers, a negative number, a string, as well as an non-ad number. Further, I ask you to add, uh, additional unit tests as that it can break the code and it comes up with an interesting large number format where we also know that it basically causes a concussion error.

So that is one such scenario it has initially added. And now considering those s case possibilities and the HK scenarios, I asked her to further optimize the original code says that, uh, ne the code never breaks and it has done its best to optimize the code. A added a a couple of other exceptions, but it also mentions that, um, at least this will handle, uh, the stack overflow errors.

However, if we still consider extremely large inputs, it may, uh, cause further memory issues. So it is, um, at least this word it says is more resilient than the regressive approach. com and trying to purchase a book using his credit card.

And I ask it to basically come up with all positive, negative and h case, uh, scenarios in our tabular structure and it has done that as well. Pretty, uh, good response. Next, what I do is, uh, you can see certain data is missing.

So I ask it to come up with certain tests data and also, uh, add a column which which can highlight where, um, certain tests are critical and those need to be automated on priority. So you can see it has considered those same scenarios. Again, come up with those set and combinations as well as, um, the priority of those that need to be automated in, in terms of high, medium and load.

So this is some scenarios that we can tie out, uh, experiment with chat, same, we can do it with Bard as well. With Bard. There's one additional functionality where you can also upload images and ask it to describe, um, what, what exactly is that?

So in this case you can see I've uploaded a model of dev test desktops with AI and it also highlights brilliantly, uh, how AI has been integrated within the dev test desktops and, um, best practices and a lot of it around it. And you can further query it and it'll also explain on those, uh, pieces. Next you have Google Labs.

Uh, Google has come up with investing AI experiments like, uh, there's one which is recently introduced called Project idx, where, uh, this is used for your app development where it's completely AI integrator. And uh, right now it's in the wait list stage. So if you are interested, you can just visit labs, Google and join the wait list.

Something similar with, uh, the messaging system. So you can, you have basically a new I icon which shows there as help me write or just reply to this particular text. And you'll have AI generated responses for the same.

It has also integrated with Google Search, your Google Workspace Suite, which shortly I'll show you with Gmail and something similar with slides, docs and various others. And you have your notebook, LM Music, lm, um, notebook L Also, I'll show you a quick demo. This is basically project I x.

Um, you can just join the wait list and um, as soon as you get access, you can experiment with this for app development. Then you have your Maker Suite. So Maker Suite is basically a platform that helps you to prototype a generator ai, which is provided by Google and it makes use of your palm a p i.

Then you have Notebook L, which uh, basically you can use it to, uh, upload certain documents and then query based on the same. So you can see I've asked what are some best practices for DEF desktops and it comes up with that accordingly. Again, is it important to follow all of these?

It comes up with its own scenarios. Uh, what if my manager doesn't care about following these practices? What are some alternatives that I should consider?

And it comes up with, uh, pretty accurate responses for those as well. So moving on to Dali by OpenAI. Uh, you can see I have asked her to um, basically create an image regarding bulge, khalifa's air aerial view with evidence.

And it has come up with decent, uh, images as well. Something similar, what we see with Mid Journey, where you can always give it a prompt, uh, by putting slash Imagine. And when you hit enter, you can just prompt, uh, in, in this format, like, uh, a person has asked finding in country Abu Dhabi Regional Airport, and uh, there other prompts based on which it creates the image and further you can optimize, create variations out of it.

And so moving on to uh, the Gmail Workspace suite. So, uh, you can see I have received an email and I wanted to basically create a quick response for the sim. So what I do is I'll just give a quick prompt, you can see the help me write option over, click on that and just give this quick prompt, which is positively reply to the semen.

And then I hit end up, it comes up with, uh, the best possible response that you can think of and I can always clear around with this. You can see by already in each generating the entire text or refining it by formalizing, shortening, or uh, just playing around with the same. And lastly, you have your hugging face.

So you can just visit the website and uh, check out the different models and and so on. And so, alright, so your was an interesting demo. Um, how we have leveraged AI into different areas of testing and, uh, let's look at certain best practices now.

So the very first thing I'd like to highlight is it's important to start small and focus on, uh, one or two cases initially to demonstrate the value of charge g bt Bald and AI in testing. Ensure that the data used for training and testing the AI models is of the highest quality and, uh, representative of, uh, the real world scenarios. Use a test automation framework to incorporate charge G B T Bard and AI into the testing process.

Develop a strong feedback loop to continuously improve the AI models and testing process. Monitor the performance and effectiveness of charge G B T Bard and AI regularly to ensure that it is delivering the expected results. Lastly, ensure ethical use.

By following these best practices, organizations can successfully integrate charge G B T bar and AI into their testing process and achieve significant benefits in terms of efficiency or accuracy and quality. Some of the industry-wide AI tools are listed out aware. You can see there's G P D four, which has been very recently released.

Uh, Bert GitHub, copilot code g BT with an extension to uh, vs. Code Watson assistant Amazon comprehend Mid Journey Dali, uh, two Dali three has now, uh, released, which is available with, uh, chat gbt, uh, plus and enterprise customers hugging face WI ai, TensorFlow, and Germany, which is very soon coming out by Google as a competitor to G B T four. So what's next?

Um, well serverless computing, chaos engineering, ml DevOps, low-code, no-code observability, GitHubs and the list is endless here. Um, certain key takeaways that I'd like to bring out from my session today. Um, the very first thing is DevOps is a great way to improve release cycles and streamline the product deliveries as per market and user requirements.

Um, follow five principle guidelines as we saw, uh, stated in the DevOps manifesto. And the main goal of dev test desktops is not to silo test DevOps, uh, but simply to raise the visibility of testers and testing as integral parts of the quality culture. Embracing changes and using them to drive innovation is the key to achieve, um, a successful dev test desktop strategy.

The future of testing with charge G B T Bard and AI is promising with many advancements and tends to keep an eye on. Um, you are some of the one 20 mindblowing AI tools. I'm sure you must have tried at least some of them.

And alongside that you can see 12 uh, factors or 12 use cases that I have listed out as future possibilities or applications of where we can use AI in testing. So they are advanced test automation with ai shift left testing with ai, predictive analysis for quality assurance, automated bug triaging and reporting, continuous testing and optimizations, natural language testing, scripts, enhanced security testing, AI ops for operations, data-driven testing and decision making, regulatory compliance testing, ethical AI testing and documentation. I'm sure most of you have this question now, will AI take my job away?

And to answer that is this image that I'd like to present. So if, if you, um, don't shift closer to the branch and make use of AI to assist you in your work, I think you might fall off soon. So here's a quote, um, by Steve Jobs and that's why I'd like to end my session today.

Technology is nothing. What's important is that you have faith in people, that they're basically good and smart and if you give them tools they'll do wonderful things with them. So yeah, uh, that's all.

Thank you so much folks. I'm happy and I hope all of your like the presentation. Thanks again.