Trista Pan – How to Build a Distributed and Secure Database Ecosystem With PostgreSQL

you Hello everyone. This is Trista. So I'm so happy to be here to give this talk around your postgresql around how to build a distributed and secure database ecosystem around your existing postgresql.

So there is the Samsung about myself. I'm the Trista Pan the Sofia Yax co-founder and the CTO. So it's just my job title my professional area.

It's around the distributed database around the database am management of platform developing and also about open source. So now I'm the Apache member and incubator Mentor. So sometimes I will give some the tips and statins to other incubator for jazz to help them to round your open sales Community to have a successful and diverse Community.

Sometimes I will post some articles around the my professional area around the open source and the open source commercial stuff on my Twitter or linking if you're interested in such topics topics. Welcome your look at that the channels Before a way started today's and answer. I mean give more introduction about and how to create this distributed and secure database around your postgresql.

It just allowed to give some introduction around your background, especially around new needs or new requirements about or existing database and cluster. Maybe they're a postgrass. Well, maybe they're a MySQL cluster So currently we are and are going the Big Data era because big data will bring some more new features where new requirements around the word traditional dbms for them all the scalability or Elias take a stealing out or Skilling and also about a high performance even though you were database needed to help users to manage all the tremendous data you also Application the developers also want your database to give a quick answer about their question and that's part about the high availability because our users cannot taller than to like the one word three hours and crashed it down.

Right the last part also about how to help or developers were companies to manage the tremendous database cluster or instance though. That's all the new needs for the from the big data for your existing database cluster. That's the first liar the second layer.

It's around the cloud because wow, we totally change your delivery and deploy Master approaches and also a totally change your infrastructure, maybe your application or your database or other stuff and it's our deployed on As or on cloud at this time or this point, you can consider that how to become the cloud native how to make your database or make your application your services become multi-cloud so that you don't need to be locked in one Cloud platform or when Venture so because of the big data and the cloud it totally changed or it gave us the new requirements around or existing database cluster your math curl or Oracle. We're sicko see where or postgrass Last part let us come back to the internal database because if we want to answer my title my question how to we need to consider that the factors or the elements of your database system traditionally your database and the consist of the two part the first part it's in Computing liar. I mean this one because we need to consider that's and what's the function of your database and first it can help you give your answer for your question.

That means Computing right a second part it can help you store and manage perceived your data full ever. So that's about this one storage liar. So the database is made of the Computing layer and storage layer for A monolithic database for example Oracle or civil server their Computing layer and story layer are deployed either in the same machine or some same the silver.

So that's what we call it a traditional databases or the monolithic databases, but currently regularly found that all your applications become elastic they have they are splitted in different small units like the macro Services, right? So, how about your database or database currently, it also needs to consider assuring architecture or Distributing architecture because that's the shirting actor can help you make your database be calm elastic make it an automatically and elastic take away skew out and skewing and also can made this and these three Beauty. System help you manage tremendous enormous data and give your answer so quickly.

So that's the magic of the distributed and feature. And because we have the Computing layer and slowly layer then we also need to provide a standard. Interaction interface for or end user a traditionally there are ocal sico.

It's the standard language or standard interaction phase interface for any other YouTube or any other and or databases also currently we found that and some of the cloud database they will provide the services or API to your database to your application. That's how could that's fine. You can pick up a one of them.

It's your scenarios and ask for us about infrastructure because you're the best living the cloud or just on the premise. It's your choice. So the NASCAR isn't lightest and entering to solution how to that's question the beginning maybe you can make your MySQL or postgrass or works well because well support your application your traffic will requirements.

But on this timeline at this point, we have the new phenomenon. We have the new requirements about your database then when you consider how to migrate or upgrade all the this tea database cluster be calm the distribute system to elaborate all the new needs from Big Data and Cloud today. I will give efficient to solution for this question because to answer this question.

There are so many answers you can choose the database as the service you can. Is a new SQL database or no SQL database? Ever since okay, but you can see there how to make it become efficient quicker at a low cost right make all the migration procise or upgrading process become fluent.

So that's the magical or benefit from my today's solution first. It can help you leverage the existing basis and because we can't avoid that phenomenon the truth that you were production environment. Alrighty fall of the different traditional dbms.

It works. Well, they they are function is so wonderful. So amazing.

So how to memorize them at this time at this point a second as can upgrade our grade your existing database become distribute to the one our low cost the third one. Is it help you to solve the database moving in Cloud issue you need Seether the mounty cloud or Cloud native and Nas part. It's about our Mount cloud and no locking issue.

Okay, so that's the benefit from today's solution. To risk that go. We need to leverage open source Community or project named Apache sharing Sofia.

We can have the first glance at his slogan were the definition. What's the shortness of fear sharing Sofia? It's a ecosystem to transfer Annie traditional database into a distributed database system and also enhances with your shirting elastic schooling and encryption features and more.

So just to give a brave glass out of this solution. You will know that it can really help us to reach it out go that to reach that all the benefits and a transfer your database into a distributed ones. So that's the definition a second question.

We need to consider. That's why should I use this and first it can help us solve the issues, but another question that we need. Consider that it's a mature community.

Or it's a it's have the active community. That means if I have some questions or have issues we can get the help. We can get the help from this community and they can help me to solve some fix or some issues and they can listen to my voice about this project.

And if I'm interested in this open source Community I can join beer to do some contribution. That's the open source community. So from the statistics on the GitHub, we can see here it receive more than 70,000 stars and say salmon forks, and also it has more than four for hand for Android and contributes here.

This this project has been released for nearly 50 times. So from all this data that can tell or users that it's okay. It looks healthy for me to consider is for jacked and then we can see that this project also Prevail a lot of Manus or documents can help us to Side Up This solution as quick as possible, right?

So that's the first issue. That's yeah, it looks healthy. We can't consider it the last person how how to leverage or existing database and do the transformation.

So I guess at the side before we need to consider. What's the database database consist of two part can building there and slowly layer, but for the monolithic database they merge them to gather in the one part or one unit, but for the distributed database system, you can see here. We can leverage your existing Oracle or mass ql or pills grass Crown as the story to layer or storage note of this new.

distributed database system The Nest part the shorty and Sophia will works as the Computing nodes of this distributed database system. Okay, so it looks possible for us to adopt this solution because we have the story layer to help manage the data persist the data we can have the global Computing layer to help us to manage all the post graphs of instance all of the storage notes and help us to prevent where to enhance the Basin database system with the shirting feature with the Distributing the architecture with the data encryption or austication authority the ETC a lot of useful and enhance the features for this distribute system. So shorty not just help you to transfer this and monolithic database into distribute the one it's also can help you.

Enhance this distribute system with a lot of data management features, right? So here we look at as a distributed database and insurance severe when when to receive users question or statement will first the sequel random sequel become the internal objects and optimize your SQL requirements and found the target post-round instance send that to different instance and and then quite the local community read out murder the readout become the final one or Global one then return where any user but all the precise is transparent or any other or your application. So that's magical or functions of showings of fear.

so let us give them more details about joining Sofia or about this solution currently shortensify provides two two like the clients for you to choose first one is turning should be gdbc. It's a lightweight gdpc driver. So if you just want to have the lightweight driver to help you up to upgrade your database, then you can pick up it and it also have the high performance because you can see here compared to your shooting proxy shorting process.

You need to do independently deployed in independent receiver. So there is no Aster visiting or no and actor TPC TVC Vivid, right? So that means it just directly send the equipment do some local Computing computers and help you manage the distributed them stories.

Power but shortness of your proxy it also amazing because also I have to say that it has no so high performance compared to this one, but it can pretend it as a database solver. So if you were storage notes, I mean if you use the postgres or database shortages of the process will pretend it as a distributed postgresql silver. So you don't need to change it any connection approaches with your existing post-grass or database.

You're just to change the IP and the sort of so reasoning to shirt and Sophia crossy and then you can finish all the transfer transformation. So that's the magical shortens the approxy you can it can works or act it as the database and silver. So no matter your choose the shortness of the jdbc or proxy actually, they share the same features.

For example, first one data Surety and elastic skewing and skew out based on your current needs. Right and as part is your distributed transaction and read restylating Because I guess most of your postgraph instance will have the rapidly right. So starting Sophia can help you leverage or man the rapidica and the rebound balance all their traffic among different rapidly because and your family notes.

So now for the global authentication and data encryption data encryption that means Because like like take you there for example if we have the user table and we need to like to store the telephone number of a user. So that's all the prophecy right? We cannot put all the Privacy into your Storage nodes I mean your database and sure needs to feel can help you automatically to encrypt your data from application and store or put send the cipher tax into this database and system.

And then if you're application, why don't you get the plane tax then your Sofia prosely. We're automatically decrypt The Cypher text and sand and plain tax to your application. So all those things is transparent to your and user that can help it reach that the data encryption feature or that goal.

So the oh, I have known time to give more introduction about the internal precise how to do the parts or optimized or router SQL then best of this and internal basic precise. You can have the such features you can help use Shirley's feel to help you manage an average grade your existing postgrad clusters, right? So, Nas part, I have to say that for your existing like the architecture you don't need to do much more changes around them.

We just want to make this transformation become fluent and efficient with the lowest cost. So you can just deploying certain severe proxy cluster and import this layer between your application and your databases then you can finish the upgrading. Yeah, that's so easy.

And also another benefit from this solution is that It can help you elastically skew in skill out your storage nodes and Computing notes and separately how to understand a sentence if you found currently you want to more computing power. Then you don't need to change or don't you do any modification around your health scale instance. You just need to deploy more stainless Computing notes, that means assuring severe cluster.

Then you can have more computing power button. If you don't care about computing power you found that you need more compute storage nodes or your database instance to help you store or persist the data, then you don't need to care about the starting Sophia Closter. You just need to pull up a more a more your posts.

Well database or instance then you can finish A disco that means you want more storage power. Yeah. So to make the skewing and skill out independently as separately is amazing.

All right. So next part is about the distribute sequel if we want to use solution that we found by usuring to feel with upgrade this system, but now another question that and how do we use this features and at the beginning if you don't import the shortness to be you can use seaco to help YouTube create a table create a TV job table author table. That's okay.

But now if we want to leverage such a features when it's another sequel longer language sequel like language to help use this features, that means distribute to the sequel distribute the sequel here and give them all if you think of this the definition the so commonly located and just to give the lookout example You use SQL to create a table, right? But here you use a district of SQL to create a sharing table or author shirting table or job are starting table. That means you use it to you this and use district with the SQL to use the shirting feature.

Right? Actually. This is the language and on the document you found a different to top of the distribel to help you with different features.

So we have a lot of the knowledge around this solution that NASCAR about and let's do some demo tasks. In today's test. I will give the three part to show you the first one about how to do the deployment the second one how to test the high availability of the system the NASA.

The last one is how to use distribute SQL to create your distributed security device. That means that will help me to answer the question how to that's my title. Yeah.

So the the first the first part the first part it's about deployment deployment here on this repo sharing student crowd. We provide a shirt in your charts on the helmur ripo. So your users or administrators just to use the how command to to use the shortness of their towers to deploy all the the elements or factors of this and system on your kubernetes and platform here you can just Still use the shooting Sophia charts today.

I use the shortness to feed chars to use the helm comment to deploy a distribute system. That means we can have two students to be a proxy instance or server and one or two shortlisted the operator that can help us to guarantee the habitability of the shortness of the proxy or instance and then governance for for this distributed database system to store a lot of global mental data or finish the synchronization processing. So by youth sharing severe charts, you can deploy this Computing layer, but you also need the story layer today.

I use the postgresql chart to deploy to post real instance to act to make it work as the storage notes if you're right have your post Our instance then you can just ignore this part just use certain Institute charge to deploy this Computing layer. So Computing layer Plus Storage layer, that means your application can visit this distributed database system so that we finish the first step that the deployment. The nas bars we want to use the shortness of the operator to test the habitability.

So half of this it means here for example, wait, when we do some modification our stack on this configuration here enable become the true and the Ming instance from one to two then you found at the beginning. I have the story shorty Sofia prostate in this cluster and after that it skewing become the two instance become here. I sign this mean instance true, right?

So if you side is one, then you will get to only one shortly in Sofia proxy, right so that can plus plus we it starts to be operator. It can test the activity actiness of your shortness of your prostate if they found one of the part to crash it down it will A new one so you don't need to care too much about that the high availability. All the things is automatically you can just give the number here it will guarantee.

That's now there are three or two or one acting and proxy working while alright, so the naspers we need to use the distribute C code to help you create this and database right distribute secured database first, when we finish the test and the deployment then we can just logging this local shirting. Sophia proxy when you log in you found this combined where this CLI as so same. We're familiar with your well with your existing post-reli, right?

So all the familiar for you and that's part light us to Trade this architecture here. I give this test for the that is the shortness of your proxy. That's where database your application view this part at the home complete database.

So for the application or from the application view it will sync that this over just have a one table logic table T user but actually this to you there is native for Summit tables or after tables and their existing different posts. Well instance, this one is one and two of them and for this in table columns, you can see here the telephone number for the application. They think there is the only telephone number and they sing they start all the telephone number data are playing tax for day to use for application to use but sharing Sofia just to do the Automatically decryption and in encryption.

So for your post-real instance for the persistence and node, you have found all of the data here that all the cipher or safer tax. So you don't worry about your debit your user or password link because all the data has been encrypted already, right? So we use the distribute to create this feature and also to create the sharding Drew shirting table because here we found we have the two sub-tables so you can see here shortly on count is the four right and that's we need to tell or shortening Sofia that we just need to increase and decrease the data on this column of this table.

Not other tape just the days what so you can see here the column is TL and this table is to Sugar and we use the AES algorithm to help us to different and encrypt automatically then create a table insert some test data and then you will found you can select some the data from proxy. You see the plane tasks or telephone, but when you log in your postgresql, you found all the data become the sever text, right? So that's the magic word function about this solution.

I hope you enjoy this process and enjoy the solution. And that's all about today's talk and thank you. So if you have any questions, you can just contact me or ask me on my Twitter or on my linking.

And also we already have a book to give more introduction and Sophie and the currently if you want to get the giveaway or events have the copy. You can just detach your needs to be account on Twitter. Then we will give you a copy about this book.

So since everyone for this listening, bye.