Embrace DevOps as a Database Administrator – Build container images with latest code release

In the previous posts of this series, automating code release and putting T-SQL code under source control was discussed. In this post, we will discuss about how the latest code release can be put on the containers directly and leverage the same image to build Dev/Test/Prod environments.

The flow of this post will be :

1. Spin a SQL container on Linux

2. Restore a database on the container

3. Run SQLPackage.exe to automate the code release to container

4. Build an image with all the changes

5. Spin containers with the latest changes for dev/test/prod

Note : – I presume, you have an understanding about containers. If not, please check this post before reading further.

In the last post, we discussed about SQLPackage.exe to automate the code release.

image

and the input parameters for this exe were as follows:

Script:

“C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\sqlpackage.exe”

/action:Publish

/sourceFile:”C:\Users\harshch\source\repos\DBcouncil\DBcounil\bin\Debug\DBcouncil.dacpac”

/targetconnectionstring:”Data Source=localhost; Initial Catalog=WideWorldImporters; Integrated Security=true;”  << target connection string will point to the container

1) Let’s look at the container environment, we have got here.  Container was spun using the following commands and we have a container instance running with a DB:

docker run -e “ACCEPT_EULA=Y” -e “MSSQL_SA_PASSWORD=test123@” -e “MSSQL_PID=Developer” –cap-add SYS_PTRACE -p 1401:1433 microsoft/mssql-server-linux

image

2) Let’s copy the backup file to the container:

image

3) After restoring the database to the container instance , let’s run the schema comparison tool to identify the delta of changes:

image

4) Same set of changes are identified:

image

5) Let’s use SQLpackage.exe to deploy the change to container:

“C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\sqlpackage.exe”

/action:Publish

/sourceFile:”C:\Users\harshch\source\repos\DBcouncil\DBcounil\bin\Debug\DBcouncil.dacpac”

/targetconnectionstring:”Data Source=localhost,1404; Initial Catalog=WideWorldImporters; Integrated Security=false; user id=sa ;password=test123@”   << target connection string pointing to the container>>

image

6) Let’s connect to SQL instance built on container and see if the changes have been deployed:

image

7) Now let’s commit these changes to the container image so that, next container we spin has all these changes inbuilt. Again, this entire lifecycle can be automated through powershell or windows scripts. Moreover, this script can be part of build process in TFS for automating the application release.

Syntax to commit the changes to image is  – docker commit < containerID > < newcustomimagename >  (mentioned in step 2 in the below screenshot)

image

if you see the size of image named SQL_2017_release1 is higher than the other images because the database has been restored in this image.

9) Now, let’s spin up a container from this image and see if we have the database already created:

c:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin>docker run -e “ACCEPT_EULA=Y” -e “MSSQL_SA_PASSWORD=test123@” -e “MSSQL_PID=Developer” –cap-add SYS_PTRACE -p 1405:1433 -v  -d sql_2017_release1

image

10)  Let’s connect to the new container SQL instance and see if we have the database created with all the changes:

image

We can see all the deployed changes were successfully ported to the new container. This new image could be used to build dev/test/Prod environments. In the next post, we will discuss about how to orchestrate containers using Kubernetes on Azure.

HTH!

Advertisements

Embrace DevOps as a Database Administrator- automate the T-SQL code releases

In continuation to the series of posts on DevOps for a DBA, lets learn how the SQL code release can be automated. In the previous post, we discussed on importing the schema of the database in SQL Server Data Tools and putting  T-SQL code under source control.

Historically, all the releases have been manually executed on the production server. Let’s see how using SSDT this can be optimized. There are two ways to do that:

1. Leverage SQL Server Schema Comparison tool –> Manual

2. Leverage SQLPackage.exe  -> Automated

1. Leverage SQL Server Schema Comparison tool –  Let’s see how this tool can be used:

1) Click on New Schema Comparison tool:

image

2) Select the source of the comparison, in this case it’s the project where the changes have been made:

image

3) Target is going to be the database where the code needs to be deployed:

image

4) Lets click on Compare and see the results:

image

If you see the above output, it clearly shows the action which is to remove old column named Sales.Invoices.BillTOCustomerID and add Sales.Invoices.OrderID1 and subsequently change the procedure where that table is being used.

4) Moreover, you can also control the behavior of the deployment as follows:

image

So far, we have identified the delta of changes to be applied and now let’s deploy the change to the actual database. One way to apply the changes is to click on the update and make the changes happen:

image

5) When you click on yes, the changes will be applied to the target database:

image

6) Let’s get into the database on SSMS and see if the change has reflected there :

image

DevOps is all about automation and the above method has lots of manual intervention. Let’s discuss about SQL Package.exe and how it can help to automate the entire code release process.

1) Let’s locate SQLPackage.exe:

image

2) Let’s check the parameters of this exe:

image

3) In our case we will use the following command to publish all the changes automatically:

“C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\sqlpackage.exe
/action:Publish
/sourceFile: Location of the DACPAC file which will contain all the changes made to the db along with the entire script of the database
/targetconnectionstring : Target instance where all the changes need to be applied or need a fresh DB with all the changes

Location of the DACPAC file can be found under:

image

4) In our case, let’s apply the changes to the target database through script which can be automated using windows or through TFS build process :

Script:

“C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\sqlpackage.exe”

/action:Publish

/sourceFile:”C:\Users\harshch\source\repos\DBcouncil\DBcounil\bin\Debug\DBcouncil.dacpac”

/targetconnectionstring:”Data Source=localhost; Initial Catalog=WideWorldImporters; Integrated Security=true;”

image

5) Let’s check the changes in the SSMS:

image

This script can be put in TFS build or automate in windows scheduler to pull all the changes periodically and apply those to the test/dev/prod servers. Moreover, to create master tables in the build process, script items could be added as follows:

image

In the next posts, we will discuss on

1: How to deploy these changes directly to container

2. How to update the image with the latest changes

3. How to deploy the containers with the latest changes

4. How to manage containers with Azure Container Service

 

HTH!

Embrace DevOps as a Database Administrator

I have been writing posts related to embracing new trends like Cloud/NoSQL/Business analytics etc. Currently, I have been speaking about DevOps and containers in various events and this will be a great learning to share.

I think this is a new evolution for software development industry which everyone is embracing with open arms. The agility and speed it provides to manage the software code and deployments, has been really impressive. Being a data guy, I was figuring out a way to join this bandwagon. Finally, I got something interesting on how as a Database professional , one can be part of this. DevOps is anyways a big change but I will just talk about how a database guy can contribute.

Generally, DBAs have to do lots of code release on daily basis and move around lots of changes from dev to test, staging or pre-prod and then production server. Generally, DBAs struggle to keep track of all the scripts and it takes up 20-30% of their working hours throughout. In this series of posts, I will explain how this can be optimized and how as a DBA, the practices like Source control can be helpful and adding containers to this can make the entire process really simple and easy to manage. While writing this post, I have presumed that the reader has understanding of DevOps concept.

 

In this post, let’s talk about a free of cost tool , SQL Server Data Tools(SSDT) which can be downloaded from here. Let’s see how we can play around with this tool and how it can help in code release automation .

 

Open Visual studio and create New SQL Server Database project and click OK:

clip_image001

 

Right click on the solution name and then click import to add the database:

image

 

Click on select connection and then browse to the right instance and the database:

image

 

When the DB is connected, click on import to import all the DB schemas

image

 

Now if you see the solution explorer, you will find all the schemas and the objects related to the DB:

image

 

Let’s add this solution to the source control and in this case I connect to Visual studio team services online account:

image

 

In this case, I will connect to Database_Migration project – You first need to have your account here or you want connect to the local TFS server (Connect with your development team on this):

image

 

After connecting to the TFS server, let’s check the source code by clicking on “Add Solution to source control under source control option”:

image

 

Create the project name, in this case, it’s DBcouncil and click OK :

 

image

 

When the project is created, go back to visual studio project and check in the code:

image

 

Now let’s see how the code looks like on TFS. In this case we have used Visual studio team services i.e. TFS online account. let’s open the link to access the online account of team services. However, even On-premise TFS or Git and other source control tools are also supported:

 

image

 

when you login into this account, you will be routed to this dashboard:

image

 

Click on Database_Migration where our project was created and then click on code to “DBCouncil” project:

 

image

 

Here you can see all the schemas and their objects:

 

image

 

Now let’s make some code changes and see how the changes could be tracked. In this case, we have renamed the location column for cities table and it also shows where this change will impact e.g. if there is any reference in stored procedure or the other objects.  Here it shows the reference of this column in stored procedure GetCityUpdates – if we click on apply, the references will also be updated.

 

image

 

Now , lets checking the changes here and see on the TFS online portal:

 

image

 

If you see the changeset for DBCouncil project, you will be able to see:

 

image

 

it saves you a hassle of checking the history of changes manually. I have seen DBAs keeping all the change scripts in either emails or a shared folder. Instead, if that can be put under source control, you can find out changes like this:

image

 

All these changes are still in the project in SSDT, these changes are yet to be deployed to the database.

In the next blog posts, I will share

1. How to automate the code release process

2. Integration of the DBs with containers on Windows and Linux

3. How containers can help to setup the dev/test/prod environments within seconds

4. How to manage these containers with Azure Container Service – Kubernetes

 

HTH!