I am currently working on the migration of our time tracking system from Microsoft SQL Server/.NET to Java. Most of the logic resides in Stored Procedures and Stored Functions inside the database schema. Because of some reasons (testability, maintainability, migration from MSSQL to PostgreSQL in a far future) the whole logic must be converted to Java. As the system has a high criticality all of the end user applications must be running parallel. There are 4 tools in total, written in different languages: an old PHP web application, a bridge from JIRA to our time tracking system, another JIRA-to-time-tracker converter and the original C#/.NET fat client. All systems will be migrated bit by bit to the new Spring Boot web application.
Using Liquibase for database versioning
After collecting information about the current application environment I noticed that there were no database versioning system in use. The installation of the MSSQL schema was a pain: there were a lot of plain SQL files which had to be executed by hand. Since Java was the target programming language I decided to use Liquibase and moved the whole SQL scripts into a new Git repository, added the pom.xml and wrote a self-explaining Readme.md how the .NET developers had to use Liquibase.
Running Liquibase migration with the command line Maven plug-in
I decided to describe only the pure Maven approach and not the Liquibase installation. The execution of the Liquibase migrations are trivial and no dependencies had to be installed by hand. The pom.xml contained the following definitions:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>my.package</groupId> <artifactId>database-schema</artifactId> <version>1.0.0</version> <packaging>jar</packaging> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <!-- from internal Artifactory; Microsoft does not make the sqljdbc4.jar available in official repositories --> <groupId>com.microsoft.sqlserver</groupId> <artifactId>sqljdbc4</artifactId> <version>4.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-resources-plugin</artifactId> <version>2.7</version> <configuration> <encoding>UTF-8</encoding> </configuration> </plugin> <plugin> <groupId>org.liquibase</groupId> <artifactId>liquibase-maven-plugin</artifactId> <version>3.0.5</version> <configuration> <propertyFile>src/main/resources/liquibase/liquibase.properties</propertyFile> </configuration> <executions> <execution> <goals> <goal>update</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>
The liquibase.properties file did only contain default values and is not important. With the configuration above I were able migrate the MS SQL schema with
mvn -Dliquibase.url=jdbc:sqlserver://$HOST:1433;databaseName=$DATABASE-Dliquibase.username=$USERNAME -Dliquibase.password=$PASSWORD liquibase:update
At this point I could update the database schema by hand. This was necessary when someone had to develop inside a .NET environment or we had to migrate a staging or production database.
Making the schema available for developer environments
I realized quickly that the approach did not work really well for my Java development environment. A lot of database migrations had to be developed and the integration test environments should be automatically in sync with the defined migrations. My idea was to let Jenkins push the Liquibase defined schema as a Maven JAR into our internal Artifactory. I should be able to include the JAR as a normal Maven dependency and let Spring’s Liquibase integration to execute the latest migrations.
Reference a db-changelog.xml inside a JAR in your application.properties
I took a look in LiquibaseProperties and saw that the changeLog attribute supports the resource syntax. All I had to do was defining the db-changelog.xml by adding the following line to the application.properties:
liquibase.change-log=classpath:/liquibase/db-changelog.my-app.xml
Please note that I changed the filename from db-changelog.xml to db-changelog.my-app.xml. This should prevent ordering issues if there is already another XML file present with the same file name. The classpath prefix is used by Spring to scan all JARs in the classpath for the requested path.
Do not use the full path name of included SQL files
As I mentioned above all SQL statements resided in their corresponding SQL files. I used the following definition in the db-changelog.my-app.xml to include the SQL files:
<changeSet id="5" author="ckl"> <comment>Bla</comment> <sqlFile dbms="mssql" encoding="utf8" path="install/20151126-005_sp_calculate_worklogs.sql" relativeToChangelogFile="true" splitStatements="true" /> </changeSet>
This worked if Liquibase was either executed only through the Maven command line or as a Maven JAR dependency, but not both.
How Liquibase calculate database migration differentials
Liquibase iterates through all changesets defined in your db-changelog.xml. The XML attributes id, author and path are used at first to check whether this migration already exists in the database table DATABASECHANGELOG. If a row with the given parameters does exist, the checksum of the SQL file is calculated by normalizing the SQL file content (replacing new lines and so on). After that a MD5 checksum is generated by using the header and the content of the file.
The content of the “path” attribute differs
When executing mvn … liquibase:update inside the Git repository, the column path is filled with src/main/resources/liqiuibase/install/20151126-005_sp_calculate_worklogs.sql. Executing the migrations during the Spring startup process will result in a value classpath:/liquibase/install/20151126-005_sp_calculate_worklogs.sql for the path columns.
This means that every migration will be executed again, resulting in DDL errors.
Ignoring the path attribute
The easiest way was to use the attribute logicalFilePath in my databaseChangeLog tag. This forces all rows to have the same value of the path column:
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd
http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd"
logicalFilePath="path-ignored">
....
</databaseChangeLog>
In the DATABASECHANGELOG table the column path is filled with path-ignored.
Do not mix up different Liquibase versions
After I had fixed the previous error Liquibase showed that the calculated checksum of the files differed. At first I thought I had some encoding issues and forced everything to UTF-8 but the error was still there. It took a while until I noticed that the Maven dependency liquibase-core in the Spring Boot app and the Maven Liquibase plugin for command line execution had different versions (3.3.2 versus 3.0.5). Both versions calculates the MD5 checksum in different ways. The checksum inside the DATABASECHANGELOG table differed with the newly calculated checksum. All I had to do was changing the Liquibase Maven plug-in to use the same version:
<build> <plugins> <plugin> <groupId>org.liquibase</groupId> <artifactId>liquibase-maven-plugin</artifactId> <!-- same version --> <version>3.3.2</version> <configuration> <propertyFile>src/main/resources/liquibase/liquibase.properties</propertyFile> </configuration> <executions> <execution> <goals> <goal>update</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
TL:DR
I moved the definition of a Microsoft SQL Server database schema into its own repository, made the schema migratable with help of Liquibase and made it executable in standalone/command line mode and as a Maven JAR dependency.
