Results for tag "teamcity"

6 Articles

Connecting ASP.NET MVC, MSBuild, CasperJS and TeamCity for user interface testing

0 Comments
Schakko

I must admit I have been ducking of implementing a working integration test suite in our current project for a long, long time. Some days ago I had to react on this fact as we needed to test some code of the  JavaScript frontend which had to interact with the ASP.NET backend through AJAX requests. The last time I tried to implement integration tests, I used Selenium which was some kind of … ugly so I decided to give CasperJS a try.

You can find the complete example in my Github repository: https://github.com/schakko/asp.net-msbuild-casperjs-teamcity.

Build and deploy your Xtext DSL with Maven Tycho

3 Comments
Schakko

A few days ago I finished my bachelor thesis with the title Integrating Xtext in an existing Software development process. I developed a domain specific language with Xtext and some extensions for easily adding new generators as new Eclipse plug-ins.

After finishing the thesis I added a Maven Tycho based build configuration. The build configuration can be easily deployed to TeamCity, Hudson or any other build server of your choice. I assume that you have already developed an Xtext based DSL and you know a little bit about OSGi and Maven.

The complete project source code can be found at https://github.com/schakko/xtext-plugin-with-maven-tycho

Preparations

First of all you should think about a sensible directory structure. I use the following organization

  • /plugins contains your Eclipse based plug-ins
  • /features contains the Eclipse features for bundling your plug-ins
  • /releng is used for the release engineering process and holds the project for the p2 update site
  • /pom.xml is the master POM for your complete project.

Directory structure of your DSL project

Set up the master pom.xml

Every pom.xml inside the subdirectories inherits from the master pom.xml. The configuration defines all modules of your project, sets the needed plug-in repositories and plug-ins. The most interesting part of the lifecycle configuration is, that the xtend-maven-plugin is not executed in generate-sources but generate-resources. This is needed as the MWE2 workflow of your DSL project has to be executed first. If you have both plug-ins inside the generate-sources phase the xtend-maven-plugin will be called first, resulting in a ClassNotFoundException as the DSL infrastructure is not generated yet. So the compilation of Xtend is moved to generate-resources which is executed after generate-sources phase. Another solution would be moving the xtend-maven-plugin to every child pom but this means duplicate configuration code.

Set up your DSL project

pom.xml

The pom.xml contains the plug-in for executing the MWE2 workflow. Notice that your <workflowDescriptor> must begin with src/ or you will receive the message Cannot create a resource for…. All the other stuff is straight forward. The packaging type eclipse-plugin is introduced by the tycho-maven-plugin.

build.properties

The file build.properties contains the source and binary folders. This is important, as tycho-maven-plugin uses this file as reference for compiling and packaging the sources. The xtend-gen directory must be added.

META-INF/MANIFEST.MF

It is important, that your Bundle-Version must end with the suffix .qualifier, e.g. 1.0.0.qualifier. The string is automatically replaced with the build date.

Set up your test project

Copy generated files

At first the two automatic generated files from src-gen (*InjectorProvider.java and *UiInjectorProvider.java) must be copied to the src folder if you want use the xtend-maven-plugin. The Xtend Maven plug-in does not allow (at least in version 2.2.1) any additional source pathes so that none class in src-gen can be found by xtend-maven-plugin. Remove the src-gen directory from your Eclipse classpath too.

pom.xml

The pom.xml for the test project uses a different Xtend version because I ran into heavy problems with the xtend-maven-plugin using the ValidatorHelper. Referencing the ValidatorHelper class results in a GenericSignatureFormatError. Sven Efftinge mentioned yesterday that the bug should be fixed in the latest snapshot.

The packaging type must be eclipse-test-plugin.

pom.xml of your test project

build.properties

Include the xtend-gen directory and make sure that the src-gen directory is not included because of the xtend-maven-plugin problem.

META-INF/MANIFEST.MF

The MANIFEST.MF uses the Import-Package statement for importing Mockito. See the next section how to set up this dependency.

Creating a dependency project

I used Mockito as mocking framework inside my tests. Adding this dependency is a little bit tricky:

  • Adding the mockito-all-1.9.5.jar to your Eclipse build path does not work. Maven does not know anything about your .classpath settings.
  • Adding a dependency to the pom.xml means that the compilation with Maven would work but not the installation on the client because there is no OSGi package for Mockito available.
  • Converting the mockito package to an OSGi package is not ideal. Every dependeny must be repackaged.

A much better solution is including the required JAR files inside an own OSGi package. I took the idea from the atlassian-connector project.

New Plug-in project

Create a new Eclipse plug-in project, create a folder lib and copy the  mockito-all-1.9.5.jar inside.

pom.xml

Nothing new, just use eclipse-plugin as packaging type.

build.properties

Your build.properties must include the lib directory.

META-INF/MANIFEST.MF

Here comes the interesting part. We use Bundle-ClassPath to define the additional path to the mockito JAR file. Every package inside this JAR file is re-exported via Export-Package.

Include the mockito framework in an OSGi bundle

Set up your UI project

pom.xml

The pom.xml must use the packaging type eclipse-plugin.

build.properties

I added the schema and icons directory and plugin.xml. This depends on your needs.

META-INF/MANIFEST.MF

Nothing new.

Set up your feature projects

Features are bundles which consist of different plug-ins. Lars Vogel made a good tutorial which you can use for more information

We need to create two features:

  • A runtime/core feature with your DSL and UI plug-in
  • A test feature with the test and dependencies plug-in

pom.xml

Create a pom.xml and use the packaging type eclipse-feature.

Configure the core feature

Set up your update site

The update site contains every feature the user can install. Lars wrote a tutorial about this but my needs were different. I wanted to deploy the content on a local webserver with a simple copy command.

To do so, generate a new update site project inside your releng directory and add a category definition (in Eclipse: Plug-in Development > Category Definition).

category.xml

The category.xml contains the features which are available through he update site. First of all, add a new category named “Xtext” and assign both of your previously created features to this category.

pom.xml

The packaging must be eclipse-repository. I use the copy-maven-plugin for copying the update site to the local webserver.

Configure the update site for your features

Run it local

With all these settings the project can be build inside Eclipse without m2e plug-in and on the command line. Go to the command line and execute mvn install:

Successful build on command line

Move it to your build server

Create  a new configuration in your TeamCity instance and add a Maven buildstep. I pass the -DupdateSite.target parameter for specifying the destination directory of the update site.

Configure the Maven build step in your TeamCity instance

Install your plug-ins

Enter the path to your update site and you will see your installable Xtext DSL

Install your Xtext DSL in Eclipse

TeamCity: Make the NuGet repository available without authentication

0 Comments
Schakko

Task for today was making the NuGet repository of TeamCity available in our local network. Sounds easier as it was as our TeamCity instance is available from the Internet but you can only access the rontend with valid domain credentials (LDAP/Active Directory authentication). Enabling the guest account feature in TeamCity would have mean a small security break in our concept.

Here is the success-story how to enable the NuGet repository for local accesss without authentication and without enabling the TeamCity guest account:

Creating a new proxy instance

One of our Apache servers got the order to act as reverse proxy for the NuGet repository. The reverse proxy was running inside an own virtual host (nuget.mydomain.local) and was registered in our local DNS.
The configuration looked like this:


<VirtualHost *:443>
 SSLEngine on
 SSLProxyEngine on

ServerAdmin admin@mydomain.local
 ServerName nuget.mydomain.local
 ServerAlias nuget.mydomain.de

LogLevel warn
 ErrorLog logs/nuget.mydomain.local-error_log
 CustomLog logs/nuget.mydomain.local-access_log combined

# Map other source to FeedService URL
ProxyPass / https://teamcity.mydomain.local:443/guestAuth/app/nuget/v1/FeedService.svc/
ProxyPassReverse / https://teamcity.mydomain.local:443/guestAuth/app/nuget/v1/FeedService.svc/

</VirtualHost>

Adding a domain proxy user

With the configuration above we still have the problem that there is no user who can access the repository without credentials. So we need to add a domain user, let me call it teamcity-nugetfeed-proxy-user. For security reasons this account has a very long random generated password and is only used for providing the NuGet feed.

Using the domain proxy user in our Apache configuration

We have to add a Basic Authorization header in our Apache configuration so that every access between the proxy and TeamCity is authenticated.


RequestHeader set Authorization "Basic <Base64(teamcity-nugetfeed-proxy-user:$password)>"

The phrase <Base64(teamcity-nugetfeed-proxy-user:$password)> must be replaced by an base64-encoded string with format $username:$password.

After restarting the Apache you should be able to use the NuGet stream in Visual Studio. Look at the Apache logs for occuring errors.

TeamCity is failing – NuGet return “File contains corrupted data”

Till yet you are able read the stream but TeamCity will fail downloading your repository files. The reason is easy: The FeedService.svc returns an XML file which contains absolute pathes mapping to teamcity.mydomain.local and not nuget.mydomain.local. So again we have no authorization and the error File contains corrupted data occurs because of the TeamCity login dialogue.

We have to rewrite the content of the FeedService.svc replacing every occurence of teamcity.mydomain… with nuget.mydomain…
At first I tried mod_proxy_html but in the end using mod_substitute was the only solution that worked.


# Rewrite FeedService URL
 Substitute "s|https://teamcity.mydomain.local/guestAuth/app/nuget/v1/FeedService.svc|https://nuget.mydomain.local|i"
# Rewrite repository access
Substitute "s|https://teamcity.mydomain.local/guestAuth/repository|https://nuget.mydomain.local/repository|i"

# Execute every substitution only for XML/HTML
 AddOutputFilterByType SUBSTITUTE text/html
 AddOutputFilterByType SUBSTITUTE text/xml
 AddOutputFilterByType SUBSTITUTE application/atom+xml

# Map local repository path to remote repository
 ProxyPass /repository https://teamcity.mydomain.local:443/guestAuth/repository
 ProxyPassReverse /repository https://teamcity.mydomain.local:443/guestAuth/repository

# Unset gziped accepted encoding
RequestHeader unset Accept-Encoding

After restarting everything should work fine.

The complete configuration is now


<VirtualHost *:443>
 SSLEngine on
 SSLProxyEngine on

ServerAdmin admin@mydomain.local
 ServerName nuget.mydomain.local
 ServerAlias nuget.mydomain.de

LogLevel warn
 ErrorLog logs/nuget.mydomain.local-error_log
 CustomLog logs/nuget.mydomain.local-access_log combined

# Rewrite FeedService URL
 Substitute "s|https://teamcity.mydomain.local/guestAuth/app/nuget/v1/FeedService.svc|https://nuget.mydomain.local|i"
 # Rewrite repository access
 Substitute "s|https://teamcity.mydomain.local/guestAuth/repository|https://nuget.mydomain.local/repository|i"

# Map local repository path to remote repository
 ProxyPass /repository https://teamcity.mydomain.local:443/guestAuth/repository
 ProxyPassReverse /repository https://teamcity.mydomain.local:443/guestAuth/repository
 # Map other source to FeedService URL
 ProxyPass / https://teamcity.mydomain.local:443/guestAuth/app/nuget/v1/FeedService.svc/
 ProxyPassReverse / https://teamcity.mydomain.local:443/guestAuth/app/nuget/v1/FeedService.svc/

# Execute every substitution only for XML/HTML
 AddOutputFilterByType SUBSTITUTE text/html
 AddOutputFilterByType SUBSTITUTE text/xml
 AddOutputFilterByType SUBSTITUTE application/atom+xml

RequestHeader set Authorization "Basic <Base64(teamcity-nugetfeed-proxy-user:$password)>"

 # Unset gziped accepted encoding
 RequestHeader unset Accept-Encoding
</VirtualHost>

NuGet and TeamCity: Repository for third party libraries

0 Comments
Schakko

Later this day I switched over to another task. TeamCity provides a fully functional NuGet repository server. My plan was migrating our own libraries and some 3rd party libraries from a direct dependency (binary file or as Git submodule) inside the project Git repositores to a system like Maven or Ivy or … well … NuGet for .NET.

First I set up a new Git repository containing all our 3rd party dependencies which are not available in the public NuGet repositories. The Git repository had the following structure

/.nuget
  /NuGet.config
  /NuGet.exe
  /NuGet.targets
/External.Lib.1
  External.Lib.1.nuspec
  /lib
    External.Lib.1.dl
/External.Lib.2
  External.Lib.2.nuspec
  /lib
    External.Lib.2.dl

The .nuget directory came from Visual Studio. I used the NuGet.exe binary for local testing.

In TeamCity I created a new Build Configuration and chose **/*.nupkg as Artifact Path.

The rest of the Build Step configuration can be seen on the attached image.

After running the first build I realized that the .nupkg had the wrong directory structure. The structure External.Lib.1.nupkg for example was

/External.Lib.1
  External.Lib.1.nuspec
  /lib
    External.Lib.1.dl

This is not the expected directory tree.

Reason for that is, that TeamCity automatically appends the parameter -BasePath to the NuGet.exe statement resulting in (taken from build log)

NuGet command: <tc>\buildAgent\tools\NuGet.CommandLine.2.0.40001.nupkg\tools\NuGet.exe pack <tc>\buildAgent\work\80f29a1b916940aa\External.Lib.1\External.Lib.1.nuspec -OutputDirectory <tc>\buildAgent\work\80f29a1b916940aa\dist -BasePath <tc>\buildAgent\work\80f29a1b916940aa -Verbose -Version 0.7 -Properties Configuration=Release

There is no frontend option to disable the setting of BasePath parameter. The configuration parameter Additional commandline arguments did not work as expected as it quotes the entire string.

Thanks to the  parameter Build Path > Properties we can do a little bit of quoting magic and overwrite the BasePath option

Configuration=Release " -BasePath "

Yes, there are two quotes as the usage of Properties puts the NuGet command itself in quotation marks.

The complete command resolves to

... -BasePath <tc>\buildAgent\work\80f29a1b916940aa -Verbose -Version 0.15 -Properties "Configuration=Release " -BasePath ""
and everything is fine 😉

Remove version signature from TeamCity, Confluence and JIRA

0 Comments
Schakko

If you want to make your TeamCity, Confuence or JIRA instance accessible from outside of your LAN, you should remove all version signatures so that no attacker can easily lookup for existing exploits.

  • TeamCity: Open <TeamCity installation dir>/webapps/ROOT/WEB-INF/tags/version.tag and remove the full content from this file. You must restart the TeamCity service afterwards.
  • Confluence: Open <Confluence installation dir>/confluence/decorators/includes/footer-content.vm and remove the line
    <li class=”print-only”>$action.getText(‘printed.by.atlassian.confluence’,[“$generalUtil.versionNumber”])</li>Restart of Confluence service/instance is required.
  • JIRA: Open <JIRA installation dir>/atlassian-jira/WEB-INF/classes/templates/plugins/footer/footer.vm and remove the line
    <span id=”footer-build-information”>(v${build.version}#${build.currentBuildNumber}${commitId}${partnerName})</span>

    Restart of JIRA service/instance is required.

Fehlerhafte Unittests in NUnit und xUnit von TeamCity nacheinander ausführen lassen

0 Comments
Schakko

Aus äh… historischen Gründen ist es so, dass eines unserer C#-Projekte sowohl NUnit als auch xUnit für die Unittests nutzt. Über Sinn und Zweck dieser Konfiguration brauchen wir nicht zu diskutieren – Fakt ist, dass sich das nicht auf die Schnelle ändern lässt.
Ich wollte nun dieses Projekt in TeamCity integrieren, so dass sowohl die Tests von NUnit als auch von xUnit ausgeführt werden. xUnit muss in TeamCity zwangsweise über ein MSBuild-Script integriert werden, was aber einfach zu bewerkstelligen ist. NUnit wird hingegen von TeamCity nativ unterstützt.
Mein erster Ansatz war nun, für den Buildprozess zwei Buildsteps anzulegen. Einmal für NUnit (nativ), einmal für xUnit (msbuild). Problem dabei: wenn einer der beiden Buildsteps fehl schlägt, wird der nächste Buildstep ebenfalls nicht ausgeführt. Das bedeutet also, dass man mit einem Buildlauf nicht die komplette Fehlerliste enthält, sondern immer nur die, des gerade laufenden Unittest-Frameworks (treten Fehler erst im zweiten Buildstep auf, ist dies natürlich egal).

“Schuld” an diesem Verhalten ist die Option Fail build if: at least one test failed. Ist diese nämlich aktiviert (Standardeinstellung), werden nachfolgende Buildsteps beim ersten auftretenden Fehler nicht mehr ausgeführt – was ja auch durchaus Sinn macht. Deaktiviert man die Option hingegen, werden alle Buildsteps ausgeführt, auch wenn in den Unittests Fehler aufgetreten sind. Problem dabei: Der Build wird auch bei aufgetretenen Fehlern in den Unittests grün mit “OK” markiert. Somit bekommt bei fehlgeschlagenen Unittests keiner eine Benachrichtigung über etwaige Fehler.

Ich entschloss mich zu folgendem Ansatz, der mit (N)Ant auf jeden Fall funktioniert hätte: Unser Test-Teilprojekt bekommt ein eigenes MSBuild-File, dass nur die Targets für das Ausführen der NUnit- bzw. xUnit-Tests enthält. Glücklicherweise gibt es von der .NET-Community bereits einige vorgefertigte Tasks, die das bewerkstelligen.

Zuerst müssen die MSBuild Community Tasks installiert werden (letztes Release). Dass die Dependencies zu den Unittest-Frameworks (xunit.runner.msbuild.dll) im Projektverzeichnis vorhanden sind, setze ich einfach mal voraus. Im Projekt mit den ganzen Unittests wird nun eine neue .msbuild-Datei mit folgendem Inhalt erstellt:

<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
	<UsingTask AssemblyFile="$(MSBuildProjectDirectory)..DependenciesLibsxunit.runner.msbuild.dll" TaskName="Xunit.Runner.MSBuild.xunit" />
	<Import Project="$(MSBuildExtensionsPath)MSBuildCommunityTasksMSBuild.Community.Tasks.Targets"/>
	<PropertyGroup>
		<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
	</PropertyGroup>
	<ItemGroup>
		<CompiledAssemblies Include="$(MSBuildProjectDirectory)bin$(Configuration)YourTestProject.dll" />
	</ItemGroup>

	<Target Name="Build">
		<MSBuild Projects="YourTestProject.csproj" Targets="Build" Properties="Configuration=$(Configuration)"></MSBuild>
	</Target>

	<Target Name="Test-xUnit">
 		<xunit assembly="@(CompiledAssemblies)"  ContinueOnError="true" />
	</Target>

	<Target Name="Test-NUnit">
		<NUnit Assemblies="@(CompiledAssemblies)" ContinueOnError="true" />
	</Target>

	<Target Name="TestOnly" DependsOnTargets="Test-xUnit;Test-NUnit">
	</Target>

	<Target Name="Test" DependsOnTargets="Build;Test-xUnit;Test-NUnit">
	</Target>
</Project>

Das gesamte Projekt ist so strukturiert, dass verschiedene Unterprojekte in die Solution integriert sind. Das Test-Projekt ist abhängig von den anderen Projekten. Diese Informationen liegen in YourTestProject.csproj und hängen natürlich von den Projektgegebenheiten ab.
Im UsingTask-Tag muss noch der korrekte Pfad zur xunit.runner.msbuild.dll hinterlegt werden.
Wenn nun das Target Test bzw. TestOnly aufgerufen wird, werden beide Unittesting-Frameworks aufgerufen. Wenn eines von beiden fehl schlägt, wird trotzdem weitergermacht (ContinueOnError=”true”).

Weiter geht es: Als erster bzw. zweiter Buildstep wird nun Test bzw. TestOnly als Target aufgerufen. Sieht dann so aus:

Sobald nun der Build ausgeführt wird, werden die Ergebnisse von xUnit und NUnit aggregiert und als ein Ergebnis dargestellt. Das bedeutet also, dass es keinen Unterschied mehr macht, aus welchem Framework die Tests stammen. Sollte nun ein Test fehlschlagen, werden alle anderen Tests trotzdem ausgeführt und es findet kein Abbruch statt. Mission accomplished!