789 posts

w32tm: “No valid response has been received from manually configured peer”

This was one of those network bugs which took some time to fix. To give you a short background: In the last weeks we had random time NTP drifts in our Linux infrastructure, nothing serious but it was odd as they disappeared after restarting the ntpd or chrony service. During the weekend my co-worker and I were upgrading some parts of the network infrastructure (new cables, new cabling between server and switches, upgrade to the latest Sophos UTM firmware, upgrading to XenServer 7.0 and so on). After getting everything up our OMD showed a lot of NTP errors which led to problems with FreeIPA, Kerberos and SMB. Besides that, on some of the virtual machines we had NFS problems. My co-worker dealt with this problems. Parallel to that and completely unrelated I was preparing a new Active Directory domain and wanted to setup the w32tm service on Windows Server 2012 R2.

To my suprise I was not able to synchronize the time with our firewall. Using

w32tm /stripchart /computer:$FW_IP /samples:10 /dataonly

worked flawlessly but when i tried

w32tm /resync /discover

I received the error

Der Computer wurde nicht synchronisiert, da keine Zeitdaten verfügbar waren.
(The computer did not resync because no time data was available)

With w32tm debug logging enabled I got more information:

151915 13:50:46.8092075s - Logging information: NtpClient has not received response from server $FW_IP (ntp.m|0x0|>$FW_IP:123).
151915 13:50:46.8092075s - Logging information: NtpClient: No response has been received from manual peer $FW_IP (ntp.m|0x0|>$FW_IP:123) after 8 attempts to contact it. This peer will be discarded as a time source and NtpClient will attempt to discover a new peer from which to synchronize. Error code: 0x0000005C
151915 13:50:46.8092075s - Reachability:  removing peer $FW_IP (ntp.m|0x0|>$FW_IP:123).  LAST PEER IN GROUP!
151915 13:50:46.8092075s - AddNewPendingPeer: manual
151915 13:50:46.8092075s - PeerPollingThread: PeerListUpdated
151915 13:50:46.8092075s - Logging error: NtpClient has been configured to acquire time from one or more time sources, however none of the sources are currently accessible and no attempt to contact a source will be made for 15 minutes. NTPCLIENT HAS NO SOURCE OF ACCURATE TIME.

At first I suspected a bug in the latest Sophos UTM firmware, so I checked the incoming packets on the firewall:

# NTP runs on UDP/123
tcpdump -vvv -i eth5 udp and port 123

Running w32tm /stripchart again I received the expected packets. Running w32tm /resync /discover showed nothing. My guess was that the UTM was not responsible for the error. I took a look into the changelog of XenServer 7.0 if anything had changed with the virtual interfaces of the virtual machines. The changelog itself did not have any interesting points, so it had to be an error on the lower network layers. Do you remember that I mentioned the recabling of server and switches? Well, some months ago one of our Netgear switches literally vaporized (btw, I wlll never buy a Netgear again) and had been replaced with a Cisco SG220-50. The SG220 is a nice piece of hardware and does not only work on layer 2 as the old Netgear but also on layer 3. After checking all available options we stumbled upon the settings in Security > Denial of Service > Security Suite Settings. The settings UDP Blat and TCP Blat caught our eyes: if enabled, the Cisco drops any UDP or TCP traffic where source port equals the destination port. Disabling both solved our NTP and NFS errors immediatly. By the way, w32tm /stripchart (and according to this ntpq in debug mode and with an unpriviliged source UDP port) ran flawlessly because it does not use source port 123 but another one.

Side project: A tool for preparing and calculating quotes with Neo4j

One part of my work is to inspect request for quotes (RfQs), gather customer requirements, estimate the effort and prepare a written offer. Most of our customers do not want to pay a separate bill for every sprint or user story but get a number or range for the total costs of the project. A big problem is the complexity of our projects. We do not only do software development but also do user interface design and application hosting.

In the last ten years I have tried and seen a lot of methods to calculate the estimated costs of software development projects: Function Point Analysis, COCOMO, Microsoft Excel, using modificators considering project management, risk and other factors, using tool combinations like JIRA or Confluence. In the end every method I have tried has disadvantages in one way or another.

This being said, one year ago I started to gather my own requirements for a cost calculation tool which fits in my workflow and should reduce my work:

  • I need a user interface to capture tasks (user stories, subtasks) and required material (server, licenses, hosting costs, etc.). Each of these items can be valued (e.g. effort in hours, material cost in Euro etc.).
  • Each of the items can be grouped together to form components. Each component can contain components itself.
  • Changing the cost or effort of any item should result into a recalculation of parent components and the whole offer.
  • I do not want to contaminate JIRA with estimations. JIRA is a project management tool and not a tool for estimating the cost of projects. Portfolio for JIRA does also not match these requirements.
  • After I have structured the requirements and estimated their costs/efforts I want to export the offer in a distributable format (PDF, XML, …).
  • After the customer has agreed to order components/tasks of our offer, I want to export these items from the calculation tool into JIRA.

The ideal workflow would be:

  • The customer gets in touch with us, we receive the specifications.
  • I break down the specifications into components and derive the tasks and the required material.
  • Our team estimates the effort.
  • Me and my superior are defining the quality levels of each tasks and the costs. The total costs are automatically calculated.
  • We are finalizing the offer and sending the link of it to the customer.
  • The customer selects the components/tasks (user stories) he wants to buy.
  • The cost/pricing of selected components/tasks can be exported to the billing tool.
  • Selected components/tasks can be exported to JIRA.


The whole project idea had been matured over months before I started with a first architectural draft. One big decision was choosing the DBMS to store the data. Instead of using PostgreSQL I chose Neo4j. One of the reasons for choosing Neo4j was the requirement that components can be interleaved in unlimited depth. Yes, I know Common Table Expressions (CTE) but hierarchical structures like quotes or Bill of Materials (BoMs) can be easily implemented with graph databases.


The current graph schema does not look exactly like this but it should give you an idea of the internal structure.

Schema of Request for Quote tool

Schema of Request for Quote tool

As you can see, containers can have an unlimited depth. For every node (task, item) multiple metrics can be defined. For performance reasons the parent’s container of a node contains the aggregated metrics of its children. The sample data can be found in the Neo4j console.

Current state

Until today I have implemented a first version of the repository and service layer in the backend. With Java’s Lambda expression I realized an internal DSL to calculate different metrics which can be based upon each other. For example I can specicy that if the amount or retail price of an item changes, both are multiplied and stored into the turnover metric for materials:

forMetrics(MetricType.AMOUNT, MetricType.RETAIL_PRICE_PER_UNIT)
			multiply(MetricType.AMOUNT, MetricType.RETAIL_PRICE_PER_UNIT, MetricType.TURNOVER_MATERIAL)

All metrics can be calculated and aggregated up to the root node. The service layer allows the concurrent editing of parts of the whole hierarchy (moving, adding or deleting subtrees).


There is still a lot to do but I am convinced that this project is cool and offers a real value for every person planning complex offers. I have no idea when (and if) I will ever finish the project. I am trying to build the tool as a SaaS platform so it should be relatively easy to make some money with it.

If you are interested in more details, drop me a line at me[at]schakko[dot]de.

Executing a CQL wildcard search in CMDBuild’s REST API

For our internal search engine I am currently developing a simple microservice to make our CMDBuild instance searchable. The microservice provides a fairly simple JSON API which itself queries the REST API of CMDBuild. Because of the insufficient documentation of CMDBuild I had to dig into the the source how to write a wildcard search query. CMDBuild has its own query language called CQL (CMDBuild Query Language). The CQL statements are converted into SQL which can be executed natively by PostgreSQL. CQL does also allow to include native SQL statements into the CQL queries. Native SQL statements are masked with (/( … )/). Between us, the combination CQL and SQL produces a absolute messy code, but this another story.

One problem is, that the REST search API of CMDBuild is exposed through HTTP GET. Accessing the HTTP endpoint with a filter like

GET https://cmdbuild/services/rest/v2/cql?filter={CQL: "from Department where Description (/(LIKE 'Develop%')/)"}

does unfortunately confuse the Apache CXF interceptor which struggles upon the percent sign. Encoding the percent does not help and a POST request is not allowed.

To fix this problem I took a look into the source of CMDBuild. Luckily for me the CQL parser is automatically generated with help of ANTLR. The grammer file is much better than any incomplete example from the official forum. So I discovered that CQL natively provides the following operators: CONTAINS, BEGIN, END, BETWEEN, NULL.
In the end it worked as I had expected:

GET https://cmdbuild/services/rest/v2/cql?filter={CQL: "from Department where Description CONTAINS 'Develop'"}

Fixing “Unable to obtain lock on store lock file” when using the embedded database of Neo4j

After some years without using Neo4j I had the chance to use the graph database in my current project. Neo4j was a good fit because it makes it really easy to prototype an idea and the project itself relies heavily upon tree structures. Modelling tree structures in relational database management systems is always a PITA and takes a certain time to implement in the backend. The good news is that trees are only directed graphs which can be easily modelled with Neo4j.

As a Spring guy I used the latest spring-data-neo4j 4.1.x release to connect the application to the graph database. For easier prototyping I used the embedded driver.

import org.neo4j.ogm.session.SessionFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.annotation.EnableTransactionManagement;

@EnableNeo4jRepositories(basePackages = "de.schakko.neo4j")
public class Neo4jConfiguration extends {
	public SessionFactory getSessionFactory() {
		return new SessionFactory(getConfiguration(), "de.schakko.neo4j");
	public org.neo4j.ogm.config.Configuration getConfiguration() {
		org.neo4j.ogm.config.Configuration config = new org.neo4j.ogm.config.Configuration();
		return config;

When running the integration tests (or after automatic reloading of the application by spring-boot-devtools) the application failed to reconnect to the database:

org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: file:///var/tmp/graph.db

In the end I fixed the error by using

import org.junit.After;
import org.junit.runner.RunWith;
import org.neo4j.ogm.drivers.embedded.driver.EmbeddedDriver;
import org.neo4j.ogm.service.Components;

class ItemRepositoryIT {
	public void tearDown() {
		((EmbeddedDriver) Components.driver()).close();
// ...

in the integration tests.

Open Tabs: Resources for Web Developers, How to write perfect blog posts, Argus

Resources for Web Developers

  • Freepik is a search engine for finding free vector graphics, PSDs, icons and and photos.
  • If you have to quickly create a landing page for your AppStore product, you can use the free service Landing Harbor.
  • Searching for a tool for your freelancer job? Take a look at The Freelance Stack.
  • Need a new Bootstrap-based theme? Bootstrap Zero should give you a good start.

How to write better blog posts

You have no idea how to start your next blog post? Try the free Content Idea Generator. And after you have found your topic you should definitely follow the rules of the awesome article After 10,000+ data points, we figured out how to write a perfect Medium post. The article contains a lot of hints how to optimize your blog posts.

Monitor your time series with Argus

Argus is an open source project proided by Salesforce to monitor your time series and receive alerts. I will take a deeper look at this project in the coming weeks.

On you can find some interesting videos about neural networks.

Making money with GitHub

I stumbled upon these two projects on HN: CodeMill and Bountysource both provides a marketplace service for GitHub pull requests. You can search for open tickets, make a pull request and get paid by the community or the project owner.

GitHub project management is a payed service to organize your GitHub projects. The open source alternative gh-board can be hosted on your own servers and has a lot of features like linking multiple GH repositories to one Kanban board.

Migrating InfluxDB from 0.9.6 to 0.10.0 GA

You may have already heard that InfluxDB 0.10 GA has been published a few days ago. In my case the most interesting improvement are the much higher compression rates: At the moment my co-workers of NeosIT and I are collecting performance data from four internal virtual machines. Have been running the collectd daemon on the VMs for about a month, the size of InfluxDB’s data directory increased by 3.5 GByte with default retention policy.

Testing the update

After setting up a new virtual machine with InfluxDB 0.9.6 for migration testing, I updated InfluxDB to the current version (Fedora 23 in our case):

sudo wget
sudo dnf install influxdb-0.10.0-1.x86_64.rpm

Two notes:

  1. As mentioned during the update procedure, the location of the configuration file moved from /etc/opt/influxdb/influxdb.conf to /etc/influxdb/influxdb.conf. The old configuration file is neither copied nor merged. You must edit the new configuration file or copy the old configuration to the new location. Keep in mind that 0.10.0 introduces some new settings.
  2. In addition to the changed configuration location the data directory has been moved from /var/opt/influxdb/ to /var/lib/influxdb. My advice is to move the old folder to the new location and overwriting any new files. Please create a backup from the /var/lib/influxdb folder before doing this. In my case this procedure worked without any problems.

Unfortunately it doesn’t seem to be possible to just copy a single database from one separate InfluxDB instance to another. Because of this drawback I imported some data from our virtual machines through Logstash and InfluxDB’s collectd backend. The Grafana dashboards still worked as expected so there seems to be no breaking changes in the HTTP API, at least as far as I could see.

Converting data from bz1 to tsm1

Now it was time to convert the data from the old bz1 format to the new tsm1. InfluxDB 0.10 delivers a nifty tool named influx_tsm. The usage is very easy and it worked as expected. Just pass the parameter “-backup” or “-nobackup”, the data location to be converted and you are done:

systemctl stop influxdb
influx_tsm -backup ~/inflxudb_backup -parallel /var/lib/influxdb/data/
systemctl stat influxdb

Results of the conversion

In my test environment everything worked as expected so I migrated our production environment. The influx_tsm took around ~50 minutes for 3.6 GByte of data but the results are awesome. Our data directory shrinked from 3.6 GByte to 400 MByte. All Grafana dashboards work as expected as well as our collectd and Graphite InfluxDB backends. This is the output of influx_tsm:

# ...
2016/02/07 13:34:29.100186 Still Working: Completed Shards: 29/30 Points read/written: 81626859/81626859
2016/02/07 13:34:31.969792 Conversion of /var/lib/influxdb/data/collectd/default/60 successful (49m41.016488953s)

Summary statistics
Databases converted: 3
Shards converted: 30
TSM files created: 30
Points read: 81655978
Points written: 81655978
NaN filtered: 0
Inf filtered: 0
Points without fields filtered: 0
Disk usage pre-conversion (bytes): 5239898112
Disk usage post-conversion (bytes): 451150534
Reduction factor: 91%
Bytes per TSM point: 5.53
Total conversion time: 49m48.08131854s

Thanks guys, good job!

Executing Liquibase database migrations from command line and as a shared Maven JAR

I am currently working on the migration of our time tracking system from Microsoft SQL Server/.NET to Java. Most of the logic resides in Stored Procedures and Stored Functions inside the database schema. Because of some reasons (testability, maintainability, migration from MSSQL to PostgreSQL in a far future) the whole logic must be converted to Java. As the system has a high criticality all of the end user applications must be running parallel. There are 4 tools in total, written in different languages: an old PHP web application, a bridge from JIRA to our time tracking system, another JIRA-to-time-tracker converter and the original C#/.NET fat client. All systems will be migrated bit by bit to the new Spring Boot web application.

Using Liquibase for database versioning

After collecting information about the current application environment I noticed that there were no database versioning system in use. The installation of the MSSQL schema was a pain: there were a lot of plain SQL files which had to be executed by hand. Since Java was the target programming language I decided to use Liquibase and moved the whole SQL scripts into a new Git repository, added the pom.xml and wrote a self-explaining how the .NET developers had to use Liquibase.

Running Liquibase migration with the command line Maven plug-in

I decided to describe only the pure Maven approach and not the Liquibase installation. The execution of the Liquibase migrations are trivial and no dependencies had to be installed by hand. The pom.xml contained the following definitions:

<project xmlns="" xmlns:xsi=""
			<!--  from internal Artifactory; Microsoft does not make the sqljdbc4.jar available in official repositories -->

The file did only contain default values and is not important. With the configuration above I were able migrate the MS SQL schema with

mvn -Dliquibase.url=jdbc:sqlserver://$HOST:1433;databaseName=$DATABASE-Dliquibase.username=$USERNAME -Dliquibase.password=$PASSWORD liquibase:update

At this point I could update the database schema by hand. This was necessary when someone had to develop inside a .NET environment or we had to migrate a staging or production database.

Making the schema available for developer environments

I realized quickly that the approach did not work really well for my Java development environment. A lot of database migrations had to be developed and the integration test environments should be automatically in sync with the defined migrations. My idea was to let Jenkins push the Liquibase defined schema as a Maven JAR into our internal Artifactory. I should be able to include the JAR as a normal Maven dependency and let Spring’s Liquibase integration to execute the latest migrations.

Reference a db-changelog.xml inside a JAR in your

I took a look in LiquibaseProperties and saw that the changeLog attribute supports the resource syntax. All I had to do was defining the db-changelog.xml by adding the following line to the


Please note that I changed the filename from db-changelog.xml to This should prevent ordering issues if there is already another XML file present with the same file name. The classpath prefix is used by Spring to scan all JARs in the classpath for the requested path.

Do not use the full path name of included SQL files

As I mentioned above all SQL statements resided in their corresponding SQL files. I used the following definition in the to include the SQL files:

	<changeSet id="5" author="ckl">
		<sqlFile dbms="mssql" encoding="utf8"
			splitStatements="true" />

This worked if Liquibase was either executed only through the Maven command line or as a Maven JAR dependency, but not both.

How Liquibase calculate database migration differentials

Liquibase iterates through all changesets defined in your db-changelog.xml. The XML attributes id, author and path are used at first to check whether this migration already exists in the database table DATABASECHANGELOG. If a row with the given parameters does exist, the checksum of the SQL file is calculated by normalizing the SQL file content (replacing new lines and so on). After that a MD5 checksum is generated by using the header and the content of the file.

The content of the “path” attribute differs

When executing mvn … liquibase:update inside the Git repository, the column path is filled with src/main/resources/liqiuibase/install/20151126-005_sp_calculate_worklogs.sql. Executing the migrations during the Spring startup process will result in a value classpath:/liquibase/install/20151126-005_sp_calculate_worklogs.sql for the path columns.
This means that every migration will be executed again, resulting in DDL errors.

Ignoring the path attribute

The easiest way was to use the attribute logicalFilePath in my databaseChangeLog tag. This forces all rows to have the same value of the path column:

<databaseChangeLog xmlns=""
	xmlns:xsi="" xmlns:ext=""

In the DATABASECHANGELOG table the column path is filled with path-ignored.

Do not mix up different Liquibase versions

After I had fixed the previous error Liquibase showed that the calculated checksum of the files differed. At first I thought I had some encoding issues and forced everything to UTF-8 but the error was still there. It took a while until I noticed that the Maven dependency liquibase-core in the Spring Boot app and the Maven Liquibase plugin for command line execution had different versions (3.3.2 versus 3.0.5). Both versions calculates the MD5 checksum in different ways. The checksum inside the DATABASECHANGELOG table differed with the newly calculated checksum. All I had to do was changing the Liquibase Maven plug-in to use the same version:

				<!-- same version -->


I moved the definition of a Microsoft SQL Server database schema into its own repository, made the schema migratable with help of Liquibase and made it executable in standalone/command line mode and as a Maven JAR dependency.

Open Tabs: Awesome CV, & Mobile Patterns, re:Work, d3.compose,, SqlPad

In the last week the following links stayed longer open in my browser tabs:

Awesome CV

If you are searching for a LaTeX template for your resume you should definitely take a look at Awesome CV. I will probably use the template in one of my projects. and Mobile Patterns

As I already mentioned from time to time I am a lousy user interface designer. Nevertheless I am interested in UI and UX. has a ton of resources for user experience designers like tools, links and methods for concepting and planning interaction design.

Mobile Patterns is a platform for sharing screenshots of mobile applications. This site is awesome because you get a good insight how to create responsive web applications which are easy to use.


re:Work describes steps to improve processes at your workplace.

Responsive d3.js charts with d3.compose

D3.js is awesome for creating interactive charts of every type. With d3.compose you can easily create responsive D3.js-based charts.

What I tell all new programmers

I noticed the blog post What I tell all new programmers on HN in the last week. I liked the content and linked the blog post in our internal wiki for apprenticeships.

README Driven Development

README Driven Development (RDD) means that you write the before you start programming. The idea is not new but has been working for me since years, too. I use it for my personal projects in combination with something I like to call Sketched Driven Development (just kidding!): Sketching the entity relationships and workflows with my own simple form of UML.

Perceptual testing with VisualReview

Perceptual testing simply means comparing application screenshots of the baseline (= production environment) with screenshots of your staging environment. The screenshots can be captured with tools like Selenium or PhantomJS. Thoughtworks provides a good overview what perceptual testing means. Beneath the tools DPXDT, Viff and Pix-Diff mentioned in Thoughtwork’s blog post, the open source tool Visual Review from Xebia is worth a look. – Visualize your cloud architecture like a pro

I don’t know how often I sweared about Microsoft Visio and other tools for modelling infrastructures. could be a game changer in the long term. At the moment it “only” supports the visualization of AWS-based infrastructure but there is potential to use it for other infrastructures as well.


The Node.js based application SqlPad allows you to execute SQL queries on different DBMS’ like MySQL, PostgreSQL and Microsoft SQL Server and visualize the returned result set with different charts. The queries can be saved for later execution. The application is ideal for simple dashboards based upon different datasources.

Project idea: Nostradamus AKA prophetr – a social forecasting network for experts

From time to time some programming ideas come to my mind which I can not forget. I had often started a new project but due to my limited amount of free time it is hard to finish all of them. In the next blog posts I will describe my ideas with some technical and environmental background. If you are interested in getting more information, buying the idea/source code or just motivate me, just drop me a line at me[at]schakko[dot]de. I would really appreciate your feedback.

Making prophecies and verifying their occurence

It must been around five or six years ago when I first thought about the idea of making prophecies in technical topics. I remember that I had a talk during the lunch with one of my co-workers about some IT trending topic which I had been propagated a for a few months. During this talk I said that a platform would be awesome where users can enter their prophecies and verify prophecies of other users if they had occured and how exactly the prophecy matches with the occurence.

Due to the number of prophecies made and the number of verified prophecies you could calculate the relevancy of a prophet (= a person who makes prophecies) which indicates the expert level of the prophet. A higher number of verified prophets means that you have more expertise in a given topic than other users.

Possible customers for social forecasting networks

The first intention was to have a social network for selfish reasons. Through some mechanisms, like not being allowed to change a prophecy after someone has voted for it, you would have been pinned to one statement which could be falsified or verified. If you were right, you could always use the classical phrase: “I said so.”.

In larger companies you could identify hidden champions or motivate people to be more interested in their expert knowledge. One day I had a talk with my bank account manager who were highly interested in the project because of obvious reasons. The software would allow them to evaluate the efficency of share brokers without using real money.

Another possible target group were whistleblowers or persons who wanted to make sure that a prophecy would be published on a specific date. For this I implemented some functionality to encrypt the content of the prophecy with symmetric keys. The keys could be stored on remote servers so that only the prophet was in control of when the prophecy can be published. After Snowedens revelations I instantly thought about this feature again.


I have to admit that the project has one big flaw: making self-fulfilling prophecies like: “I prophecy that the share price of company VWXYZ will drop in the next few days.” If you are already an expert in your area, there is a high chance that other follower will react to this prophecy and sell their shares. The share price will drop and your prophecy could be verified… You get the idea.

Technical background

At first I started with Spring MVC but after some weeks I switched to PHP/Zend Framework 1.x/MySQL. Most of the statistical computation (relevancy of prophets, influence of prophets and so on) and the social network aspect (who follows whom, which prophecies I can see) is done through database views which made the implementation inside the services really easy.
The encryption part called remote-credential-loader (RCL) is written in Node.js. RCL polls every few minutes the deposited decryption key URLs for encrypted prophecies. To a given timestamp (e.g. five minutes before releasing the prophecy) the URL must provide the AES decryption key, otherwise the prophecy is evaluated as false.

For the frontend I used Twitter Bootstrap 2.

The whole background documentation (processes, data model, computation) I had written in LaTeX (German language only).

Current status of the project

After thinking about the idea for years I finished the beta within the scope of my Bachelor project in the year 2012. The professor who belongs to the statistical faculty and who had observed the project was really impressed about it. Since December 2012 I am the owner of and which were intended to host the social network, but it is a classical 80%/20% project status. The application misses LDAP authorization and synchronization for usage in enterprise environments, the user interface and design is pragmatical and not very user friendly and so on.

A few months after I finished the Bachelor project I read an article in the c’t. If remember correctly they were from Austria and got a lot of money for building a social forecasting network like mine. This was more or less the reason why I had abandoned the project for the last two years.

Drop me a line at me[at]schakko[dot]de if you are interested in more information.

ExceptionHandler of @ControllerAdvice is not executed

It happened again: after writing about some issues caused by different JVM class-loader order a similar problem occured on Friday. One of my collagues (Dev-A) asked me to look into a problem the team had. Because of unknown reasons the Spring Boot based application did not return a serialized JSON error object after a @Valid annotated controller method parameter had been validated.

public class MyController {
	// Validator for MyDto (MyDtoValidator) got called
	public @ResponseBody MyData myMethod(@Valid MyDto myDto) {
		return new MyData()

An @ControllerAdvice annotated class transformed any validation error into a new exception. This has been done to unify the validation errors when using Spring Data REST and Spring MVC validation.

public class ValidationErrorHandlerAdvice {

	private MessageSourceAccessor messageSourceAccessor;

	public ValidationErrorHandlerAdvice(MessageSourceAccessor messageSourceAccessor) {
		Assert.notNull(messageSourceAccessor, "messageSourceAccessor must not be null");

		this.messageSourceAccessor = messageSourceAccessor;

	@ExceptionHandler({ MethodArgumentNotValidException.class })
	public RepositoryConstraintViolationExceptionMessage handleValidationErrors(Locale locale,
			MethodArgumentNotValidException exception) {
		// this method should be called if the validation of MyController.myMethod had failed
		return produceException(exception.getBindingResult());

	@ExceptionHandler({ BindException.class })
	public RepositoryConstraintViolationExceptionMessage handleValidationErrors(Locale locale,
			BindException exception) {
		return produceException(exception.getBindingResult());

	private RepositoryConstraintViolationExceptionMessage produceException(BindingResult bindingResult) {
		return new RepositoryConstraintViolationExceptionMessage(
				new RepositoryConstraintViolationException(bindingResult), messageSourceAccessor);

All in all, the controller advice itself looked fine to me, especially as the code is easy to understand and has been used in other projects too without any problems.

Empty HTTP response body

Nevertheless the behavior was mysterious:

  • When calling /validated in the browser, the custom validator for MyDto so the controller method got definitely hit. Nevertheless none of the exception handlers in the ValidationErrorHandlerAdvice got called. To make it more mysterious the HTTP response Spring generated did only consist of the HTTP status code 400 (Bad Request) without any character in the HTTP response body. The response body was completely clear.
  • Another developer (Dev-B) uses Linux as operating system. On his machine the code above worked without any problems and returned the expected HTTP status code 400 with the serialized JSON validation error object.

Dev-A has a Windows based machine. When he had called the “/validated” endpoint on Dev-Bs host the repsonse body contained the serialized validation error. In return, when Dev-B (Linux) had called “/validated” on Dev-As machine (Windows) the response body was empty.
I checked the HTTP request headers of both browsers but they were more or less the same and did not have any influence on any HTTP pre-filters Spring had registered. Both environments uses the Oracle JDK with different update releases (u43 vs. u63). Patching both JDKs to the same level I wanted to try at last as it seemed unlikely to be the reason.

Debugging session

I started to debug through the Spring Framework and realized that the order in which the registered exception handlers got checked for their responsibility of the current occured exception was completely different. On Dev-Bs machine the ValidationErrorHandlerAdvice were the first in the list, on Dev-A the first responsible exception handler was located in ResponseEntityExceptionHandler.
After stepping further through ResponseEntityExceptionHandler it made absolutely sense that the response body was empty on Dev-As machine. But it does not made any sense that the ResponseEntityExceptionHandler got loaded in the first place.

After searching for more @ControllerAdvice annotated classes in the project I found this piece of code:

public class CustomErrorController extends ResponseEntityExceptionHandler {
	public ModelAndView notFound(HttpServletRequest req, Exception exception) {;
		ModelAndView mav = new ModelAndView();
		// ... not so important ...
		return mav;

Okay, at least the exception handler of ResponseEntityExceptionHandler was introduced without any Spring magic.

Fixing the problem

During debugging the initialization phase of Spring I saw that the order of the detected controller advices was different between both systems: CustomErrorController got registered before ValidationErrorHandlerAdvice on Dev-A and vice versa on Dev-B. As the wrong behavior only occured on Windows machines I assume that the underlying component scan is responsible for the different order.

In the end the fix for this solution was easy. I annotated both controllers with @Order and gave the ValidationErrorHandlerAdvice a higher precedence than CustomErrorController.