You may have already heard that InfluxDB 0.10 GA has been published a few days ago. In my case the most interesting improvement are the much higher compression rates: At the moment my co-workers of NeosIT and I are collecting performance data from four internal virtual machines. Have been running the collectd daemon on the VMs for about a month, the size of InfluxDB’s data directory increased by 3.5 GByte with default retention policy.

Testing the update

After setting up a new virtual machine with InfluxDB 0.9.6 for migration testing, I updated InfluxDB to the current version (Fedora 23 in our case):

sudo wget https://s3.amazonaws.com/influxdb/influxdb-0.10.0-1.x86_64.rpm
sudo dnf install influxdb-0.10.0-1.x86_64.rpm

Two notes:

  1. As mentioned during the update procedure, the location of the configuration file moved from /etc/opt/influxdb/influxdb.conf to /etc/influxdb/influxdb.conf. The old configuration file is neither copied nor merged. You must edit the new configuration file or copy the old configuration to the new location. Keep in mind that 0.10.0 introduces some new settings.
  2. In addition to the changed configuration location the data directory has been moved from /var/opt/influxdb/ to /var/lib/influxdb. My advice is to move the old folder to the new location and overwriting any new files. Please create a backup from the /var/lib/influxdb folder before doing this. In my case this procedure worked without any problems.

Unfortunately it doesn’t seem to be possible to just copy a single database from one separate InfluxDB instance to another. Because of this drawback I imported some data from our virtual machines through Logstash and InfluxDB’s collectd backend. The Grafana dashboards still worked as expected so there seems to be no breaking changes in the HTTP API, at least as far as I could see.

Converting data from bz1 to tsm1

Now it was time to convert the data from the old bz1 format to the new tsm1. InfluxDB 0.10 delivers a nifty tool named influx_tsm. The usage is very easy and it worked as expected. Just pass the parameter “-backup” or “-nobackup”, the data location to be converted and you are done:

systemctl stop influxdb
influx_tsm -backup ~/inflxudb_backup -parallel /var/lib/influxdb/data/
systemctl stat influxdb

Results of the conversion

In my test environment everything worked as expected so I migrated our production environment. The influx_tsm took around ~50 minutes for 3.6 GByte of data but the results are awesome. Our data directory shrinked from 3.6 GByte to 400 MByte. All Grafana dashboards work as expected as well as our collectd and Graphite InfluxDB backends. This is the output of influx_tsm:

# ...
2016/02/07 13:34:29.100186 Still Working: Completed Shards: 29/30 Points read/written: 81626859/81626859
2016/02/07 13:34:31.969792 Conversion of /var/lib/influxdb/data/collectd/default/60 successful (49m41.016488953s)

Summary statistics
========================================
Databases converted: 3
Shards converted: 30
TSM files created: 30
Points read: 81655978
Points written: 81655978
NaN filtered: 0
Inf filtered: 0
Points without fields filtered: 0
Disk usage pre-conversion (bytes): 5239898112
Disk usage post-conversion (bytes): 451150534
Reduction factor: 91%
Bytes per TSM point: 5.53
Total conversion time: 49m48.08131854s

Thanks guys, good job!

I am asking you for a donation.

You liked the content or this article has helped and reduced the amount of time you have struggled with this issue? Please donate a few bucks so I can keep going with solving challenges.


2 Comments

Announcing the Graphite/Whisper to InfluxDB Migration Utility (Beta) | InfluxData · February 17, 2016 at 4:43 pm

[…] down to around 2 to 5 bytes per point in some cases. Learn more here and see a real world example here. I haven’t seen anything official from the Graphite project about compression, but this old […]

Graphite/Whisper to InfluxDB Migration Utility | Beta Version · July 1, 2017 at 2:22 am

[…] down to around 2 to 5 bytes per point in some cases. Learn more here and see a real world example here. I haven’t seen anything official from the Graphite project about compression, but this old […]

Comments are closed.