Synchronize OpenStack code with Syncthing

Syncthing is free and open source cross-platform decentralized file synchronization program. It can be considered as open source alternative to btsync. And btsync in its turn is decentralized alternative to well known Dropbox.

It's very flexible program and it has many use cases. I use it on Windows 8.1, Android, Debian and Ubuntu machines to synchronize folders for more than half year and it seems pretty reliable. I even synchronize database passwords

As you may know, it is recommended to install and run DevStack on the virtual (or just dedicated) machine as it alters system configuration.
So I like to use Syncthing in my development process to synchronize the code I'm working on between my development machine and testing machines. For example, with Syncthing I'm able to modify the code on my development machine and, after automatic synchronization, quickly run this code on the virtual machine I created specifically for manual and automatic testing. The virtual machine may even reside on another physical machine, more powerful, for example, as long as it is accessible in the network (preferably local). The solution works nicely together with git: I'm able to perform git operations on development machine with configured Gerrit review system, and changes then are spreaded to testing machines.

After completing this guide you will know:

1. Nice way to install Syncthing on Ubuntu (Installation part).
2. How to synchronize OpenStack projects with Syncthing (Using Syncthing with OpenStack repositories part).
3. Some tips (Useful tips part).

Note: in this setup guide the development environment is created from scratch. If you already have nice development environment setup, just use your imagination and adapt these instructions. =)

Installation

You should install Syncthing on two or more machines.
In our case it's development machine and testing machine.

Step 1. Install Syncthing on Ubuntu

In this step we will install Syncthing daemon with web UI.

Just follow these instructions on how to install Syncthing repository with Syncthing itself. It will allow easy updates through apt-get in the future.

If you want to omit step 2, add syncthing to autostart programs and use the embedded Syncthing's web UI instead of Syncthing-GTK.

Step 2. Install Syncthing-GTK (optional)

Syncthing-GTK is a wrapper for Syncthing daemon. It provides nice desktop GUI, almost 100% similar to Syncthing's web UI. And shows notifications from the tray about synchronization events.

But what is more important, it's able to make Syncthing use inotify mechanism to monitor filesystem changes in real time. Without inotify Syncthing just periodically rescans folders on given time interval.

To start using Syncthing-GTK:

1. Install it from main Web Upd8 Ubuntu PPA, which contains Syncthing-GTK. Run:

$ sudo add-apt-repository ppa:nilarimogard/webupd8
$ sudo apt-get update
$ sudo apt-get install syncthing-gtk

2. Start Syncthing-GTK via icon or running syncthing-gtk command in the terminal.

Finish the First run wizard.
It will ask if it should start Syncthing daemon or not.
Hit the checkbox "Always start daemon automatically" and then click on the "Start" button.

You will see the main window:

3. Make Syncthing run automatically on system start. Click on Syncthing logo (blue one), select "UI settings" and choose it as option:

Step 3. Edit firewall rules

Edit your firewall rules, if needed.
Note: depending on how you first started Syncthing, different configuration will be created for Syncthing:

• When started Syncthing by syncthing command, web UI will listen on default port 8384.
• When started Syncthing by syncthing-gtk command, web UI will listen on port 8080.

The configuration is stored in ~/.config/syncthing directory.

Using Syncthing with OpenStack repositories

Step 1. Connect Syncthing machines (devices)

My development machine is called "yelly". My testing machine is called "devstack". Syncthing-GUI is running on both of them. Now we need to make these machines recognize each other through Syncthing.

On one of the machines, for example, on testing machine in Synchting-GUI click on the gear icon and choose "Show ID":

On the development machine click on the gear icon and choose "Add Device", input the ID you saw before and choose name for this device, then click "Save":

Wait for machines to discover each other. Then on your testing machine you'll see something like this:

Click "Add" button and choose name for development machine.

Now your development and testing machines will discover and recognize each other.

Note: Device ID is actually public key. The private key was generated on first Syncthing start and stored with other Syncthing settings in ~/.config/syncthing directory. The private key is not tied to hardware and new one will be generated, if you delete ~/.config/syncthing directory and start Syncthing again.

Step 2. Delete default shared folder that was created by Syncthing on first run

On both machines in Syncthing-GTK right-click on it and choose "Delete":

Step 3. Synchronize DevStack repository

Let's add our first custom shared folder with DevStack repository to Syncthing.

On development machine:

Git clone DevStack repository somewhere. You probably did it already. For example, I cloned it to ~/Sync/Repos/devstack directory:

yell@yelly:~$ git clone https://github.com/openstack-dev/devstack.git ~/Sync/Repos/devstack
Cloning into '/home/yell/Sync/Repos/devstack'...
remote: Counting objects: 32362, done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 32362 (delta 11), reused 0 (delta 0), pack-reused 32327
Receiving objects: 100% (32362/32362), 11.69 MiB | 7.31 MiB/s, done.
Resolving deltas: 100% (22495/22495), done.
Checking connectivity... done.

In Syncthing-GTK click on gear icon and choose "Add Shared Folder". In new window choose a name for shared folder and specify path to cloned devstack repository, check "Monitor filesystem for changes", disable "File Versioning" and click "Save":

Important: before we'll share DevStack folder with testing machine, we need to specify ignore patterns for files (the concept is similar to .gitignore file and most of the patterns are taken from there). It would be bad if runtime files created by running DevStack will be shared amongst devices.

Right click on the created shared folder in Syncthing-GTK and choose "Edit Ignore Patterns". In new window paste these lines and click "Save":

*~
.*.sw?
*.log
*.log.[1-9]
*.pem
.localrc.auto
.prereqs
.tox
.stackenv
accrc
doc/files
doc/build
files/*.gz
files/*.qcow2
files/*.img
files/images
files/pip-*
files/get-pip.py*
files/ir-deploy*
local.sh
localrc
proto
shocco
src
stack-screenrc
userrc_early

.idea

As you see I intentionally haven't included local.conf to ignore patterns, so I can edit this file on development machine and use it on testing machine.

The ignore patterns are stored in the .stignore file inside shared folder. This file was created by Syncthing as well as .stfolder file. These files will be shown as untracked by git and we don't want git to see these files. We also don't want to edit .gitignore file. The solution is simple: add these files to the local git exclusion list. Inside devstack repository open file .git/info/exclude and add names of these files, so it will look like this:

# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
.stfolder
.stignore

Now these files will be invisible to git and we will not accidentally commit them.

Right click on the created shared folder in Syncthing-GTK and choose "Edit". Share the folder with testing machine by clicking on checkbox in the bottom:

On testing machine:

In Syncthing-GTK you'll see notification from development machine asking to add shared folder:

Click "Add" button. In new window specify path where you want devstack repository to be synchronized to, for example, ~/Sync/Repos/devstack and disable "File Versioning". Do not check "Monitor filesystem for changes" as we don't need to monitor file changes on testing machines, in some cases it may even may lead to problems with git, I suppose. Then click "Save":

The contents of shared folder (except ignored files) will be copied from development machine to testing machine and from now on it will be synchronized.

Important: on testing machine right click on the created shared folder in Syncthing-GTK and choose "Edit Ignore Patterns". In new window paste same lines as before and click "Save":

Now you can play with local.conf file, for example. You can modify this file on any machine and the change will propagate to the other machine.

Step 4. Start DevStack on testing machine

Configure DevStack on testing machine, for example, by using this manual.
Note: I recommend to specify "RECLONE=no" in DevStack's local.conf for now. I will explain why in the Useful tips part.

Now let's start DevStack on testing machine. Inside the DevStack local repository (shared folder) run:

stack@devstack:~/Sync/Repos/devstack$ ./stack.sh

stack.sh script will clone repositories of OpenStack projects to the default /opt/stack/ directory. We can use created repositories as shared folders later.

Let's stop DevStack for now. Inside the DevStack local repository run:

stack@devstack:~/Sync/Repos/devstack$ ./unstack.sh

Step 5. Synchronize an OpenStack project

I will use Keystone project as example.
So on the testing machine in Syncthing-GUI create a shared folder "KeystoneSync" and it point to /opt/stack/keystone directory. You already know how to do it from step 3. But now use these ignore patterns:

*.pyc
*.sw?
*.egg/
*.egg-info
vendor
.eggs
.ksl-venv
.venv
.update-venv/
.tox
keystone.egg-info/
*.log
.coverage
coverage.xml
cover/*
covhtml
pep8.txt
nosetests.xml
doc/build
.DS_Store
doc/source/api
doc/source/modules.rst
ChangeLog
AUTHORS
build/
dist/
etc/keystone.conf
etc/logging.conf
etc/keystone/
keystone/tests/tmp/
.project
.pydevproject
keystone/locale/*/LC_MESSAGES/*.mo
.testrepository/
*.db
releasenotes/build
doc/source/_static/keystone.conf.sample
.idea

Sync this shared folder with development machine (I prefer to use same directory - /opt/stack/keystone) and don't forget to specify ignore patterns as well.

Now on development machine you can add this shared folder to your IDE as python project and work with it. You can run unit tests or keystone-all command on the testing machine. Also you can checkout some patch from git on development machine and see the update on testing machine.

As you can see most of the ignore patterns I take from project's .gitignore file. I noticed that most common ignore patterns for OpenStack projects are:

*.pyc
.idea
.testrepository
*.egg-info
.tox
.eggs

Step 6. Synchronize an OpenStack project's client

Same as step 5, but I usually clone OpenStack clients to subdirectories inside ~/Sync/Repos/ directory and run them in python virtual environments.

Useful tips

• If you have some uncommitted changes, running DevStack on testing machine will erase the changes, if you have specified "RECLONE=yes" in your Devstack's local.conf. So only run DevStack with "RECLONE=yes" if you have nothing to commit. If changes are committed to some branch, nothing to worry, DevStack will not erase these branches on reclone, but will just switch to master branch.
• It's very important to edit ignore patterns properly in Syncthing shared folders to exclude all runtime files. By doing this, you greatly reduce size of files to syncronize, and also prevent some errors that could come from sharing runtime files.
• You can also synchronize configuration files for OpenStack projects.
• It's better not to show your Syncthing device ID in public. For this tutorial I used temporary Syncthing configurations 
• Sometimes Syncthing might show synchronization conflict in one of the shared folders and will show the files affected. For example, this often happens when not enough ignore patterns were specified and same files were modified in the same time on multiple devices. The solution is simple:

1. Choose the machine which has the right files.
2. On the chosen machine edit the shared folder in Syncthing and check "Folder Master" option, restart Syncthing.
3. After restart and connection to other machines "Override Changes" button will be appear on shared folder in Syncthing on chosen machine. Click on it.
4. On the chosen machine edit the shared folder in Syncthing and uncheck "Folder Master" option, restart Syncthing.
5. Problem solved.

• Have fun! ♥