@niftylettuce - Automated Continuous Integration Setup for Graceful and Zero-Downtime ...
: 17
April 2016
tldr; This article shows you how to configure an insanely simple automated continuous integration and deployment setup for a Node.js app using GitHub, PM2, Digital Ocean, and SemaphoreCI. I wrote it because nothing like this in its entirety exists. It should take you 30 minutes to set up properly.
Index
Preface
I’ve used or explored nearly every CI testing tool there is for Node.js (maybe?). I have tried TravisCI, but grew tired of constant downtime and slow, very slow build times (…OK, the builds ran fast, but they did not kick off quickly!). Also I’ve tried CircleCI, but their founders removed my thoughts from their community because they didn’t agree to allow the file name for YAML config to be .circle.yml
instead of circle.yml
. I also faced troubles while trying to configure and set up Jenkins (though it was while I was working with an inexperienced team, whom were the ones setting it up). I’ve also looked at Shippable, but it really didn’t interest me, just like the rest – because I now enjoy working with SemaphoreCI – namely since the prodigy TJ Holowaychuk recommended it to me.
Definitely the nicest CI I've used https://t.co/ykgXUcLK1o, the others feel clunky
— TJ Holowaychuk (@tjholowaychuk) November 28, 2015
For anyone interested in getting into the automated CI deployment business, it’s relatively straightforward to market yourself – just list yourself in all the Wikipedia articles, on Quora (with some upvote magic), have a good service that doesn’t shut down or lie about build times, and have clear docs. If you do those four things, you’re on the way to at least some passive income!
With regards to server hosting, I chose Digital Ocean because they rock. I have never had a problem with them in over five years. That’s something! I also printed t-shirts for Digital Ocean before I sold Teelaunch, and really liked working with them.
Not only all that, but their service has great uptime, and their boxes “droplets” are really fast to set up and reliable. I’m not a huge fan of using Amazon EC2 and AWS in general for building Rapid MVP’s (of course I would definitely use load balancing or something for scaling an app that has thousands of users across the world). If your first question about building an app is “How can I scale it?” or “Will Digital Ocean let me scale?” – take my advice, you’re doing it wrong. Stop it. Think Rapid MVP.
To put it simply, Amazon has an interface that resembles a wild jungle with overgrown vines on every tree, and Digital Ocean’s interface is a beautiful oasis in a vast VPS desert.
As a side note, I can almost guarantee you that sometime in the future, everyone will want barebones boxes connected to ethernet plugs. Because imagine when everyone has fiber internet and anyone can host their e-commerce store from a RaspberryPI running from their kitchen table.
1. Create your Droplet
First, you need a Digital Ocean account. Be patient as their signup process may require you to verify your email and enter your credit card.
Sign up with this link to get $10 of free credit (2 months of hosting): https://m.do.co/c/a7fe489d1b27
When you create your Digital Ocean (“DO”) droplet be sure to only allow SSH only access and add your SSH key to Digital Ocean. You can do this from DO’s dashboard and you can find more about this on a Digital Ocean article.
Make sure you create a droplet using the latest stable Ubuntu release.
Now we’ll SSH into the droplet and install dependencies for your stack with Node.
In my case, I needed to install Node, MongoDB, and Redis. Of course, MongoDB and Redis are optional dependencies, but I use them because they allow me to build Rapid MVP’s (quick prototypes in other words). Also, I really like to use NVM to manage various version of Node installed, which was created by another prodigy, Tim Caswell. We’ll install NVM later when we’re doing stuff with SemaphoreCI.
Make sure you replace all instances in this article of
droplet-ip-address
with the IP address given to you by Digital Ocean for your droplet.
ssh root@droplet-ip-address
If you only have a $5/mo droplet through Digital Ocean, you will encounter memory errors later on. Therefore I highly recommend to add swap to this droplet right now. Or you could upgrade to a $10/mo droplet, which has more memory, and is less likely to run out during NPM installations. If you’d like to add swap to your droplet, here are the official instructions from Digital Ocean:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
Install the basic requirements needed for the server:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install vim build-essential libssl-dev git unattended-upgrades authbind openssl fail2ban
Install MongoDB, which is optional:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
service mongod status
Install Redis, which is optional:
sudo add-apt-repository ppa:chris-lea/redis-server
sudo apt-get update
sudo apt-get install redis-server
redis-benchmark -q -n 1000 -c 10 -P 5
source ~/.profile
Later, we will install Node using NVM while logged in as the SemaphoreCI user.
You might also want to look into installing
fail2ban
, changing the default SSH port, and remove password-based login access. You can find how to do this from this section in my security article, or just Google it. Keep in mind that if you follow my security article, I remove root user access. Therefore before removing root user access in that article, you should make sure that thesemaphoreci
user hassudo
access (just search for “how to add user sudo permission” on Google). Once you do that, then you can simply typesudo -i
to switch to the root user if you SSH in assemaphoreci
. Or you could create a new user, such asniftylettuce
as the username. Then you can add your SSH public key located in your local~/.ssh/id_rsa.pub
file as a new line in theniftylettuce
user’s authorized key file located at/home/niftylettuce/.ssh/authorized_keys
. Very similar steps described here are shown in step 3 below. I suggest you read ahead before you follow this security article or other security setup steps.
2. Write your Node App File
This article assumes you already have created a GitHub repository for your project and that you already have some app.js
file in the root of it. If you haven’t done that yet, then this section is for you. This section also describes how to configure that app.js
file for zero-downtime and graceful reloading upon deployment of code.
For the purpose of this article, I share a basic app example that will respond with “hello world” when you visit your droplet later on (over port 3000
).
Answer yes to all the prompts or just hit ENTER
to breeze through it:
npm init
Now save the basic express
dependency:
npm i --save express
Create a new file called app.js
(or edit your existing to include SIGINT
):
vim app.js
var express = require('express');
var app = express();
app.get('/', function(req, res) {
res.send('hello world')
});
app.listen(3000);
process.on('SIGINT', function() {
// TODO: do stuff here to clean up before reload
// (e.g. close out DB connections, queue a job)
setTimeout(function() {
// 300ms later the process kills itself to allow a restart
process.exit(0);
}, 300);
});
Let’s test this out locally before you bother to continue further.
node app.js
Visit this URL in your browser (it should say “hello world”): http://localhost:3000
By default, PM2 will allow
1.6
seconds for your app to gracefully exit, and you can read more on how to configure your app for zero-downtime here: http://pm2.keymetrics.io/docs/usage/signals-clean-restart/
3. Set up SSH for SemaphoreCI
First, go to https://semaphoreci.com and sign up for an account.
Once you’ve logged in, create a project and connect with your GitHub account.
Make sure that your “Node version” shown under your SemaphoreCI project’s build settings matches the output from your droplet when you run node -v
.
For example, in this screenshot I have selected the
v5.8.0
that I’m using.
Now we need to add a user to the droplet to let SemaphoreCI deploy the app after all tests have successfully passed.
Keep your SemaphoreCI browser tab open, because we will come back to that in just a bit!
Copy to your clipboard the contents of your local ~/.ssh/id_rsa.pub
file. If you have not yet already created this file, see GitHub’s instructions.
I’m using pbcopy
(while on Mac OS X) to make it easy and do it the CLI way:
cat ~/.ssh/id_rsa.pub | pbcopy
Now SSH back into your droplet if you’re not still connected:
ssh root@droplet-ip-address
Add the user semaphoreci
on the droplet, so you can then SSH in as them. When you are prompted for a password, write it down or make it memorable.
sudo adduser semaphoreci
Switch user to semaphoreci
and paste your clipboard contents into the file called ~/.ssh/authorized_keys
. This will let you test deployments from your local computer as the semaphoreci
user later on. In other words, you can SSH into your droplet as the semaphoreci
user easily. It’ll make sense later, don’t worry.
su semaphoreci
mkdir ~/.ssh
chmod 700 ~/.ssh
vim ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
Since we’re logged in as the SemaphoreCI user, now is a good time to install Node.js for them on your droplet. To do so, follow these commands:
Install NVM and set it up to use the latest stable version:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash
source ~/.bashrc
nvm install stable
nvm alias default stable
Install PM2, which will handle deployments for us and manage our processes:
npm i -g pm2
Since PM2’s deployment command runs on a non-interactive SSH connection, we need to resolve this by commenting out a few lines from the semaphoreci
user’s ~/.bashrc
file. We’re already logged in as this user on our droplet, so we simply need to open up this file and comment this out:
vim ~/.bashrc
Here are the changes:
# If not running interactively, don't do anything
-case $- in
- *i*) ;;
- *) return;;
-esac
+#case $- in
+# *i*) ;;
+# *) return;;
+#esac
Reload the file to instantly get our new changes in our shell:
source ~/.bashrc
Next we need to create an SSH key for the actual semaphoreci
user, so we can share the contents of the private key we create on the SemaphoreCI dashboard.
Change directories to your local box’s SSH folder and create a key:
cd ~/.ssh
ssh-keygen -t rsa -b 4096
When you’re prompted to enter a file to save the key, enter the following:
semaphoreci_id_rsa
Don’t enter a password for simplicity.
Again, copy the contents of this SSH key now to your clipboard using pbcopy
:
cat ~/.ssh/semaphoreci_id_rsa.pub | pbcopy
Now SSH back into your droplet, switch to the semaphoreci
user (see above), and add this as a new authorized key to that same file you created earlier (and added your own SSH key into). You should add it as the next line in the file on your droplet at /home/semaphoreci/.ssh/authorized_keys
. This will allow SemaphoreCI access to your droplet later on:
ssh root@droplet-ip-address
su semaphoreci
vim ~/.ssh/authorized_keys
Now go back to that browser tab you have open for SemaphoreCI, and click on the link for “Set Up Deployment”. This link is found on the page that looks like this:
It will then present you with options to choose from. Scroll down and select the option titled “Generic Deployment”, and then click “Automatic”. You should now be on a screen that looks like this:
Add the following deploy commands where it says “Enter your deploy commands”:
Make sure you replace
droplet-ip-address
with the IP address of your Digital Ocean droplet. Also, if you changed to a non-standard SSH port, change where it says22
in-p 22
below.
# install pm2 so we can run the deploy command
npm i -g pm2
# add this server as a known host, since we can't enter y/n when prompted
ssh-keyscan -p 22 -H droplet-ip-address >> ~/.ssh/known_hosts
# run the deployment command
pm2 deploy ecosystem.json production
After you enter this command, it will now prompt you to paste in the value of the private key file for the semaphoreci
user. You don’t have this on your clipboard yet, so you need to use pbcopy
again locally:
cat ~/.ssh/semaphoreci_id_rsa | pbcopy
Paste the contents of your clipboard in the box shown in this screenshot:
If you want to easily simulate SemaphoreCI logging in as the semaphoreci
user then you can do this by the running following from your local box:
ssh -i ~/.ssh/semaphoreci_id_rsa semaphoreci@droplet-ip-address
You can also do this command much easier by creating a file on your local box called ~/.ssh/config
with these contents (replace your droplet IP):
Host semaphoreci-droplet
Hostname droplet-ip-address
User semaphoreci
ForwardAgent yes
Port 22
IdentityFile ~/.ssh/semaphoreci_id_rsa
Then you can just run ssh semaphoreci-droplet
and save a bit of typing. Note that I left the line Port 22
in there in case you change your SSH port. The line that says ForwardAgent yes
means it forwards your SSH agent.
I’d highly recommend you test this out right now to make sure it’s set up OK.
4. Add new GitHub Deployment Key
Since we have a semaphoreci
on our droplet, we now need to add a deployment key on GitHub for our project, so that we can test deployment locally.
SemaphoreCI already has added a deployment key for your project (if you set it up correctly), so don’t be alarmed if there’s already a key created when you get to the GitHub Deployment Key settings page for your repo. You’ll be creating another one for local testing purposes, don’t worry!
First SSH into your repository as the semaphoreci
user:
ssh semaphoreci-droplet
Now create an SSH key pair:
cd ~/.ssh
ssh-keygen -t rsa -b 4096
When it asks you where to save the file, use the default and hit ENTER
.
Don’t enter a password for simplicity, again.
Go to https://github.com and click on your project, then go to its Settings.
Under “Deploy keys” add a new deployment key, allow it write access, and paste the id_rsa.pub
public key file’s content we just created. To easily get the contents of this public key on your clipboard, from your local box run this command:
ssh semaphoreci-droplet "cat ~/.ssh/id_rsa.pub" | pbcopy
Here’s the screen showing where you enter your key. Don’t be alarmed if you already see a Deploy here in here; it’s supposed to be there, as it was added automatically by SemaphoreCI in a previous step (yes, you’re adding another!):
If you get stuck on this step or need more instructions, see this article:
https://developer.github.com/guides/managing-deploy-keys/#deploy-keys
Finally, you should test that SSH access to GitHub works. SSH into your droplet as the semaphoreci
user and run GitHub’s test command:
ssh semaphoreci-droplet
ssh -T git@github.com
It should output something like this if it was successful:
Hi USER/REPO! You've successfully authenticated, but GitHub does not provide shell access.
5. Share /var/www Access
We created the user semaphoreci
in the previous section, and now we need to give it recursive read and write access to the /var/www
folder on the server – so that the pm2
command can deploy to the server (from both our local box if we want to deploy manually, and also from SemaphoreCI’s environment for the automated continuous integration deployments).
We need to SSH into the droplet as the root user, so we can then add this folder and then give permissions on it to the semaphoreci
user.
ssh root@droplet-ip-address
Now create the folder using sudo
:
sudo mkdir /var/www
To stay in compliance with standards used widely by infrastructure teams, we’ll use the classic
www-data
group to manage permissions on this folder.
Add the semaphoreci
user to this group:
sudo adduser semaphoreci www-data
Change ownership of the folder and its files recursively:
sudo chown -R www-data:www-data /var/www
Grant the group read and write permissions (say that phrase five times fast!):
sudo chmod -R g+wr /var/www
That’s all.
If you wanted to test it out, then SSH in as the semaphoreci
user, and try to run the command touch /var/www/test.txt
. It should let you create a blank text file in that folder as the semaphoreci
user. If you did not do this properly, then you will encounter the following read/write error later on:
pm2 deploy ecosystem.json production setup
--> Deploying to production environment
--> on host droplet-ip-address
mkdir: cannot create directory ‘/var/www’: Permission denied
mkdir: cannot create directory ‘/var/www’: Permission denied
mkdir: cannot create directory ‘/var/www’: Permission denied
6. Configure PM2 for Deployment
We’re going to set up a configuration file to be read by PM2.
On your local box, make sure you have pm2
installed globally:
npm i -g pm2
Create a new file in the root of your GitHub project called ecosystem.json
.
vim ecosystem.json
Note that you can automatically create this file (with defaults) from PM2’s CLI using pm2 ecosystem
, however for the purpose of this article I’m providing you with the content here. You need to replace the following:
droplet-ip-address
with your droplet’s IPrepo
property value with the path to your GitHub repo
{
"apps": [
{
"name": "App",
"script": "app.js",
"exec_mode": "cluster",
"instances": "max",
"env_production": {
"NODE_ENV": "production"
}
}
],
"deploy": {
"production": {
"user": "semaphoreci",
"host": "droplet-ip-address",
"ref": "origin/master",
"repo": "git@github.com:username/reponame.git",
"path": "/var/www/production",
"post-deploy": "npm i && pm2 startOrGracefulReload ecosystem.json --env production",
"forward-agent": "yes"
}
}
}
If you need a reference for the options here, see the official docs here: http://pm2.keymetrics.io/docs/usage/deployment/
Note, if you have a custom port, you’ll need to add that as a
"port"
property in yourecosystem.json
‘sdeploy
nested object for each env.
Now run setup for deployment with PM2 using the CLI command, and make sure you run this command from the root of your project’s folder locally:
pm2 deploy ecosystem.json production setup
You could (for fun) try running this command twice. If it worked the first time, you will get an error on the second try; it will say the folder exists already at the path
/var/www/production
!
Go ahead and deploy the production environment and start its processes:
pm2 deploy ecosystem.json production
You can test it out at the following link (replace with your IP): http://your-droplet-ip:3000
If all is OK, then make sure that PM2 is scheduled to startup automatically if your server reboots or something happens.
Make sure you run this command as the semaphoreci
user on the droplet:
ssh semaphoreci-droplet
pm2 startup ubuntu
It will give you output which you will then need to run as a user with root access, which you can get by running:
ssh root@droplet-ip-address
# paste output and run it here
Now save the current processes to automatically restore them if your server reboots or something happens. To do this, first make sure we have PM2 processes running that we’ll be able to save:
ssh semaphoreci-droplet
pm2 status
If no processes appear, go back to the section with PM2 deployment commands,
If your processes appear, then run this command as the semaphoreci
user on the droplet, so that these processes will get restored if something happens:
ssh semaphoreci-droplet
pm2 save
All done! Now try to commit some code and watch SemaphoreCI deploy it for you.
For example, you could make it say “thanks nifty” instead of “hello world”:
vim app.js
app.get('/', function(req, res) {
- res.send('hello world')
+ res.send('thanks nifty')
});
git add .
git commit -m 'testing out semaphoreci automatically deploy my project'
git push origin master
Now just wait and watch the SemaphoreCI dashboard. It will run a build, then it will deploy it to your Digital Ocean droplet for you using PM2.
If you want to see the pm2 save
do its magic, then just run sudo reboot
, or reboot your droplet from Digital Ocean’s interface. When it powers back on, SSH into it as semaphoreci
, and run pm2 status
to see your app is running.
7. PM2 Deployment Commands
This documentation is sourced directly from Keymetrics Blog and also from the official PM2 deploy documentation.
# update the code for production
pm2 deploy production update
# revert to [n] th commit for production
pm2 deploy production revert 1
# execute a command on the production server
pm2 deploy production exec "pm2 restart all"
This deploy
command option is inspired from TJ’s deploy
shell script at:
https://github.com/visionmedia/deploy
Notes
- If you found this article useful, or if you want to ask a question, or need my help with something, please reach out by emailing niftylettuce@gmail.com or on Twitter @niftylettuce.
- Make sure you add a SemaphoreCI badge to your Readme, and write (some) tests!
- If you want to run your app on port
80
or443
(restricted ports), then you can easily do this usingauthbind
. Read PM2’s docs on this. You may also need to modify yourecosystem.json
file to use authbind too. For example, mypost-deploy
line, looks something like this:{ "post-deploy": "npm i && npm test && authbind --deep pm2 startOrGracefulReload ecosystem.json --env production" }
- If you want to run a redirection from
http
tohttps
on your server, then you could use node-http-proxy, ornginx
‘s reverse proxy. - I created this doc using vim-instant-markdown, and I would have used my own project Seuss.md, but it wasn’t ready yet when I started to write.
- There might be typos or instructions I’m missing, if so send a pull request.
niftylettuce@gmail.com | Github | Twitter | Updates | RSS/XML Feed
