A shop owner writes: the developer who built the site has moved on, a payment page has started failing, and nobody is sure how changes reach the live site. The obvious move is to open the code and fix the payment page. It is also the riskiest one, because nobody yet knows whether the code they would open is the code that is running, what else a change would touch, or how to undo it if it goes wrong.
Most software that reaches me was written by someone else. Sometimes the previous developer is still around and helpful, often they are not, and occasionally nobody can say who wrote it at all. Either way, the situation is the same: a business depends on a system, something needs to change, and the person about to change it does not yet know the system.
Before I change anything, I make the same four checks. None of them is clever. All of them are skipped surprisingly often, usually because there is an urgent problem and the checks feel like a delay. They are the opposite: each one can stop a project on its first day, and it is much cheaper to find that out before a change than in the middle of one.
This article goes through the four checks in detail, with the commands and the lists I use, so you can make them yourself, or check that whoever takes over your system makes them.
First: collect what exists
Before the four checks, collect whatever already exists, even if it looks useless:
- any handover notes, README files or wiki pages
- the contract or statement of work with the previous supplier, which often lists what was delivered and where
- invoices from hosting, domain and software providers, which tell you which accounts exist and who pays for them
- e-mails with passwords or server details, however embarrassing
- the names of the people who use the system every day, and what they would call "broken"
The last item matters more than it looks. The people who use a system every day know its real behaviour, including the workarounds they stopped noticing years ago. Ten minutes with them before reading any code saves hours of guessing.
1a. Is the running code the code in the repository?
The first question is whether the code in the repository is the code that is running. It often is not. Common reasons:
- a fix made directly on the server, in a hurry, and never committed
- a file uploaded by hand to change one thing
- a branch that was deployed and never merged
- a repository that was abandoned when the developer changed jobs, while the server kept running
On a server that runs from a git checkout, the fastest check is to ask git what differs from the last commit:
cd /path/to/app
git status --short
git log -1 --format="%h %ad %an %s"
git diff --statAny modified or untracked file is a change the repository does not know about. Before touching anything, copy those changes somewhere safe and commit them to a separate branch, so they are not lost the first time the repository is deployed over them.
If the application runs from built files or containers, compare what is running with what the repository would produce: the image tag or build number in production against the last build of the main branch.
docker ps --format "{{.Names}} {{.Image}} {{.Status}}"If the image names end in latest and nobody knows which build that was, that is a finding in itself.
1b. How a change reaches the live system
Then: how does a change get to the live system? The answers fall into three groups:
- An automated pipeline: a push to a branch builds and deploys. Good, as long as someone knows which branch, and the pipeline still works.
- A script someone runs: better than nothing, and it needs to be found, read and tried.
- Files copied by hand, over FTP or SSH. This is the most common answer for older systems, and the most fragile.
And is there a copy of the system where a change can be tried before customers see it, a staging site? If not, creating one is usually the first real piece of work. It does not have to be elaborate: a copy of the application, pointing at a copy of the data, on a separate address, is enough to try a change safely.
While looking at the server, list the things that run on their own, because they are the easiest to miss and the most painful to lose:
crontab -l
sudo ls /etc/cron.d/
systemctl list-timers --allNightly exports, invoice runs, feed updates, clean-up jobs: each one is a piece of behaviour the business relies on that appears nowhere in the application’s screens.
2. Access: whose name is everything in?
Make a list of every account the application depends on. For each one, write down whose name it is in, who can log in, and whether two-factor login is on:
- the server or hosting account
- the domain registrar, and the DNS provider if it is different
- the database
- the code repository
- the payment provider
- the e-mail sending service
- the courier, accounting or other outside services with an API key in the code or the configuration
- analytics and advertising accounts that the site reports to
The warning sign is an account held only by a former developer or agency. It is a risk before it is a nuisance: the day it is needed is the day it cannot be reached. The domain is the classic case. A domain registered in a former supplier’s account, with renewal reminders going to their e-mail, can expire without anyone in the business being told.
Moving these accounts into the business’s name is unglamorous and among the most valuable things done in a takeover. Where an account cannot be moved, at least add a person from the business as a second owner.
Secrets, certificates and mail
The same list shows which secrets exist: database passwords, API keys, the application’s signing keys. Find where each one is stored. Configuration files on the server, environment variables, sometimes the code itself.
Any secret the previous supplier could see should be changed once the takeover is done, and any secret written into the code should be moved out of it, because every copy of the repository carries it. Changing a secret is a change like any other: plan it, try it on the staging copy, and know how to roll it back.
Two more quick checks while you are there, because they fail silently:
# When does the TLS certificate expire?
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -dates
# Is the domain’s e-mail set up so messages are not marked as spam?
dig +short TXT example.com
dig +short TXT default._domainkey.example.comAn expired certificate takes the site down; missing mail records send order confirmations to spam.
3. Data, and getting it back
Are there backups, where are they, how old is the newest one, and has anyone ever restored one? A backup that has never been restored is a hope, not a backup.
The check is a restore drill. Take the latest backup, restore it somewhere that is not production, start the application against it, and look at yesterday’s orders or records. For a PostgreSQL database it looks roughly like this:
createdb restore_test
pg_restore --no-owner --dbname=restore_test latest.dump
psql restore_test -c "select max(created_at) from orders;"Three numbers come out of the drill, and they are worth writing down:
- How old the newest backup was. That is how much data the business would lose if the server disappeared now.
- How long the restore took, from "we need the backup" to "the application works". That is how long the business would be down.
- Whether anything was missing: uploaded files, which are often backed up separately or not at all, configuration, the secrets needed to start the application.
If the drill fails, fixing backups comes before every other task on the list, including the urgent one.
How old are the parts?
Before the first change, it helps to know how old the parts are, because an update pulled in by a small change can bring a large one with it:
- the language and runtime versions, and whether they still receive security updates
- the framework version, and how far behind the current one it is
- the libraries, and which of them have known vulnerabilities
Most ecosystems have a command for the last one:
npm audit # Node.js
composer audit # PHP
pip-audit # PythonThe output is not a to-do list to finish in week one. It is a map of where a change might pull in more than it seems to, and of what needs planning.
4. The first safe change
The first change should be small, reversible and easy to check. Its purpose is not the change itself but proving the path: that a change made in the repository reaches the live system the way we think it does, and can be undone the same way.
Good candidates:
- a harmless text change on a page nobody depends on
- a dependency update with a clear changelog and nothing else in it
- adding the error logging that turns out to be missing, which pays for itself immediately
Make the change, deploy it through the path found in check one, confirm it on the live system, then roll it back and confirm that too. Only when that path works in both directions does the change the business is actually waiting for, that failing payment page, get made. It usually goes faster for having waited, because by then there is a staging copy to try it on and a known way to undo it.
What you should end up with
The result of the four checks should be written down, briefly, in words the business owner can read. A page or two with:
- what was found, check by check
- what is at risk right now, and why, in order of urgency
- what should be fixed before new work starts, and what can wait
- the first real change, and how it will be tried and undone
This is also what a new supplier should hand to you if you ask how the takeover went. If they cannot, the checks probably were not made.
- No copy of the live system to try changes on.
- Changes uploaded to the server by hand.
- Accounts, and especially the domain, in a former supplier’s name.
- Backups nobody has restored.
- Secrets written into the code.
- Scheduled jobs nobody knew about.
- No error logging, so problems are reported by customers.
None of these is unusual, and none is a reason to panic. Each one moves a task up the list.
Inherited a system nobody wants to touch?
Tell me what it is and what it does.