As our platform grows, we also have to scale up our global Uptime Monitoring infrastructure along with it.
We usually perform one new deployment per year, if there are no unexpected events.
In this article, we’ll go through the entire process of adding or removing nodes from our Uptime Monitoring infrastructure and explain how these changes would impact the small percentage of our users that decide to whitelist our monitoring IPs in their firewalls. The majority of our users, that do not whitelist our monitoring IPs, are not impacted by the Uptime Monitoring network changes.
1. Scouting
Our team will carefully consider new providers when choosing where to deploy the new monitoring nodes. We usually try to stick with the same hosting providers in all/most of our uptime monitoring locations, if possible. So we’re always on the lookout for hosting providers that offer a wide variety of geo-locations, that’d be suited for our needs. We look for well-established providers, that offer at least 1Gbps connections and at least the minimum specs needed for our uptime monitoring servers.
2. Testing
Once we have a few providers on our short-list for a new deployment, we start purchasing the new nodes and take them through different test runs on development environments, in order to make sure that both the nodes and their network are suitable for our uptime monitoring needs.
3. Adding the new nodes to our Uptime Monitoring IPs list
As soon as we decide that a server will be part of our next deployment, we’ll immediately add its IP address to our Uptime Monitoring IP addresses list. Even though the new node isn’t going to be used for live monitoring until Step 5, it’s best to have the new deployment IPs out on our IP lists as soon as possible. This is done so that any of our new users that decide to whitelist our IPs would have much of the new deployment (if not all) already whitelisted on their firewalls, when the deployment goes live. This is the reason why you might already have the newly deployed nodes whitelisted in your firewall when you receive our announcement email from Step 4. Some of our newly deployed nodes can sit on the IPs list for months before being announced and before going live, but not all of the new nodes are added to the IPs list at the same time.
4. Announcing the network changes
At this point, we’ve found all of the new nodes that we needed for our new deployment, and everything is ready to go live. All affected users are being sent an email notification regarding these changes; you will receive this email if you have at least one uptime monitor checked from any of our monitoring locations that suffer network changes (i.e.: nodes added or removed), and if you have “Receive our Network Changes Emails” setting enabled in your Account Settings (this setting is enabled by default on all accounts). The announcement email will contain a date after which the new deployment nodes will start going live. The users that have previously whitelisted our monitoring IPs in their firewalls are urged to perform the needed changes to their firewall until that set date. The majority of users, that have not whitelisted our monitoring IPs in their firewalls, don’t need to take any action.
5. Going live
Starting with the date mentioned in the announcement email from Step 4, we’ll gradually start using the new monitoring nodes in the live uptime monitoring environment. Not all of the new nodes are going live at once. As each new node goes live, our system will monitor and notice if this particular node is experiencing more errors than the old nodes within the same location. This usually means that some users that had previously whitelisted our monitoring IPs have not yet whitelisted the new monitoring nodes.
6. Subsequent email warnings
Our system tracks the errors encountered from all of our uptime monitoring nodes, and it will notice if your uptime monitors are only failing when checked from the newly deployed nodes. If this is the case, it’s most likely that you’ve previously whitelisted our monitoring nodes, but have not yet updated your firewall to include the newly deployed nodes. Our system will send out one subsequent email warning regarding this, for each monitored location where such issues are detected on any of your uptime monitors. This system ensures the fact that the end-user is completely warned that there’s an issue between their monitored target and the newly deployed nodes.
7. Finalising the new deployment
Once we’ve cycled through all of the newly deployed nodes and warned all of the affected users, the new uptime monitoring nodes become a permanent part of our global uptime monitoring infrastructure.
Unlike most other uptime monitoring providers that only mention such network changes on their blog or Twitter feed or don’t announce changes at all, we take great care to make sure that our end-users know about these changes, even though just a very small percentage of them actually whitelist the monitoring IPs in their firewalls. As you may already know, whitelisting of the monitoring IPs in your firewall isn’t needed in most cases, and should only be done if you’re monitoring a private access server/port, or if you have any monitoring issues otherwise.