This post will provide a series of questions on scaling an application with each question building off the one before it. Try to answer the questions first and then see how your answers compare!
Let’s begin.
How would you handle a server crash?
Restart the server.
You restarted the server due a crash, but your db is empty now. How do you handle this in the future?
You routinely backup your db to a cloud service like AWS S3.
Backup Strategy:
Keep the backup db on a different server. If it were on the same server it would also be cleared and it would defeat the purpose.
Also, routinely verify the backups to make sure they are restorable. Verify = the data is accurate, is not corrupted, is able to be used for db restoration.
You have your backup db. But users are getting 404 errors. What do you do?
We know that its a 404 so the application went down → simple immediate fix is to restart the application on the server.
Long term fix:
Check the logs to see where the error occurred?
no logs? configured to writing just the standard output (console.log) ?
create logs. Have them written to files so we can access them upon errors.
Also, implement auto-restart logic so whenever the application is down we bring it back up. Allows users to continue to use the app without us having to manually go in a restart it.
After some time your app produces 500 errors. What do you do?
Check the logs to see where the error is → errors indicate the database connection pool has been saturated.
The database connection pool has been saturated. What is the database connection pool? Why is it saturated? How do you fix this?
Connection pooling: a strategy to keep the database connections open and reused because connecting to a db can be an expensive task.
The connection pool has a number of connection nodes that are available for clients to latch onto and establish a connection with the db. the connection nodes are always connected to the db, they’re like open channels that allow us to maintain access to the db without having to close them and pay for opening them again.
As clients connect to the connection pool and leave, anyone else can come and reuse the connection node.
Why is it saturated?
It could be that some clients haven’t ended their connection and are taking up all the connections in the pool, preventing new users from accessing any of the connections.
This is because we did not configure the database properly for our needs.
How to fix:
So we go back to the db and configure it from base settings to our settings → increase the pool size for the db and that should resolve the issue.
That’s the end of part 1. Next, we’ll see what happens when we expand globally! Till then — happy hacking!