Welp … sometimes “stuff” happens … and you find yourself having a really bad day. We’d like to believe that every database is well configured from the start with optimal log rotation, correct alerting of high CPU consumption and cache hit ratio monitoring. By Elizabeth Christensen.
One frustrating thing that can happen to your database has nothing to do with performance and queries. Rather, it’s running out of disk space to store data. Today we’ll drill into some good practices you can put in place to help prevent such from happening. And if it does, show how to get you out of a bind as quickly as possible.
The advice here is aimed at someone in a production situation prioritizing minimizing data loss:
- Backups: The most important thing
- Full disk
- A brief overview of how Postgres WAL archiving works
- Broken Archives
- What NOT to do:
- Never remove WAL
- Don’t immediately overwrite the existing data directory with a restore from a backup
- Don’t just resize in place
- What you SHOULD do:
- Take a filesystem-level backup right now
- Create a new instance (or at least a new volume) with sufficient space
- Fix the underlying issues
Important thing is to remember that Postgres has tools to help you recover quickly and efficiently. Nice one!
[Read More]