Hit soft quota, removed 50% of the total filesize...still hitting quota?

[csammis]

Yesterday I hit the soft quota (again - I would really like a facility to automatically remove all logs so that they don't keep taking up space, does one exist?), so I removed ~185MB of photos that weren't doing anything useful. However, although the Disk Usage Breakdown report notices that they're gone, the dashboard disk usage graph still shows the space as used, and my blog (http://csammisrun.net) won't run. Is something else going on here?

[csammis]

Looks like it fixed itself, and it just took a while. Carry on here :)

[dkozinn]

Regarding log rotation, if you're talking about the web server logs, you can configure those from the Esprit control panel. Take a look under the "Web Server" heading, there's an item called "Log Rotation". You can tell it how many archived versions of the log to keep and how big each log is allowed to get, plus how often to do the rotation. Hopefully, that's what you're looking for.

[Matt]

The reason the blog wasn't immediately working was due to how MySQL (sometimes incorrectly) handles connection backlogs. Your soft quota was set much lower than it should be thanks to some terrible ideas by Ensim, but I'll get to that shortly. Once you hit your soft quota, you have a seven day grace period to clear up space before it is enforced. Upon enforcement, soft quotas act just like your hard quota preventing you from further writing. MySQL sees this as the disk being full and delays writes to the table for I believe 10 minutes. After 10 minutes it retries and either succeeds or fails and flushes out the connection. 10 minutes is a sizable delay and that's what you perceived to be a continuance of the problem. It was actually resolved once you deleted some files, but the backlog had to clear itself out (writes lock the table). To make matters even worse, I've seen some wonky code in MySQL 5 that causes the exclusive table locks to apply globally. This in turn locks every table in every database, not just the database that ran over quota. The good news though is that happens rarely.

Now, as for the low soft quotas? This is a fun one that could be answered with several expletives cursing the outsourced boneheads over at Ensim that wrote the control panel that esprit is slowly replacing. MySQL can corrupt a table due to a failed write, but this has been fixed by implemented the delayed write mentioned previously. Now what was Ensim's solution to customers who knowingly run over their disk quota and neglect it 100%?

A soft quota set to 80% of your hard quota... just pause and absorb that number. This means to prevent corruption, if you hit 800 MB out of 1000 MB on your site, you would have seven days to free up space (but you have 200 MB free!) before the soft quota transforms into the hard quota. Ensim was thinking ahead of the pack though! See, they wanted to warn you ahead of time if you are close to infringing upon your 80% soft quota, so they added a notification script that runs nightly. There's one huge problem though... these e-mails are sent to root@<your domain>. No, they're not sent to the administrative contact in the control panel, but rather just to root@<your domain>. I can understand this reasoning 100%, because everyone has a root@<your domain> and uses it every day.

Fantastic foresight, guys. :rolleyes:

The good news is I tracked down the source of the 80% quota and adjusted it back to 95% a few weeks ago. That 95% will tie into the early warning notification system in esprit for disk quotas, but it's a while from being implemented. If you are teetering on 95%, chances are you're going to run over pretty soon anyway. Hopefully that value won't cause any issues unlike the 80% :mad:.

I would probably have enough material to write a book on all of the terrible ideas and performance problems that are in the retail version of Ensim. If you're on Gauss, then you already read about the experimental monolithic Apache configuration. I'm going to change subdomain behavior around a bit too to cut down on memory usage in a couple of weeks. Each Apache process is approximately 100 MB.

Edit: oh and that quota setting utility that's called whenever a site is added/removed, guess where it resides? /usr/bin/. Not /usr/local/sbin/ or /usr/sbin/ like its brethren edquota, repquota, and setquota or even /usr/local/bin/. No, it sits in /usr/bin/.