Fatal error: Allowed memory size of 33554432 bytes exhausted
(tried to allocate XX bytes) in /some/file on line XX

This is a general PHP error, but the exact amount (33554432 bytes) occurs very frequently on WordPress sites. Here’s what’s happening, and how to fix it.

Every installation of PHP has an ini value called memory_limit, which hardcaps how much memory PHP may use on any given request. There are other ini settings that dictate the memory usage for specific actions, like post_max_size, but this one is universally applied; you can’t load anymore than what this value dictates.

I do a lot of stuff with files in PHP, so I blow this value out from it’s default of 128 MB up to 2 GB. I set this globally in my php.ini file like so:

memory_limit = 2G

I could also set this value from inside an .htaccess file:

php_value memory_limit 2G

Since this value was set before the site even loads, imagine my surprise when PHP is telling me that the limit is now 33554432 bytes, or 32 MB. The limit was changed by a define in WordPress called WP_MEMORY_LIMIT.

Here’s where things go awry. WordPress tries to check the existing memory_limit before applying it’s own terms, however the value must be set in terms of MB. Look back at my example settings; the PHP memory shorthand allows me to set in gigabytes using a “G“. But WordPress does a straight integer comparison to figure out if the existing setting is more/less than 32 MB. Here’s the offending code, a portion of an if statement on line 42 in wp-includes/default-constants.php

( (int) @ini_get('memory_limit') < abs(intval(WP_MEMORY_LIMIT)) ) )

What you end with in that sample is essentially if (2 < 32), which resolves true and thus WordPress applies it's own default limits.

You have three options to rectify the situation: two easy fixes, plus one badass "server-baller" fix. First the easy ones. As WordPress outlines in the article on increasing memory, simply define WP_MEMORY_LIMIT in wp-config.php using MB syntax. If I wanted to set it to 2 GB, it would look like this:

define('WP_MEMORY_LIMIT', '2048M');

Your value will be honored as long as it is an integer value greater than 32, but could break if WordPress changes it's memory size comparison calculation.

Alternatively, I could change my the memory_limit setting directly in php.ini or .htaccess to use MB syntax:

memory_limit = 2048M

[.htaccess]
php_value memory_limit 2048M

Realistically I would probably edit php.ini. Well, that is, I would, if I didn't know about the badass server-baller third option, which overrides php.ini AND prevents WordPress (or any other site for that matter) from making any changes to the value.

Look back at my .htaccess setting, and notice that I set the value using the command php_value. PHP has a similar command called php_admin_value, which you can use for setting all the same stuff as php_value.

The difference: php_admin_value can only be set within Apache configurations, i.e. it cannot be used inside of .htaccess files. Furthermore, any value set using php_admin_value cannot be overridden by any later calls to php_value or ini_set. You can override php.ini defaults, and lock that value in for the remainder of the request.

So back to our WordPress example, I opened the VirtualHost for frankkoehl.com and entered this line:

php_admin_value memory_limit 2G

Now the site will always allow a memory limit of 2 GB, no matter what. WordPress can try and set whatever value it likes, it will never be recognized.

Note that this function requires that you have access to Apache configurations, at least the ones running your site. Most shared hosting packages won't offer this level of access.

It's worth mentioning that WordPress sets a default value of 32 MB, lower than PHP's own default of 128 MB. I disagree that such a low limit is necessary, but they are trying to be good stewards to servers everywhere, so I can see where they're coming from.

I still have no idea why this memory issue popped up in the first place, but it gave me an excuse to share a cool configuration option that has saved me from having to scour code on several bloated, crappy sites to find obscure ini settings. I have a recent real-world example you can read about on Stack Overflow.

However, by default Apache rewrites will escape the hash (#) symbol, converting to its hexcode equivalent %23.

So this rewrite…

RewriteRule ^/foobar/?$ /some/other/url#foobar/ [R=301,L]

will produce this URL…

http://example.com/some/other/url%23foobar

In order to make this rewrite work, we must prevent Apache from escaping the hash mark by using the noescape flag. A little tweak to the rewrite, and we’re good to go…

RewriteRule ^/foobar/?$ /some/other/url#foobar/ [R=301,L,NE]

See that little NE on the end? That’s all we need to make rewrite anchors work.

Documentation on all the rewrite flags can be found in the Apache docs for the RewriteRule Directive.

This problem always drove me nuts when creating MySQL database backups. I want to make a tar backup of /var/lib/mysql/database_name and call it database_name.tar.gz. If I then unpack the tar under /root, I’ll have a bunch of worthless subfolders to drill down through: /root/var/lib/mysql/database_name. All I really want is the database_name folder, the /var/lib/mysql/ parent directories can all go away.

Here’s how to get rid of those pesky parent folders once and for all. Before running your tar command, you must first use cd to move into the parent directory that contains the folder you want. Then, within your call to tar, leave the parent folders off your target files/folders.

Here’s the command sequence for previous example:

cd /var/lib/mysql
tar -czf /any/folder/you/want/database_name.tar.gz database_name

The resulting file will unpack directly into a single subfolder called database_name.

If you run your tar process as part of a larger scripting event, your script might not run/maintain a shell interface. In these cases, the cd command will be executed in a vacuum, and won’t have any effect. I run into this problem when I try to perform shell magic inside of PHP scripts.

Fortunately, there’s an easy workaround: you can append multiple shell commands together using &&. With &&, each command will be executed and completed before the next one begins, mimicking a person typing out commands one line at a time.

Here’s what our MySQL example looks like as a single line:

cd /var/lib/mysql && tar -czf /any/folder/you/want/database_name.tar.gz database_name

Enjoy your tar backups sans worthless parent directories!

Here’s how to enable key authentication for the root user on your remote Linux server. Once you’ve gone through the process you’ll be able to easily replicate it for any other user accounts that require SSH access.

1. Create the key pair from your server
I don’t know what your local system looks like, nor do I care. We know the server has everything you need since it already has SSH installed, so let’s just do the heavy lifting from there!

SSH into your server as root, and run the following commands:

cd ~
mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa

Your output from the call too ssh-keygen will look something like this:

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

You may enter a passphrase to encrypt the key, but keep in mind you’ll have to enter that passphrase every time you wish to connect, effectively defeating our purposes for using keys in the first place. Just leave the password lines blank when prompted.

[Sidebar]
In practice, the connection procedure for passphrase’d SSH keys and standard username/password combo look exactly the same: connect to server > enter password > get access. However, using keys with a passphrase is far more secure than a simple username/password challenge. A would-be attacker needs the key file AND the passphrase in order to get in. This extra layer of connection requirements raises the security bar significantly, especially if you’re careful about where/how you store the file and passphrase.
[/Sidebar]

2. Which keys go where?
If you navigate to /root/.ssh, you’ll find 2 files: id_rsa (the private key) and id_rsa.pub (the public key). Here’s the part that causes the most confusion for newcomers to key-based authentication:

The public key (id_rsa.pub) belongs on your remote server.

The private key (id_rsa) resides on your local client, or whatever machine is connecting to the remote server.

It seems counter-intuitive for some people, espcially if you’re familiar with SSL. The SSH connection process has the remote server giving offering up the public key for every connection request, placing the onus on the client (i.e. you) to respond with the proper private information.

3. Install the public key on the server
The public key is on the server, but not in the right place yet. We need to place it where SSH will look for it when a key-based connection is made.

cd ~/.ssh
cat id_rsa.pub >> authorized_keys2

SSH looks for potential public key matches in the file authorized_keys2. The file might be called authorized_keys (no number two) on some systems. The cat command appends the entire key to that file, or creates it if it doesn’t already exist.

At this point the file id_dsa.pub is no longer needed. For the sake of security, you may tuck it away somewhere for safe-keeping, and delete the server copy.

4. Install the private key on your local client
Pull id_rsa (without the “.pub”) down to your local machine using your file transfer method of choice. Once you have a local copy, you should remove the one located on the server, and throw another copy in a dark corner somewhere for safe keeping. This is especially important when setting up key-based authentication for the root user: if you lose the key and disable password logins (discussed below), you will be unable to connect to your server via root. No bueno.

At this point you need to add the key to the SSH connection details of whatever SSH client you use. This fairly straightforward with nearly any SSH client, save one: PuTTY. For whatever reason, the PuTTY team uses a special format for SSH keys, and we need to convert your private key to that format before we can connect.

5. Convert SSH private key for PuTTY on Windows

1. Open the PuTTY Key Generator. The executable is called puttygen.exe
2. From the Menu select Conversions > Import key
3. Browse to your local copy of id_rsa and choose Open.
4. Click the Save private key button, listed under Actions on the lower portion of the window.
5. Save the resulting .ppk file in a permanent location (PuTTY will look for the file in this location every time you connect)
6. Open PuTTY, and load the saved session for your server
7. Navigate to configuration menu to Connection > SSH > Auth
8. Click the Browse button under Private key file for authentication, find your new .ppk file, and click Open
9. Be sure to save the changes made to your PuTTY session!

There are plenty of tutorials on PuTTY around the web should you require more information.

6. Harden SSH by disabling password authentication
Using keys-based authentication allows you to enforce better protection on your server from SSH-based attacks. Test your connection before completing going any further.

Connect to your server using your fancy new SSH key, and open /etc/ssh/sshd_config in your favorite editor. Find the entry for PasswordAuthentication, or add the line as follows:

PasswordAuthentication no

Save and close the file, then restart SSH (on Debian/Ubuntu systems the command will be /etc/init.d/ssh restart). That will prohibit all SSH connections allow you to still connect as root, but only using your key. Again, make sure your key-based authentication is working properly before applying this change.

Unfortunately, I went to the PHP IDS site today to grab the latest version, only to discover the site is either temporarily down or closed entirely. The error message is in German, but Google Translate gives the following translation:

This server is no longer in operation.

Please tell the operator that the DNS on the new IP 46.4.40.248 convert their.

schokokeks.org

Anyone know what’s going on?

Update April 4, 2011
Looks like they had some domain issues, and are now back at phpids.org. The lead developer of PHPIDS, .mario, has a post detailing the shutdown and return.

Let me first begin by acknowledging that, yes, requiring Javascript flies in the face of site accessibility. Policy wonks for this sort of thing will tell you that Javascript should never be a necessity, that a page should “degrade gracefully” when Javascript isn’t enabled. This mindset is complete crap, in my opinion. Every site has a minimum browser requirement, and Javascript is built into every modern browser. I fail to see why we are still separating the two, and can make a darn good case for expecting Javascript on the user’s end. But that argument’s for another day. If you agree with me, read on.

The example
This example assumes you understand HTML, CSS, and Javascript, and makes use of jQuery, so you may have to adapt it to fit your Javascript framework of choice.

This technique could be applied across the board to an entire site, but let’s start with a single page. A login or account registration is a great choice, since they are already gateway pages to your site. Start by adding the following CSS definition to your page or stylesheet:

.enable-javascript{
  display:none
}

Next, open the HTML for your page and add the enable-javascript CSS class to the top level container containing the body of your page. You should try to leave the header and footer (and their accompanying navigation links) visible and accessible. For example, after my header DIV, the body of my pages are typically wrapped in something like this:

<div id="content" class="enable-javascript">
  [...]
  <div class="base"></div>
</div>

At this point, you’re looking at a blank page. Now let’s give our non-JS users something to look at. Place a <noscript> code block outside and above the body that you just hid. You can use CSS and elaborate HTML inside a <noscript> block, so feel free to make it look good. I actually duplicate my body wrap HTML so the look is similar:

<noscript>
  <div id="content" class="enable-javascript">
    Javascript is not currently enabled in your browser. <a href="http://www.google.com/support/bin/answer.py?answer=23852">You must enable Javascript</a> in order for this site to work properly.
    <div class="base"></div>
  </div>
</noscript>

Finally, make sure the jQuery library is loaded, and add this bit of code to the Javascript for your page:

$(function(){
  $('.enable-javascript').show();
});

What’s happening
We’re using the absence of Javascript to halt the user’s access to a page. By default, page content is hidden via CSS, and we use Javascript to reveal it. If the user’s browser does not have Javascript enabled, the reveal never happens. Visibility for the <noscript> block is handled automatically by the browser, depending on whether Javascript is running or not.

A live example
I’m using this technique on the Fwd:Vault login page. In order to see it in action, you obviously have to disable Javascript. You can use the Google support link to turn Javascript on, and Firefox users can use the Web Developer toolbar add-on to do this a little more efficiently.

A couple caveats
Javascript is a client-side language; it actually lives inside the user’s browser, and runs from the user’s machine. Javascript code essentially tells the browser to manipulate the static HTML of the page, and the server isn’t involved in the process at all (AJAX is something of an exception). This means that there is absolutely no way for us to look for Javascript on the server side using PHP or other server-side technologies.

In addition, since Javascript control is firmly in the hands of the user, you cannot trust Javascript for security. Your average user will be stopped dead with this technique, but hackers and code monkeys can easily circumvent it. This is the case with all Javascript effects, and all have the same countermeasure: scrub all incoming data regardless of Javascript rules enforcement.

Feel free to post comments or questions, and please link your own examples if you put this technique to use!

Your instinct may be to dismiss my decision as laziness, but hear me out. I built most of the base site with just 1 state: free (remember that unlimited free beta period last year?). That allowed me to — rightly — focus purely on features, functions, bugs, etc. Dealing with subscription tiers at the same time would have clouded everything, slowing everything down and likely leading to more rewriting. Staying focused allowed me to get the cornerstone stuff right before building on top of it.

I applied the same thought process when it came time to offer paid options. The game plan has always been to have three paid options, plus the free account. However instead of initially coding four possible user states, I started with just two: free or paid.

This makes my job as a developer much more focused. There’s a LOT of logic in a service like Fwd:Vault focused explicitly on subscriptions: access permissions, showing/hiding upgrade options, setting quota restrictions, security checks to prevent hackarounds from unscrupulous users. The functionality of almost every page is affected by the user’s free/paying status, and don’t even get me started on the work it takes to process credit cards. You have to be doubly triply careful when dealing with people’s personal data like that. On and on. Getting the basics in place takes a lot of forethought and coding.

So instead of thinking about all this stuff in four dimensions — free, option 1, option 2, option 3 — I can cover most everything in just two — free or paid — and then come back later to fill in the holes for the other tiers.

Complexity is your enemy as a developer. Each task must be as tightly focused as possible. The tighter your focus, the less chance you’ll have to introduce bugs. Adding more later may require rewrites, but they are far far easier than rewriting the big sloppy mess you get when biting off more than you can chew.

With the basic subscription and tier logic in place, it’s a far simpler matter to expand the options out to infinity (though we’ll start with four). Expect to see the new pricing options in a few weeks.

Looking for more to read? There’s a new post on the Fwd:Vault Blog that details the most unobtrusive disk defragmenting process I’ve seen (that I also use for my own systems).

Wish I could take credit for it, but at least it comes complements of fellow PSL member Eight Eleven.

Update: I complimented the author Aaron, and the following conversation ensued…

Aaron:
I’m going to end up in a lawsuit with Toyota for sure on this one.

Frank:
As a design firm, what a boon for business THAT would be!

Aaron:
Yeah, I can see the headlines already: “Japanese automaker Toyota files lawsuit against New Jersey based Advertising and Marketing Agency, Eight Eleven Inc., then mid-suit, hires them to execute a new branding campaign.”

Fantastic.

The show’s writers and producers get extra bonus points from me for taking on the issue of healthcare, and having the stones to come right and say, “This is a business, and we need to make money.” They even couch the message in a scenario that most people should be able to appreciate from both sides. Very timely given all the discourse surrounding healthcare reform.

44 minutes, well worth it.

Let’s start at the source. The Firefox team made a crucial usability decision, which is at the heart of the problem. They wanted to allow the user to recover any page that may have recently opened. So by default, Firefox keeps navigation history for all your open tabs, plus the last 10 tabs that you closed. The navigation history for each of those tabs — both open and closed — can hold up to a maximum of 50 pages (i.e. the number of URLs you can traverse purely through the Back/Forward buttons)

With no limit to the number of open tabs, plus the high limit on the Back/Forward navigation, it’s easy to see why Firefox slows to a crawl. If you do a lot of browsing in a lot of tabs, your memory disappears in a hurry. Managing all that extra memory causes the processor to work overtime to keep Firefox hippo moving.

There are two ways to fix this issue in a pinch. First you can simply restart the browser, making sure that it doesn’t save your tabs (you are prompted to save tabs at close by default). Second, you can clear the Recently Closed Tabs to eliminate a portion of the tab history bloat (History > Recently Closed Tabs > Clear Closed Tabs List).

For a more long-term solution, we need to mess with the system settings. Type about:config in the address bar to bring up Firefox’s complete configurations list. The latest Firefox versions present you with a warning before opening the page.

A warning: this page handles everything in your browser. Everything. Don’t mess with stuff if you don’t know what you’re doing.

In the “Filter” textbox at the top, enter

browser.sessionstore.max_tabs_undo

This setting controls how many closed tabs to track. Less old tabs = less memory usage. Double click the lone entry in the list and change the value from “10″ to “5.”

Back to the filter box, enter

browser.sessionhistory.max_entries

This setting controls the navigation history limit. Double click the entry and drop the value from “50″ down to “25″.

Close the about:config tab and restart your browser.

Your mileage on these tweaks will vary depending on your system specs. If you can go a day of heavy browsing without hitting the creep, slowly increment the settings back up, until you hit the sweet spot.