Tag Archives: apache

APC usage and hit chart

Alternative PHP Cache (APC) on Debian 6

APC usage and hit chartAfter spending some time reviewing way to increase WordPress performance in ways beyond simple file caching I decided to give APC (Alternative PHP Cache) and extended try. I use mod_fastcgid on my server, as opposed to mod_fastcgi. Web searches indicated that mod_fastcgi would be a better choice with APC since it could share the same cache between processes. With mod_fastcgid each process has its own cache, which would also mean more memory usage. I have a relatively small server, a 512 MB VPS. But in looking at free memory it appeared I’d be OK as long as I was careful and had some upper limits. Also, I had has some issues getting mod_fastcgi to run in the past so I didn’t want to try that again.

Why APC

APC is an opcode cache. Opcode caches will cache the compiled php script in memory so it does not have to be interpreted each time it is called. WordPress is built with PHP so this should help performance. (Technically “compiled” is probably technically wrong but does get the point across.) Other opcode caches are XCache and eAccelerator.

eAccelerator hasn’t been updated in a couple years so I was hesitant to try it. APC is by the same guys who develop PHP and it’s supposed to be rolled into PHP 6. So I decided to give it a try first. I may give XCache a try after running APC awhile. Although, if I’m honest, I probably won’t bother if APC works OK.  They can’t co-exist so I can only install one at a time.

The one reason not to use APC is memory. I’ll be improving cpu utilization and improving performance at the expense of memory.

Throughout this article the terms cache and caching refer to the opcode caching provided by APC.

Installation

APC is in the Debian 6 repositories so it can be installed from there. I use aptitude, so I run:

aptitude install php-apc

Once it’s installed I reload Apache and APC is running. No admin interface is running so I extract the apc.php file by running:

gzip -dc /usr/share/doc/php-apc/apc.php.gz > /path/to/my/mngmt/website/apc.php

The /path/to/my/mngmt/website is a website I have that’s a catch-all for various server related pages. Access to it is restricted and since it does include some version information and files paths you may want to restrict access. I can open this page and get statistics about APC usage.

When I opened apc.php and did a version check I saw that I was several versions behind. I install 3.1.3p2 from 2009 and the latest stable version was 3.1.9 fro May 2011 and there was a beta version from April 2012. APC was working and limited testing didn’t show any problems but I decided to update.

Update Through PECL

PECL is the PHP Community Extension Library so its installer will be installed along with packages it needs to compile APC. Finally APC will be installed.

sudo aptitude install php-pear php5-dev apache2-dev

Once these are installed I can install (upgrade) apc

sudo pecl install apc

Reloading Apache finishes things off. Because I had first installed apc through aptitude I didn’t have to do any manual edits.

Configuring APC

Configuring APC is where the real work is needed. I did several iterations testing different configurations. At this point I was testing for performance, I was looking to see what worked and what didn’t.

The apc.ini shows the settings I was most interested in and what I’ve currently settled on. Memory is the biggest issue with APC. As I mentioned, each FCGID process creates it’s own cache. I’ve configured my server for a maximum of three processes so I can control memory usage. The easy way to determine how much memory to allocate is to divide my free memory (without caching enabled) by 3 (the max number of FGCID process I’d have) and leave a little for OS buffers and breathing room. For me this worked out to about 100 MB (I had about 350 MB free. But that calculation is a limit, which doesn’t mean it’s a good size.

I’m going to jump ahead and show the configuration I settled on.My /etc/php5/conf.d/apc.ini contains the following entries:

extension=apc.so
apc.enabled=1
apc.cache_by_default=0
apc.filters=”+(theosquest.com|stats.rrn.co)/.*”
apc.shm_segments = 1
apc.shm_size=90M
apc.slam_defense=0
apc.stat=1
apc.ttl=0
apc.user_ttl=0

The first two lines tell PHP to load the APC extension and enable caching.

I started off with a 256 MB cache and let it cache all my sites. This was much too large so I’d have to monitor usage in case things started swapping to disk. Cache usage seemed to level off at around 90 MB after a few hours. I did go through a load the home page of all my sites (many are test sites or receive little traffic). This gave me an rough (and lowball) idea of what I’d need to cache everything. So this was right at my 90 MB estimate. In practice a 90 MB cache size resulted in a lot of cache fragmentation and cache reloads which isn’t an optimal situation.

I took the approach of reducing my FCGID processes to 2 and increasing the cache to 128 MB. This provided better cache performance but shortly after midnight I received a series of “[warn] [client 69.164.211.225] mod_fcgid: can’t apply process slot for /home/rayn/public_html/fcgi-bin.d/php5-default/php-fcgi-wrapper” errors. That IP address is my server so some daily maintenance scripts needed more PHP processes than were available. Later in the morning one visitor experienced a timeout error. It may have been unrelated but timeout errors were previously unheard of. So I switched back to three FCGID processes.

At this point it was obvious I had to cache less than everything. There’s only two sites that have enough traffic to benefit from caching so I decided limiting caching to them. I handled this through the next two lines. apc.cache_by_default=0 turns off caching for all sites. It now needs to be turned on explicitly. It’s turned on through the apc.filters line. The + means the filter is to add the matching files to caching. (By default the filter removes matches from caching.) The syntax is a POSIX regular expression and can be comma delimited. I had trouble getting a comma delimited list to work although all indications are that it should. So I ended up combining them into this one expression. Here’s a regular expression cheat sheet. I cache this site and one subdomain on a second site.

Caching doesn’t have to be enabled for a site in order to use it with a WordPress plugin that supports APC. For example, W3 Total Cache will use the cache even if it’s not enabled for the site.

I specified apc.shm_segments = 1 even though 1 is the default setting. I don’t want later version to change the default and start eating more memory. I specify apc.shm_size=90 to create a 90MB cache per FCGID process.

I also specified apc.stat=1 even though that’s the default. There’s debate about whether setting this to 0 will improve performance. A setting of 1 means that APC will check each script on request to see if it’s been modified. If it’s been modified it will be re-cached. A setting of zero disables this feature, The official documentation recommends setting it to 0 on a production server. A setting of 0 can cause problems as PHP changes (such as WordPress plugin updates) would go unnoticed. Once things are solid I may change it to 0 to see if there’s an improvement or any problems. My initial feeling is any improvement would be unnoticeable for my site and wouldn’t be worth the hassle of needing to manually flush the cache when I make changes. I’m also concerned it will cause problems with some plugins.

To make sure memory usage doesn’t get out of control I’ve also limited my FCGID processes to only three and I don’t allow them to spawn any child processes. I also don’t terminate to FCGID processes since this would clear the cache. This could be a potential memory problem so I’ll have to monitor it.  The following settings are specified in my /etc/apache2/conf.d/php-fcgid.conf.

DefaultInitEnv PHP_FCGI_CHILDREN 0
DefaultInitEnv PHP_FCGI_MAX_REQUESTS 0
FcgidMaxProcesses 2

This should keep memory usage below the physical memory in my server. PHP_FCGI_MAX_REQUESTS is set to zero based on this in the Apache mod_fcgid documentation:

By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem.

I’ll have to monitor this. It’s possible the FCGID memory could keep growing until physical memory is filled.

PHP_FCGI_CHILDREN is set to 0 to prevent children from being spawned. From the Apache documentation:

PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled.

Additional Notes

The screenshot below (click to open full size) shows my status (using apc.php) a couple hours after the caching started. The screenshot was done when the cache was still configured for 256 MB. I am running a unique cache for each fdcgid process so refreshing the page moves between the instances (but randomly). The size of the three caches stay pretty close to one another but they are clearly unique caches.

APC Info Page

When running top each of my three fcgid processes levels off at about 25% memory usage. So if my math is right this is 75% of 512 MB. Yet free (and top) show 350 MB still free. If each was allocating the full 256 MB to its own cache then it would be well over 100%, assuming free and top saw the shared memory it was using. So I’m assuming free doesn’t see the shared memory and reports is as available. In my limited memory this makes sense since the shared memory is still available to any application. The actual usage of my three caches never grew about the available free memory.

APC does require some work to configure and optimize it. To start with I’d enabled it for all my sites and PHP. This resulted in a lot of cache fragmentation and a few cache reloads after a few hours.

Benchmarks and timings on a test server are fine for comparisons, but I’ll need to run a configuration for awhile to see how it does, then start tweaking it. Right now my server seems healthy. Using APC did improve performance over no caching of any type, but I’m not sure of it’s benefit once page caching is used, especially if using a preloaded cache.

Right now my main concern is that I have too few FCGID processes, only two. I used some load testing tools and it didn’t cause any errors. I do have page caching enabled to PHP shouldn’t be needed when serving a page from the cache. I’ll be monitoring my error logs.

Anyone else with experience using APC with WordPress?

Apache logo

Apache File Does Not Exist Error

Apache logoI spent some time over the past week killing some bugs and making some adjustments to my server. One of them was a message being logged to my default Apache error log (/var/log/apache2/error.log). The error was:

[Sun May 27 22:05:03 2012] [error] [client 127.0.0.1] File does not exist: /etc/apache2/htdocs

The error was being logged exactly on every five minute mark, no matter when the server was started. The interesting parts here are that none of my sites are configured to listen on the loopback address (127.0.0.1) although some are set to run on any addresses. Also, /etc/apache2 is not the document root for any of my sites. All of my sites were working and were properly configured. A Google search showed I was not alone with the problem but had no good solutions

What I ended up doing is specifying a default document root for a directory that does exist. I used the default site for the server.

sudo nano /etc/apache2/conf.d/DefDocRoot

I put in one line:

DocumentRoot /valid/path/to/site/public

I reloaded the Apache configuration and the error went away. It seems like there’s a default site of some type configured into Apache. It might be something I configured but apache2ctl –S doesn’t show any syntax errors and all my sites seem fine.

Apache logo

Adjusting Logrotate and Lessons Learned Redux

Apache logoBack in October I had some issues with logs and adjusted logrotate. This weekend I made additional changes while killing some website bugs and resolving annoyances.

By default logrotate appends a numeral to the file name as it’s rotated. (At least on Debian 6). There is a configuration file parameter dateeext that will append the date instead.  There’s also a parameter dateformat if the default extension isn’t suitable.

I also had an issue where logrotate failed when I deleted a test website but didn’t update the logrotate configuration to remove its logs. I added the missingok configuration parameter to handle this in the future.

I also changed logrotate to rotate files weekly or when the size exceeds 100MB. I’ll see how this works out and adjust it if needed.

I also had a self-inflicted wound where I left out the opening curly bracket. This caused it to rotate apache2ctl, causing my apache problems. I was probably sloppy in my testing and made a quick change after testing and somehow deleted the bracket.

To test logrotate the command: sudo /usr/sbin/logrotate -vfd /etc/logrotate.d/config_file_to_test can be used. The –vfd switch means to run in verbose mode, force logrotate even if it’s  not needed, and run in debug mode. Debug mode means no logs will actually be rotated. Sudo is not needed if logged on as the root user.

The file /var/lib/logrotate/status is logrotates memory of when files were last rotated. Deleting a file’s entry from here will force it to be acted upon the next time logrotate runs. It’s also a good place to see what files are being rotated.

So, the logrotated configuration for my websites is now:

/path/to/logfiles/*.log
{
rotate 5
weekly
size 100M
compress
dateext
delaycompress
missingok
nosharedscripts
postrotate
/usr/sbin/apache2ctl graceful > /dev/null
endscript
}

This rotates the logs weekly (or when the reach 100MB) and saves the last 5 logs. I experimented with mailing the logs when it came time to delete them but decided it would be too many emails and it would be easier for me to grab the files once a month. I may increase from 5 the number of logs I save to give me some extra time to grab them.

Tools Tile

What I Use: Web Server

Tools TileWith my recent web server changes it’s a good time to update what I use on my web server. I run the site on a “Linode 512” Virtual Private Server from Linode. I’ve been with them since December 2009 and signed up for another 2 years back in January. It’s been problem free and my initial server stayed online for over 300 days at one point, only getting a reboot when changes made it necessary and I did the reboot. It’s a VPS server without any control panel but it suits my needs. There’s no support beyond the making sure the server is running, but they have an extensive document library. All administration is over SSH and from the command line.

I switched to Debian 6 as the OS with this months move. I’d been running Ubuntu, which is Debian based, so there wasn’t much of a change. I’d been having a few minor issues with Ubuntu so decided to switch. I can’t say of the problems were introduced by the Ubuntu or something else.

I run Apache 2 as the web server. With the latest move I switched from using pre-fork MPM with Apache to using Worker MPM. For PHP is use mod_fcgid and mod_actions rather than mod_php. This certainly uses less memory resources, performance seems about the same although some benchmarks do show improvement.

The site runs on WordPress and uses the Catalyst Theme. I’ve been using WordPress since I first created  a website so I obviously like it. I also like the Catalyst Theme, my main complaint is shared with all framework type themes. There’s some lock-in if heavy use of theme-specific features are used. I use the following WordPress plugins: Akismet (anti-spam), Contextual Related Posts, Fast Secure Contact Form, FeedBurner Feedsmith, Google XML Sitemaps, No Self Pings, Recently Popular, Redirection, Skimlinks, WordPress Popular Posts, WP-Piwik, WP Super Cache, WPTouch Pro. I also use the Google Analyticator plugin but will be dropping it when I drop Google Analytics at the end of the month.

For site analytics I just started using Piwik, an open source counter-part to Google Analytics where I maintain the data. There’s also an iPhone/iPad app for viewing the data. As I write this I still use Google Analytics but plan to drop it at the end of the month.

For DNS I use DNS Made Easy for this websites DNS service rather than the registrar’s own DNS. I find it to be more reliable and has more features.

For backups I use Linode’s backup service to get a full server backup and snapshots before updates. My website files and databases get backed up to my home network using rsync.

Tools Tile

Moving Day – A New Web Server

Tools TileI completed moving this website to a new server over the past weekend. I’m still using Linode as my host and running on a 512 MB Virtual Private Server (VPS). The changes were minor and should be transparent. I moved from Ubuntu to Debian as the OS but since Ubuntu is based on Debian there’s not too much difference. I had a few strange things happen on Ubuntu after updates so I’m hoping for a little more stability without actually having too learn anything new. Of course, soon after going live I had problems which seemed to be related to an update on Debian. So much for stability as a reason.

As I was building the new server I also worked to squash some annoyances and did some testing. I’d been having problems posting updates through xmlrpc (for example, from Windows Live Writer) to all my sites except this one. That problem didn’t exist on the the new server. I also had some scripts which stopped working as expected and these worked fine on the new server.

The major change (relatively speaking) was I switched Apache from using pre-fork MPM and mod_php to using worker MPM and mod_fcgid.  Worker-MPM was the default when I installed Apache on Debian so it was easy enough to get Apache working. Getting mod_fcgid and mod_actions to handle php was a little more complicated but not too bad. This has reduced the memory usage by the server since php isn’t running with each Apache instance. Performance seems to be about the same.

I considered switching to nginx from Apache but I decided I didn’t want to much change and I didn’t want to spend the time learning nginx. Staying with Apache meant that I would be keeping most of my configuration. Maybe sometime in the future.

I only made a couple changes to WordPress itself. There were some updates related to security keys in the WordPress configuration file. It’s been awhile since I updated that file so I was missing a lot of keys. Now I have all the keys and salts used by the current version. I also changed the database name and user to match my current naming convention.

My checklist for the move is below:

  1. Set the DNS TTL for the domain(s) to a low value (300 seconds) so the DNS change will take effect quicker for the actual move. Do this a couple days in advance.
  2. Backup all website files.
  3. Backup the database
  4. Backup the Apache site file
  5. Create the database on the new server – use the same name as the old server
  6. Create the database user – use the same name/password as the old server
  7. Copy the website files from the backup to the new server
  8. Copy Apache site file to the new server – edit the file for any changes such as IP address the site is on
  9. Restore the database from the backup
  10. Edit hosts file on new server so that the server resolves to itself for the website
  11. Edit hosts file on my PC so it resolves to the new server for the website
  12. Edit my permissions scripts to set file permissions for the site. Run the script to set/test permissions (I have a script I keep updated that will set the proper permissions each night in case I change them and forget to change them back or they get changed in an update.)
  13. Edit backup scripts to include the new site.
  14. Edit WordPress update script to include the new site (I update WordPress through svn)
  15. Update logrotate to include the new site
  16. Connect to the new server’s WordPress site (using the PC with the modified hosts file)
  17. Delete cache files for WP-Supercache and restart plugin (this plugin seemed to dislike being moved). Check the rest of the plugins.
  18. Test the website on the new server
  19. Make a minor change on the new site so it can be identified
  20. Change the DNS from the old server to point to the new server
  21. Allow some time for the DNS change to propagate. Access the site from a different PC or remove the entries from the local hosts file. It may be necessary to flush the cache when changing the local hosts file.
  22. Allow a few days to pass. Check the logs on the old server and if there’s no access (except some search engines and other web spiders) the server can be shut down.

The move itself went smoothly.