I was bored last night so I thought I’d enhance one of my applications with a little bit of Queueing. (And for the record both Queueing and Queuing are valid spellings depending on your locale)
The concept is fairly straight forward. Normally when a web application runs, it runs in a linear pattern. It starts, it does stuff, it does more stuff, then it finishes with a web page. That’s pretty much it. The problem is, the user has to wait for it to do stuff and more stuff. And when they have to wait for more and more stuff, it gets slow. If some of that stuff can be offloaded because it has no affect on what the user sees right now – why not? That’s where a Queue comes in to play. Send a command to a queue that something else can process for you.
So enter Zend Queue. A feature of Zend Framework that will let you use various queue systems to your advantage. I chose MemcacheQ because it was simple to install and I knew that I’d need not do anything to my configurations to utilize it. MemcacheQ is a light-weight, fast, persistant queueing system (It’ll remember what it has when it gets shut down) based on Memcache’s protocol. So accessing the data in the queue is a piece of cake.
I tried using Zend_Queue_Adapter_Memcacheq in Zend Framework 1.10.8 – but it failed. It doesn’t seem to work properly with MemcacheQ 0.2.0 which is the latest version. That lead me to write my own little adapter – based on the existing one – to fix the incompatibilities. I also added automatic serialization to the messages sent to the queue. This allows me to send useful items, instead of just strings. I implemented the queue system in a way that I can now do queued inserts and updates on database rows wherever I see the need. It’s also setup in a way that if sending a message to the queue fails, it will do the desired action right away (slower for the user, but no loss of data.) After that I was set.
I send my messages to my queue as described in Zend Framework’s documentation ($queue->send($bundle)); My bundle is a small array containing a few useful items. One: a class name, two: a method to call, three: parameters for the method. And that’s it. Now I’m queueing magical commands. I need to process the queue.
I then wrote a queue processor. This processor is a very small command line tool that utilizes my existing Zend Framework applications configuration – so it has all the access it needs to the application database and code base. The queue processor pulls in a small set of bundles from the queue – executes them – and waits for a short period of time and then processes more. In the event that a command fails, the queue processor will re-queue the item and wait to try again. This is to prevent data loss. If for some reason the database is unavailable the updates and inserts will persist in the queue until they can be executed properly.
I hope I’ve explained it in enough detail that you can understand the process. If not, leave a comment and I’ll try and clarify it. I’m not posting any code just yet because it’s not “pretty.”
So how well did it do? Judge for yourself. You can tell when I enabled it.