Laravel 4 & Dokku: Queue Workers

As I mentioned in my last post we've been developing our new CRM system in Laravel 4 which has been great! One of the big features I'm particularly in love with is the Queue component. As we all know, a queue allows us to push code into a separate thread so that our main code will not block for as long. An example of where this is useful is Emailing.

One of the new features we have allows management to flag actions and ask staff for feedback on why they did something. This is really useful to gain a bit more information from them when they're unable to get in touch with a customer etc. When management click the flag buttons, we update the database & the UI to inform them that the action has succeeded, and we also queue up an email to the agent that alerts them that further feedback is required. Sending an email can take a couple of seconds, so we queue up this job and send it separately. In my testing I found that queuing reduced the time from 5-8 seconds down to 1s, depending on load and network speed etc. This obviously makes for a massive improvement on the client side as the UI is now a lot more responsive to actions.

Whilst I was developing this feature locally I was running php artisan queue:listen and then every time I flagged something, the email would be queued and I'd receive my email shortly afterwards.

I soon realised that I wouldn't be able to run the queue from our Dokku server, since it's just a PHP build pack with no extra or processes I can run. I could run the queue command on another server under supervisor but that would be lame and require me to run two copies of our project. No way!

If anyone has ever used Heroku they'll know that you can make use of a Procfile. These define what processes the server should run. Usually every app will use a web process, which is simply the main server code that handles your website. Another process type is the worker process. This will run alongside your web process and run the command forever.

Remembering this I figured that since Dokku is similar to Heroku I could do the same thing... It turns out that you can't. Dokku only supports the web process type and it took me a while to figure out what the process should even be running! Once I was able to override the default PHP buildpack behaviour using web: bin/run and get our application running again I simply tried adding: worker: php artisan queue:listen. Nadda. Our application was still up, but the queue wasn't being processed.

After a bit of digging I found that Dokku (and therefore Docker?) only supports the web process type, as I said before. My first thought was "oh god, this isn't going to end well...". Was that months of work wasted? Would I seriously have to maintain two copies of the app so that the worker could run elsewhere?

Thankfully not! There is a Dokku plugin that runs all process types! dokku-shoreman came to my rescue.

Once I'd installed the plugin:

git clone https://github.com/statianzo/dokku-shoreman.git /var/lib/dokku/plugins/dokku-shoreman

I pushed the code again, the queue job started being processed!

Voila!

Now I've got Dokku setup with dokku-shoreman and my Procfile looks like:

web: bin/run  
worker: php artisan queue:listen  

There is little to no information about this on the Internet, so hopefully this will be of use to someone!