Perils of pushing: Debugging, optimizing, and learning from your production server

Posted by Pixafy Team


We’ve all been there. At some point in our lives, we have all pushed something live only to have an “oh no” moment where something goes completely wrong. It doesn’t matter how much testing your code underwent, something just went wrong and now it’s up to you to solve it. How do you begin solving this task? Do you dig through the code? Do you blame the IT department? Personally, I can’t tell you what to do but I can shed some light on the situation when events like this occur. Let’s breeze through a few scenarios.

So you pushed code live and now a page isn’t rendering properly. Your local environment and staging environment both look perfect and you’re currently scratching your head trying to figure out where to start. The best thing to do in this situation is to not panic. Take a deep breath and embrace what happened. There are many things to consider when pushing to a production server.  What type of server are you pushing to? What version of PHP are you using? Are there caching protocols running? These should be the questions that pop-up right away instead of blaming your code (we’ll get to that later.) Servers are extremely sophisticated when configured properly and without the proper know-how, you’re setting yourself up to be let down.

Step 1: Know your environment

This seems like a very silly rule, of course you should know your environment but you’d be surprised how little some people actually know. For instance, I knew that our production servers ran nginx but until recently, that was all I knew. When I develop locally I use Apache (as I’m sure most of us do) with some basic APC setup. While the differences between nginx and Apache are vast, it probably isn’t the reason your code just went haywire. There are many other things running alongside nginx on our production servers to ensure each website we build is blazing fast, despite the size of the application. It’s extremely important to take into consideration each and every service that is running; redis, memcache, APC, the list goes on and on. The more you familiarize yourself with each service, the less you’ll be surprised come launch day.

Let’s go back to our main example. Our page is still acting up and all we’ve done so far is look at the server configuration. Now it’s time to look at what package versions we are using.  I’ve had code that works perfectly locally but the minute I put it on a server with a lower version of PHP than what I was using, it broke completely. Blank white page, no warning, not even an error log. So now what? From what I’ve learned, don’t downgrade to your server’s version and don’t upgrade the server to your version. If the site was working before, chances are your code contains something not recognized by the old version and the best plan of action is to continue to support it, which brings us to our next example …

Step 2: Get a good debugger

My mind was blown when a colleague of mine showed me how to use xdebug. I’m not saying it’s the best debugger out there but it certainly does a great job at showing you bottlenecks and where you can optimize your code. It also does a great job at showing you what data is being stored in your variables and how much memory you’re consuming.

I once wrote a cronjob that compiles a user’s images together into a single image in order to decrease page loads. When going live the site instantly crashed when running the method despite it working on my local and staging environments. The problem lied within the limits of the production server — my method was eating up too much memory for the site to handle traffic and that’s a big problem. By applying what I learned in Step 1, I was able to solve the problem quickly by doing a simple memory check before running the method, thus preventing the server from ever overloading and crashing.

It’s always important to get the site up and running again but it’s absolutely crucial to understand what is going wrong and how to properly fix it. Putting a check before running a function is like putting a band-aid on an open wound – it’ll stop the bleeding but it will continue to hurt until it’s healed. Later on when I wasn’t so panicked, I was able to successfully optimize the function so that it would never consume too much memory. Again, the more you learn about your environment the less stress you’ll put yourself under when launching.

Let’s assume that you’ve ruled out server processes and everything in your code still checks out according to what versions your server is running. Now what? The best thing to do in my opinion is to get the bug to reproduce locally. There are many ways you can go about this. I personally like to begin by editing my hosts file so that the server domain appears as-is on my local environment. I was baffled to learn that the reason a function of mine wasn’t working was because the cookie domain was set incorrectly in Magento. Something as simple as that was causing bugs to appear but I would have never known that unless I tried it out locally.

Step 3: Embrace learning

Get comfortable with making mistakes – okay, well don’t get too comfortable. We’re not perfect and we all make mistakes but the difference between a good developer and a great developer is how we handle the situation at hand. Just because you can code doesn’t mean you know exactly what will happen between environments. The best programmers are in fact the ones that are able to debug and dissolve issues quickly and efficiently across any platform without having to bring a site down. Programming is a learning process and (as I keep saying) the more you know, the less surprises you’ll entail in the long run.

Get acquainted with all environments you use and learn them like the back of your hand. You’ll be able to fix bugs and errors much more efficiently but you’ll also gain a better understanding of how your code works (seriously.) When you being to pick things apart you’ll find better ways to put them back together and that, my friends, is what optimization is all about.

Step 4: Optimize everything

As I mentioned before, fixing bugs is one thing but optimizing is a completely different process. You can only optimize when your code works across all platforms and the only thing left to do is to make it faster. Fix the little things first then dive into the larger things like rewriting functions and classes. Chances are you’ll start to see major improvements when fixing those little nuances in your code. Lately I’ve been tasked to optimize a few sites of ours and by simply changing around a few variables I was able to cut off a few hundred milliseconds per page load! My next step is to really dive into the functions and classes to better understand the logic but as you can plainly see, I’ve already accomplished a pretty significant reduction in load time.

As always I welcome comments and stories of others who have had that “oh no” moment when launching websites. It’s always warming to know we’re not alone but even better to learn from each other’s experiences.

Leave us your questions and comments below, or tweet us @Pixafy!