
A behind-the-scenes look at Facebook release engineering - 3lit3H4ck3r
http://arstechnica.com/business/news/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering.ars?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+arstechnica%2Findex+%28Ars+Technica+-+Featured+Content%29
======
joshuahedlund
So many gems in this article.

 _To help spot problems, Facebook employees who access the social network from
within the company's internal network will always see an experimental build of
the site based on the very latest code, including proposed changes that
haven't officially been accepted._

Probably the only place where your excuse for checking Facebook at work can be
"Looking for bugs!"

 _The many data sources tracked by Facebook's internal monitoring tools even
include tweets about Facebook. That information is displayed in a graph with
separate trend lines to show the change in volume of positive and negative
remarks_

Guess I need to tweet more about how slow their mobile app is getting...

 _One of the major ongoing development efforts at Facebook is a project to
replace the HipHop transpiler. Facebook's developers are creating their own
bytecode format and custom runtime environment.... the company can push thin
bytecode deltas representing just the parts that have changed. Facebook may
even be able to splice the updated bytecode into the application while it's
running, avoiding a process restart._

Even though I'm doing nothing nearly this awesome, this article has done more
to inspire and excite me about my own coding than anything I've read in a long
time.

~~~
mikeleeorg
I'm also really intrigued by the "karma" rating for all of their developers.

Can anyone speak to how well that is working? Has it been effective? Any
negative side-effects? Is it just for the release process, or for any
development effort?

~~~
nbm
It is "Push karma", so generally only applies to the push process. It isn't
visible to anyone but the pushers and yourself (at least, I can't see anyone
else's push karma in the expected places).

It isn't a complex rating system - there are probably 97% of people at the
base karma level, maybe 0.1% at one rung higher, 2.7% at one rung lower, and
0.2% at lower than that. The "Push" tech talk at
<https://www.facebook.com/video/video.php?v=10100259101684977> has more on it.

Mostly it is a way of letting you know that you made people's lives difficult
by holding up the push process by not being available to support your changes.
You know that you won't get away with that, that you need to make up for it,
and you also know when you've made up for it.

(It also doesn't apply to a decent number of engineers, since they work on
services and infrastructure that are not part of that process.)

------
brown9-2
This tech talk video from Chuck Rossi on the same topic also has a lot of
really interesting information
<https://www.facebook.com/video/video.php?v=10100259101684977>

------
benjaminwootton
I'm surprised by the monolithic all-or-nothing deployments that they have off
of the single binary.

I prefer to componentise applications and allow those components to be
deployed, released, and rolled back separately.

I also don't agree with the 'rollback is for losers' message as hinted at in
the article.

A fast dependable rollback (measured in the seconds) is significantly
preferable to getting a developer to implement a fix to some issue under
pressure and push it out in a rush. Much better to roll back, take stock, then
implement the right fix under the tested process.

~~~
flyt
Facebook is updating tens of thousands of servers with every push. "Rolling
back" a release could take as long as a regular push and contribute to
problems as the version in use diverges.

Instead, FB has an aggressive and flexible internal system for "ungating"
features to groups based on different criteria. Usually a feature would be
pushed out in a deactivated state, then a developer will slowly ramp up its
exposure to actual traffic. This limits the ability for a push to insta-break
the site and means they can come back around for the next day's push with
tweaks, then increase the code's exposure.

~~~
numlocked
According to the article reverting does NOT involve re-deploying. Each server
maintains the previous version of the binary and if needed the release team
can pull the switch to revert all the servers. I assume that takes seconds,
not 30 mins.

~~~
TeeWEE
In fact i worked at a dutch social network where we also used hiphop. The new
compiled binary is pushed to all servers and then it is started, the old
binary is stopped and a port handover is done. Thus deploying without
downtime. The old binary is available on the system, making a rollback very
fast. However old binaries are removed after a time. So you can only roll back
to a previous version quickly.

------
stcredzero
_Facebook's developers are creating their own bytecode format and custom
runtime environment.... the company can push thin bytecode deltas representing
just the parts that have changed. Facebook may even be able to splice the
updated bytecode into the application while it's running, avoiding a process
restart._

NB: Fast JIT byte code VMs running web app frameworks in high level languages
that can do all of this have been around since the early 2000s. (Including the
distribution of binary deltas that can be applied atomically to live servers.)
Smalltalk web app servers had all of this tech, plus refactoring capabilities
and distributed version control years ahead of the rest of the industry. It
makes me wonder what else is out there beneath the radar today.

------
jpeterson
Why would they transfer the entire 1.5g binary each time, and not a delta?
Surely it doesn't change _that_ much between releases.

~~~
jrockway
Because the continuous integration runs tests for all components at HEAD, not
at every random possible combination that could end up on a machine. The key
to releases is repeatability and consistency. Copying one big blob to every
machine is repeatable and consistent. Installing a bunch of libraries and
updating things piecemeal is much more difficult to do right. Internal
bandwidth is cheap, so this is almost a no-brainer. Even without an internal
bittorrent distribution mechanism, it's still easy.

The reason why people tend to gravitate to incremental deployments for web
applications is because the typical tools encourage it; modules get installed
in separate directories, each part of the app is a separate file (back in the
CGI days), etc. When you compile everything into one file, though, then you
just copy that file around to deploy. It's easier. (Ask a PHP programmer how
to change one file, and it will probably be "change that one file". Ask a Java
programmer, and it will be "fix the file, build a WAR, and replace the WAR".
Tools dictate process, and the "scripting language" default is to work on a
file level instead of an application level.)

I've always wanted one-file deployment for my personal applications, but I
never saw anyone doing it so I assumed I was wrong. But nope, it turns out
that everyone else was wrong :)

~~~
sciurus
jpeterson didn't suggest "installing a bunch of libraries and updating things
piecemeal". Instead of transferring the entire binary for every release they
could generate a (likely much smaller) patch, transfer just it, then apply it.
I expect they're not doing this because it's computationally intensive
compared to transferring the entire binary.

------
bonaldi
I find it most interesting that they rely on irc internally. They work on one
of the world's largest online communications platforms -- surely they could
solve their problem in a way that gives it to their millions of users too?

~~~
nbm
We do make extensive use of Facebook messages (many people, including me, make
good use of Facebook Messenger on mobile and/or desktop) and of Facebook
groups.

IRC's ability to quickly create temporary groups, temporary membership
(essentially muting discussion by leaving, or peeking by joining), bot
frameworks, and multiple clients are potential reasons why IRC might be
preferable for the sort of things it is used for, I guess.

~~~
alexgartrell
As I understand it, the main reason is that IRC is decoupled from Facebook
completely (or should be). In "Oh shit, the sky is falling!" SEV situation we
can trust (sorta) that IRC will be there.

------
raphinou
The article mentions that employees visiting Facebook from inside use an
experimental build. Any idea how they manage this if the experimental build
requires changes in the data structures used by the site?

~~~
bonzoesc
I suspect that if new features require storage changes, the changes are
strictly additive and won't affect old features, or old features are modified
to use the new storage setup.

With the amount of data Facebook has, they probably don't have the option of
an "iterate over every row and change a thing" kind of migration.

~~~
flyt
No, but they do have an extremely fast and efficient tool for performing
online schema changes to its huge MySQL deployment:
[https://www.facebook.com/notes/mysql-at-facebook/online-
sche...](https://www.facebook.com/notes/mysql-at-facebook/online-schema-
change-for-mysql/430801045932)

------
cake
_The company has two separate sets of these tests; one does some conventional
sanity checking on the code and the other simulates user interaction to make
sure that the website's user interface behaves properly._

Anyone know more about this ? How are the user interaction made ?

~~~
rheide
I'm guessing they're talking about unit tests and integration tests, the
integration tests probably simulating user input using a framework such as
Selenium(RC).

~~~
alexgartrell
It's been a while since I've had to deal with it (bootcamp), but I'm pretty
sure we still use Watir [1]

[1] <http://watir.com/>

------
zbuc
How do you wind up with a 1.5gb binary? That's incredible -- especially
considering all their static assets are on their CDN, so this is basically
their code and all the libraries they're pulling in.

~~~
kmavm
(I work on the HipHop compiler.) You start by compiling PHP source.

Simple PHP statements take a lot more space in the binary than intuition
suggests. E.g.:

if ($a == $b) ...

would seem like it should be

    
    
      cmp $rax, $rbx
      jz ...
    

But! If type inference has failed, we don't know what types $a and $b are, so
they might be strings or objects or something crazy. So we're going to have to
indirectly dispatch to $a's '==' method. We also spend a ton of space on
reference counting code; the semantics of the language basically force you to
do naive reference counting, since refcounts can be witnessed in various ways,
so every time we pass an argument, do an assignment, sometimes even evaluate
expressions, we need to manipulate reference counts, and if they've gone to
zero call a destructor.

It ends up making the code really large, and one of the things that's unique
about our efforts to run PHP fast relative to other dynamic language efforts
is that sheer code bulk ends up being our largest enemy; if we're not careful,
icache misses eat us alive.

Finally, I'll note that it's not _quite_ a 1.5GB binary. The actual ELF binary
is something like 1.1GB, and the remainder of the package we bittorrent around
production is stuff like static resources (javascript, css) and primed
contents for the APC cache that we want prepopulated on boot.

~~~
cookiecaper
Based on your work heretofore, do you think it's wise for Facebook to continue
on the PHP path instead of working on a backend rewrite in C# or some other,
saner language? I find it odd that Facebook is still using PHP and pouring
lots of effort and cash into things like HipHop when they're obviously hiring
people smart enough to use another language, and when they obviously have the
runway to perform a dark horse rewrite into a much cleaner, saner backend.

~~~
kmavm
This is a long and deep subject.

I wouldn't say that we've "continued on the PHP path." I'd say that we've
refused to throw out the precious PHP parts of our application, while not
being afraid to use more appropriate languages across Thrift boundaries when
needed. Our search engine, newsfeed, and ad serving infrastructure, for
instance, are in C++.

A drop-everything-and-rewrite of the PHP code is entirely out of the question,
for all the reasons covered in Spolsky's 12-year-old classic on the subject:
<http://www.joelonsoftware.com/articles/fog0000000069.html>. Those of us
working on making PHP perform better are a tiny fraction of Facebook
engineering as a whole; this small overhead cost is nothing compared to the
risks inherent in a ground-up rewrite.

Most of PHP's language-level faults can be engineered around. For instance, we
have a code-review-time script that parses (really parses) the code to warn
engineers (and reviewers) about dangerous or deprecated idioms.

PHP also has some affirmative virtues. The programming model is more
productive than that of compiled languages, and even many interpreted
languages; save/reload the web page is just a better, tighter loop to get work
done in than save/compile/restart my server/reload the web page. I'm actually
a fan of PHP's concurrency model, which naifs often mistake as "no concurrency
allowed"; PHP's concurrency primitive is curl[1], and if you wrap a tiny bit
of library around it, you can make it behave like actors.

[1] Seriously. curl provides a shared-nothing way to asynchronously run code,
and has the virtue of not caring what language the other side is written in to
boot.

~~~
cookiecaper
Right, I'm familiar with Spolsky's piece, but I think there are times when a
rewrite is legitimate. I think that a situation where you must roll a
completely custom in-house compiler that generates binaries which exceed 1 GB
in size in order to get adequate performance of your app is a good candidate
for a new architecture, despite Spolsky's claims. Spolsky's article discusses
throwing out pages of code because the programmers "don't know what half of
these API calls are for" and "[wanting] to build something grand" -- these are
quite different impulses than the real-world problems staring Facebook in the
face by its continued usage of PHP.

I think also that there is a difference between writing a new backend for
something that is solid and in place and just throwing the whole product out
the window and re-imagining it from the ground up, and I think the latter is
the kind of rewrite that should be avoided and considered dangerous. When you
could throw 4-5 guys on a real C# or C++ rewrite and tell them the final
product has to behave identically to the PHP version, you have a much less
volatile situation.

As for the PHP workflow, I agree it's nice not to have an intermediate step,
but that intermediate step can usually be circumvented pretty rapidly by
throwing a script or two (or just flipping a config option) into your
development environment.

~~~
ericd
I think developing in PHP and compiling to C++/binary probably results in much
higher developer productivity than developing in C++/C# directly. Developer
salaries are undoubtedly their largest expense, by far, dwarfing those
salaries of the people who make PHP performant and 1.5 gig binary updates
sane.

~~~
sciurus
I expect that running their datacenters is a larger expense than developer
salaries.

"In 2011, $606 million was allocated towards total capital investment in data
center infrastructure by Facebook, which includes the cost of servers,
networking equipment, construction, and storage." -
[http://www.colocationamerica.com/blog/facebook-data-
center-i...](http://www.colocationamerica.com/blog/facebook-data-center-
infrastructure-expenditures-a-quick-analysis.htm)

~~~
evgen
The assumption that migrating the entire www stack to something like C++ would
help with the datacenter costs is not supported by reality. Please remember
that the bits that are in PHP are mostly front-end code. This handles the
presentation of the data, but the actual heavy-lifting and data manipulation
is done by the back-end infrastructure which is mostly C/C++ with some Java
thrown in for the hadoop bits.

~~~
nbm
I would disagree here - every percent CPU saved for the same workload is a 1%
reduction in the number of machines needed. The number of web machines is
sufficiently large that savings of even 1% are praiseworthy. Quite a bit of
effort is expended to keep this going down and to the right (at least some of
the time).

------
algolicious
_Facebook's testing practices and culture of developer accountability help to
prevent serious bugs from being rolled out in production code. When a
developer's code disrupts the website and necessitates a post-deployment fix,
the incident is tracked and factored into Facebook's assessment of the
developer's job performance.

[...]

Employees with low karma can regain their lost points over time by performing
well—though some also try to help their odds by bringing Rossi goodies. Booze
and cupcakes are Rossi's preferred currency of redemption; the release
engineering team has an impressive supply of booze on hand, some of which was
supplied by developers looking to restore their tarnished karma._

This sounds like Facebook strongly rewards developers who work on trivial,
low-risk features rather than larger, more important features. Also, it sounds
like bribery factors into your overall job performance rating.

~~~
nbm
Push karma primarily affects how likely the release engineering team will
accept any breaking of the standard rules of getting your code into the push.
It generally doesn't drop if you are responsive and responsible for any
problems your change causes. The only way to restore points is to show respect
and consideration for the hard work the release engineering team does.

(I'm not 100% sure, but I think most of the booze and cupcakes come from
people who were appreciative of the release engineering team for bringing
potential issues to their attention or for being accommodating in terms of
hours and in terms of delay to get things fixed.)

Being irresponsible (not supporting your changes) will factor into your
performance review, but working exclusively on low-risk features will most
likely hurt it way more.

