Hacker News new | past | comments | ask | show | jobs | submit login
Scaling Instragram (speakerdeck.com)
112 points by tferris on May 2, 2012 | hide | past | favorite | 35 comments



I wish each and every single one of the open source developer who, every day, build amazing tools to make things like instagram possible would receive a tiny portion of that 1 billion dollars.

But that's not how the world works....

The real revolution of this century is not how many sheeps I can throw per day, but how huge a success the open source movement is, and what incredible companies we can build with it.


That'd actually be a really cool pledge to make as a company. Sorta like the 1% for the planet initiative, but something similar to pledge a slice of profits or revenue, or exit-event-cash to the OSS projects you depended on most.


A few companies do actually, including DuckDuckGo and Tarsnap.

http://fosstithe.org/

http://www.gabrielweinberg.com/blog/2010/11/help-me-start-a-...


I recall the story of Yahoo giving Larry Wall enough pre-IPO stock that they effectively paid his daughter's way through college.

Umm...

http://www.salon.com/1998/10/13/feature_269/


Slide 54 is insanely important. At my current place we've invested tons in instrumentation and metrics. Every IO operation is guarded with a semaphore and access is monitored and exposed. Every cache in the system logs everything from hit and miss rate to volume etc. Every async operation logs operation trails. We also ship a ton of tools for everything from bread and butter stuff like stack dumping, pinpointing long running threads etc. to more sophisticated stuff like being able to break in to a page rendering and profiling various element hierarchies, either over time or following specific users.

As I said, we've invested so much in getting the entire stack monitored very granularly, but every time our customers run in to issues it pays dividends.

For reference, we run on the JVM which is very pleasant. A ton of tools interop with it and basic things like MBeans being integrated in to JVM makes monitoring a lot easier.


How much overhead does this setup incur?


Run time or development time? Runtime is mostly negligible except for places with heavy congestion (caches + rendering) where you can turn the heaviest operations on and off as needed -- in our case the call to `System.getCurrentTimeMillis()` is usually too expensive to do unless we really need it.

In terms of development, it's not a lot of work continuously, but it took some while to write up, test and benchmark standard libraries for eg monitorable semaphores. In general though, what it does is raise the entry threshold for new developers a bit as you need to have a feel for when something requires monitoring and when it doesn't.


This talk was given at AirBnB in SF. Talks there are fairly new and typically occur every other Wednesday. There is a talk tonight by Yehuda Katz and upcoming talks are posted here: http://www.airbnb.com/tech_talks



Submission for a TechCrunch writeup of the talk, which includes some quotes and also embeds the same slides (via scribd):

http://news.ycombinator.com/item?id=3831865


On slides 61-69, it seems to suggest that they were storing their photos directly in the database. Is this correct? It seems rather inefficient to do this, rather than storing the images in the filesystem.


No, photos are stored on S3. Only photo meta data is stored in the db.


Wasn't this same article up on HN the same day they got acquired? Lot's of interesting stuff in here in general, especially when they get to talking about when their app actually uploads a photo.


Instragram is the modern fairy tale showing that any "2 product guys" "with no real backend experience" are able to rise from rags to riches.


Instragram -> Instagram (the title was wrong)


And also that the bubble is back, big time.


I am starting to get a little bit tired of people screaming "bubble" whenever someone mentions Intragram. Could you make a good case for it?

If one particular person (Zuckerberg) values one particular company higher then most other people, I don't see that as a sign for a bubble.


It's a sign of... wait, strike that. It's a huge red flag the size of the moon when a company with no revenue (http://www.npr.org/2012/04/10/150372288/instagram-sells-for-...) is sold for so much money.

I would argue that Facebook and Google are bubbles too since they do not provide value. Advertising is a joke of an industry - it produces nothing and is essentially the business of mass psychological manipulation. For proof that such bubbles burst, look towards to entertainment industry which also provides nothing of value yet charges through the nose for it.


> I would argue that Facebook and Google are bubbles too since they do not provide value

But you'd be wrong. Facebook and Google create enormous amounts of value. Enough to support their high valuations? Only history will show.

> Advertising is a joke of an industry - it produces nothing

Please don't confuse your personal opinion of the advertising industry with 100+ years of empirical evidence.

> look towards to entertainment industry which also provides nothing of value yet charges through the nose for it

"Entertainment" is doing fine. The current industry establishment is not. Not the same thing.


>Please don't confuse your personal opinion of the advertising industry with 100+ years of empirical evidence.

It really depends on what your definitions are. My personal take on it is that advertising is duct tape for a broken economic model. If we have an entire economic system where one of it's pillars is something so asinine (psychological manipulation with no opt-out for certain flavors like billboards), perhaps we should rethink the model.

So, from the perspective of a business which is trying to find clients/customers, sure, there's a tonne of value. From the perspective of trying to move humanity to bigger and better things, it has a huge negative value, again, IMO.

>"Entertainment" is doing fine. The current industry establishment is not. Not the same thing.

I actually wrote "entertainment industry" in my post :)


> So, from the perspective of a business which is trying to find clients/customers, sure, there's a tonne of value.

Excellent. And I'm sure you'll agree that's something of a leap from "it produces nothing"? Also, there's a corollary to this: This enables competition by lowering barriers to entry, which massively benefits consumers.


>And I'm sure you'll agree that's something of a leap from "it produces nothing"?

A diamond is worth a great deal in a lot of peoples' eyes yet it produces nothing. Interestingly, the goal of advertising and marketing is to increase the value of actual things by tricking people into believing that a) there's not enough supply - buy it now! and b) that demand is too high - buy it now! What exactly has been produced here? Demand? That's not a thing, it's an idea. Intellectual "property" is an absurd term, therefore, nothing has been produced.

On the subject of why I think that it has no value - well, most businesses see no value in me scratching my hand because it has an itch. From my perspective, there is a great deal of value in it. Similarly, I see no value in what some businesses do, since, from my perspective, they are harming the people around them. There is no absolute value in anything. Therefore, the value of something is determined by the beholder, which in my case is me.

>This enables competition by lowering barriers to entry, which massively benefits consumers.

Assuming that a competitive economy is a good thing, ofcourse.


> Assuming that a competitive economy is a good thing, ofcourse.

Yes, indeed assuming that. Good day.



The first is a paper by a psychologist and motivational speaker. Is that really what you want to lead with?

The second paper I can't really figure out, but it's dealing with individuals in a closed environment - not with matching supply to demand in an open economy.

Anyway, I'm plenty open to questioning my assumptions. That doesn't mean I'm willing to invest in lengthy discussions with random strangers in off-topic threads.


It's a redflag for the stability of Facebook being able to maintain their dominance; A blog post I did related to this, http://mattamyers.tumblr.com/post/22070489398/re-how-twitter...

Google's might be overvalued but I don't think so. They've had 10 years of search data to analyze and understand human behaviour. They're working on solving mass consumer real-life issues; Maps, Chrome, Self-driving cars, etc..


And here I thought Google is one of the most important services ever provided in human history.

Yeah, that's not how they earn money, but they do provide value for people.


What companies do actually provide value according to your definition? Don't most companies try to sell you stuff you don't strictly need?


I'm a pragmatist, so anything which makes survival easier has value. Everything else is fun and games (not important enough to build businesses around and charge money for). Personally, I think it would be good to have two economies - a 'core' economy where there is a gold standard, currency can only be traded for essential goods and patents and speculation are strictly forbidden. Then a 'extra/luxury/optional/nice-to-have' economy for all the rest which does allow a more fragile currency, speculation and what-not.

If you want food, produce something which will help someone else survive. If you want want to play around and relax, create something which will help other's do the same. Trade what you produce for what you require.


So computers and the internet don't have value... Why are you here?


Some types of information are essential to survival.


I'm pretty sure the human race "survived" before computers existed. At least, I think so...


Oops, meant to upvote.


Is there a video/audio for this talk?


Won't somebody please think of the nit-pickers, and fix the typo in the title? At least it's only in the submission to HN.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: