Large Scale Deep Learning with TensorFlow (Jeff Dean, Google)



Click on text below to jump to the corresponding location in the video (except iOS)



I am talking to you about our approach to using deep learning in lots of different Google products. And Tensorflow is the system we have built and recently open sourced to make this kind of machine learning approach easier.

One of the things over the last decade is we now have a pretty good handle on how to store and perform computations over large scale datasets.
Things like MapReduce, BigTable, Spark and Hadoop really give the tools you need to manipulate and store data. What we really want to do is understand the data.

What do I mean by understanding?
Describe this photo.

A street scene - we would like to read the text, understand it, understand where it is.


Similarly, if I am speaking, you would like a machine to understand what I am saying.
Neural networksIncredibly powerful abstractions that can do all these things. They can learn a complicated function from data. What is Deep Learning?- A powerful class of machine learning model
- Modern reincarnation of artificial neural networks

- Collection of simple, trainable mathematical functions


They are loosely based on what we know about the brain


Commonalities with real brains:

-each neuron is connected to a small subset of other neurons

-based on what it sees, it decides what it wants to say

-neurons learn to cooperate to accomplish the task


Each neuron implements a relatively simple mathematical function
Cartoonish representationCat or dog?Important property of neural networksResults get better with more data, bigger models and more computation
Better algorithms, new insights and improved techniques always help too
Growing use of deep learning at GoogleIn 2012, there was not much use of deep neural nets at Google. Over time, we built tools that other teams can pick up and use to solve their problems. The tools were general purpose and were easily repurposed to different domains just by training on different types of data. TensorFlowAllows us to express machine learning ideas - and train, use and deploy them in real products.
Released in Nov 2015

Apache 2.0 license
Strong external adoptionThere is a lot of demand to do this kind of ML and making it easy. MotivationsDistBelief was scalable and good for production deployment, but not as flexible as we wanted for research purposes
We rebuilt TensorFlow - we wanted to keep the scale, but make it much more flexible
Websitehttp://tensorflow.org/TensorFlow: Expressing high level ML ComputationsAt its core, its a graph execution engine.Different front ends for specifying/driving the computation. Example TensorFlow fragmentBuild a graph computing a neural net inference. This is sort of the lowest level model. Python API for machine learningAutomatically adds ops to calculate symbolic gradients of variables w.r.t. loss function.

Launch the graph and run the training ops in a loop
Computation is a dataflow graphThe things that flow along the edges of the graph are tensors - multidimensional arrays with various kinds of types. We have augmented the traditional dataflow model - mostly functional and stateless - with state. The parameters in the model are going to be variables in this graph.

The other important thing is to map these abstract graphs on to different kinds of computational devices.
Assign devices to opsAutomatically runs on variety of platformsPhones, single machines, distributed systems, custom ML hardware. What are some ways that deep learning is having a significant impact at Google?Speech recognitionImproved the quality of speech recognition. Reduced word errors by more than 30%. Google Photos SearchSearch personal photos without tags. Google Project SunroofFind rooftops in satellite imagery. To estimate solar installation costs. AlphaGo competitionSystem beat world champion Go player and beat him 4-1. Inbox - smart replyAutomatically replies to email. Now >10% of mobile inbox replies with this feature.
Predicts the message reply, you can just click or edit the reply.
Combined vision + translationOCR on device.Image captions researchTest image.
More examples.
Experiment turnaround time and research productivityMinutes or hours allows more interactive research. Image model training timeMore GPUs is better. How can you get started with machine learning?Four ways, with varying complexity.
1. Use a cloud based API

2. Run your own pretrained model.

3. Use an existing model architecture, and retrain it or fine tune on your dataset

4. Develop your own machine learning models
Use cloud based APIsGoogle has a set of these.
E.g. cloud vision API
Using a pretrained modelFor training your own modelsOn your own physical machines or virtual machines. Cloud machine learning in alpha stage. Training a model on your own image dataDont have to be a ML expert. Develop your own modelsExample: combining vision with robotics20 robotic arms - trained to pick things up. Over time the model gets better and better. ConclusionsModels are making a really big difference.
Video outline created using VideoJots.

Leave a Comment