Distributed TensorFlow (TensorFlow Dev Summit 2017)
Why Stack Overflow recommends it
Stack Overflow users link it when explaining or learning distributed TensorFlow concepts such as parameter servers, between-graph replication, and how MonitoredTrainingSession coordinates workers.
Recommended by 3 people across 3 different months, 2017 to 2020.
What it is
A 28-minute talk from the TensorFlow Dev Summit 2017 in which Google engineer Derek Murray gives a bottom-up introduction to Distributed TensorFlow. It covers the tools for scaling training across many GPUs and machines, including parameter servers, workers, and replication strategies.
What people said
Strong rec GPhilo · 2018-02 · Stack Overflow · on “Performance improvement using Between-Graph replication in distributed tensorflow”
Could you clarify what was unclear in the answer you linked? I answered before you fixed the link, but really that answer is a lot better and more detailed than mine (it was written by the guy explaining distributed tensorflow in this video [this video] , which I encourage you to watch by the way). If you clarify what's unclear, I can perhaps focus on those points and improve my answer
Recommends mrry · 2017-03 · Stack Overflow · on “Distributed TensorFlow - Not running some workers”
I'm guessing the cause here is the implicit coordination protocol in how a tf.train.MonitoredTrainingSession [LINK] starts, which is implemented here [LINK] : If this session is the chief: Run the variable initializer op. Else (if this session is not the chief): Run an op to check if the variables has been initialized. While any of the variables has not yet been initialized. Wait 30 seconds. Try creating a new session, and checking to see if the variables have been initialized. (I discuss the rationale behind this protocol in a video about Distributed TensorFlow [this video] .) When every session is the chief, or there are no variables to initialize, the tf.train.MonitoredTrainingSession will alw
Recommends Sam · 2017-09 · Stack Overflow · on “Scaling issues with distributed Tensorflow and GRPC”
Yes. Check out @mrry's presentation here [this video] about how to scale distributed tensorflow. Basically, each process connects to a param server and is driven by a separate Python process.
Cites Elad Weiss · 2017-08 · Stack Overflow · on “Can TensorFlow run with multiple CPUs (no GPUs)?”
I'm trying to learn distributed TensorFlow. Tried out a piece code as explained here [this video] : with tf.device("/cpu:0"): W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10])) with tf.device("/cpu:1"): y = tf.nn.softmax(tf.matmul(x, W) + b) loss = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) Getting the following error: tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'MatMul': Operation was explicitly assigned to /device:CPU:1 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device. [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=f
Cites Amila · 2018-06 · Stack Overflow · on “Is Tensorflow's Between-graph replication an example of data parallelism?”
I have read distributed tensorflow documentation [LINK] and this answer [LINK] . According to this [LINK] , in data parallelism approach: The algorithm distributes the data between various cores. Each core independently tries to estimate the same parameter(s) Cores then exchange their estimate(s) with each other to come up with the right estimate for the step. And in model parallelism approach: The algorithm sends the same data to all the cores. Each core is responsible for estimating different parameter(s) Cores then exchange their estimate(s) with each other to come up with the right estimate for all the parameters. How do In-graph replication and Between-graph replication relate to these
Cites milkshake · 2018-12 · Stack Overflow · on “Does distributed TensorFlow requires a distributed file system to save checkpoints?”
I am trying to run some sample codes based on this tf official tutorial [LINK] . And I watched this video [this video] which is really nice. As mentioned in the video above, the chief worker is responsible for saving checkpoints, and it's implemented by tf.train.MonitoredTrainingSession. Then I thought that only the chief worker needs a directory to save checkpoints. When I run the codes with ps0 on machine1, worker0 on machine2, everything seems ok. But when I run with ps0, worker0 on machine1, ps1 and worker1 on machine2, errors occurs, and the error in worker0's log is like: Traceback (most recent call last): File "distributed_train.py", line 136, in <module> tf.app.run(main=main, argv=[sys.ar
Cites PeterN · 2020-01 · Stack Overflow · on “In distributed TensorFlow training, what is a PS task? What does PS stand for?”
ps is a Parameter Server - it is a logical implementation of a physical area (be it CPU, a remote server) which is in charge of creating and preparing Tensors for computation. This is why the term "ps task" is used - to mean a task that creates Tensors. An example of parameters could be variables which represent neural network weights. The corresponding "worker" is in charge of taking Tensors and running computations on them. An example of a worker is a neural network training process running on GPU. Resources: [LINK] [LINK] Starting 3:00 [this video] Starting 5:10