Distributed TensorFlow training (Google I/O '18)
Why Stack Overflow recommends it
Stack Overflow users link it as a reference when asking or answering how to set up distributed or multi-GPU TensorFlow training with distribution strategies.
Recommended by 0 people across 0 different months, 2018 to 2020.
What it is
A Google I/O 2018 session by Priya Gupta and Anjali Sridhar of the TensorFlow team on scaling model training to multiple GPUs and machines. It introduces the Distribution Strategy API (such as MirroredStrategy) and the Estimator-based multi-machine workflow, with performance tips.
What people said
Cites user912830823 · 2018-05 · Stack Overflow · on “Correct way to run distributed training in ML Engine”
Im trying to train my model (that is not build with tf.estimator or tf.keras) using distributed training job in ML Engine. What steps should i take in order to run distributed training job in ML Engine? I found following guidelines: provide --scale-tier parameter, from step-by-step guide [LINK] use distributed strategy API in the code, from recent google io talks [this video] So if former provided in the command line does it mean i don't need to do anything with latter because ML Engine somehow takes care of distributing my graph across devices? Or do i need to do both? And also what happens if i manually specify devices using: with tf.device('/gpu:0/1/2/etc') ..and then run the command with --sc
Cites HARSH NILESH PATHAK · 2019-06 · Stack Overflow · on “How to run Tensorflow Estimator on multiple GPUs with data parallelism”
I think this is all you need. Link: [this video] More Details: [LINK] Explained: [LINK] NUM_GPUS = 8 dist_strategy = tf.contrib.distribute.MirroredStrategy(num_gpus=NUM_GPUS) config = tf.estimator.RunConfig(train_distribute=dist_strategy) estimator = tf.estimator.Estimator(model_fn,model_dir,config=config) UPDATED With TF-2.0 and Keras you may use this ( [LINK] )
Cites user2822481 · 2020-07 · Stack Overflow · on “TensorFlow Keras multi-GPU training bad performance”
tensorflow multi-GPU training with mirrored strategy (GPU VS CPU) BAD performance Asking everyone for help. I am so confused why GPU is slower than CPU on any condition I try... I want to use six GPU with the mirrored strategy to reduce the training time. I follow the below steps: [LINK] I got bad performance with the machine GPU: 6*GeForce RTX 2080 TI (10GB) CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz run on the Docker container: (1) Tensorflow version 2.0.0 (2) Cuda 9.0 (3) Cudnn 7.6 (4) Nvidia-driver 410.78 - install on server I want to ask some questions. (1) I want to know whether TensorFlow 'MirroredStrategy' can speed up the training speed?? For example, follow the link: [LINK] (2