Training Performance: A user’s guide to converge faster (TensorFlow Dev Summit 2018)
Why Stack Overflow recommends it
Stack Overflow users link it when answering tf.data and input pipeline performance questions, for advice on op ordering, parallel calls, prefetching and profiling.
Recommended by 3 people across 3 different months, 2018 to 2020.
What it is
A 2018 TensorFlow Dev Summit talk by Brennan Saeta on speeding up model training on GPUs and TPUs. It covers reading profiling tools and overlapping or parallelizing work, especially in the input pipeline, so models converge faster.
What people said
Recommends rodrigo-silveira · 2018-08 · Stack Overflow · on “How do I build an input_fn for Estimator using images stored in a TFRecords file”
As a performance improvement, be sure to always perform `map` after `batch` and `repeat` in your input pipeline when using the Dataset API. See [this video]
Recommends McAngus · 2018-12 · Stack Overflow · on “AWS, Cuda, Tensorflow”
The performance of any machine learning model depends on many things. Including but not limited to: How much pre-processing you do, how much data you copy from CPU to GPU, Op bottlenecks, and many more. Check out the tensorflow performance guide [LINK] as a first step. There are also a few videos from the tensorflow dev summit 2018 that talk about performance. How to properly use tf.data [LINK] , and how to debug performance [this video] are two that I recommend. The only thing I can say for sure is that JSON is a bad format for this purpose. You should switch to tfrecord format [LINK] , which uses protobuf ( better than JSON [LINK] ). Unfortunately performance and optimisation of any system take
Recommends y.selivonchyk · 2019-05 · Stack Overflow · on “Is Tensorflow Dataset API slower than Queues?”
Suggested things didn't solve my problem back in the days, but I would like to add a couple of recommendations for those, who don't want to learn about queues and still get the most out of TF data pipeline: Convert your input data into TFRecord ( as cumbersome as it might be [LINK] ) Use recommended [this video] input pipeline format . files = tf.data.Dataset.list_files(data_dir) ds = tf.data.TFRecordDataset(files, num_parallel_reads=32) ds = (ds.shuffle(10000) .repeat(EPOCHS) .map(parser_fn, num_parallel_calls=64) .batch(batch_size) ) dataset = dataset.prefetch(2) Where you have to pay attention to 3 main components: num_parallel_read=32 to parallelize disk IO operations num_parallel_calls=64 to
Cites Dieter · 2018-04 · Stack Overflow · on “tensorflow input pipeline & performance - images”
Best, system properties: windows 10 intel core i7-6820HQ CPU | 2.70GHZ 8CPUs 16GB ram 64bit NVIDIA Quadro M1000M approx. total memory: 10093 MB Display memory (VRAM): 2019 MB Shared Memory: 8073 MB Tensorflow 1.8 Python 3.5.2 images (i've 36k images): train: 3000 x (720x1280x3) valid: 500 x (720x1280x3) test : 500 x (720x1280x3) My story & strugles First of all, I would like to say that I really like machine learning, specially neural networks. But most of the time, when I'm working with Tensorflow, I've the feeling that it is backstabbing me the entirely time. (like for example, the speed of those releases... (1.8 :O )) & Sometimes I even don't know any more if I'm doing it right or wrong?
Cites rodrigo-silveira · 2018-08 · Stack Overflow · on “Tensorflow Data API: prefetch when training on CPU?”
I'm training on a MacBook pro (no GPU). After watching [this video] I saw that I could improve my input pipeline by reordering some ops and adding num_parallel_calls . Question: Since I'm not using a GPU, does it make sense to include dataset.prefetch(1) to the end of my pipeline?
Cites Nitin · 2018-09 · Stack Overflow · on “Parallelize tf.from_generator using tf.contrib.data.parallel_interleave”
amazing explanations and ideas. Simple and elegant! In the generator version, I think you meant `files.apply(tf.contrib.data.parallel_interleave(wrap_generator, cycle_length=N))` after fetching the files. Chose the generator approach so that the model need not wait until entire file is read - i.e. data prep and training can go in parallel. In order to test this hypothesis and finding the right settings for these APIs (like cycle_length etc), is there a way to visualize - % time spent in CPU and % time spent in IO for a run, something like this [this video]
Cites adam.hendry · 2020-08 · Stack Overflow · on “Is the class generator (inheriting Sequence) thread safe in Keras/Tensorflow?”
I have a proposed "improved" solution that may interest others. Please note this is coming from my experience with Tensorflow 1.15 (I have yet to use version 2). TL;DR Install wsl version 2 on Windows, install Tensorflow in a Linux environment (e.g. Ubuntu) here, and then set use_multiprocessing to True to get this to work. NOTE: The Windows Subshell for Linux (WSL) version 2 is only available in Windows 10, Version 1903, Build 18362 or higher. Be sure to upgrade your Windows version in Windows Update to get this to work. See Install Tensorflow-GPU on WSL2 [LINK] Long Answer For multitasking and multithreading (i.e. parallelism and concurrency ), there are two operations we must consider: fo