Inside TensorFlow: tf.data - TF Input Pipeline
Why Stack Overflow recommends it
Stack Overflow users link it when explaining how tf.data works under the hood, such as how Dataset.map turns Python functions into graphs executed by the C++ backend.
Recommended by 3 people across 3 different months, 2019 to 2020.
What it is
An internal TensorFlow team training session on tf.data, TensorFlow's input pipeline, presented by the people building it. It covers the Python view, the C++ view, support for non-tensor types, and static and dynamic optimizations, and assumes basic TensorFlow familiarity.
What people said
Recommends craq · 2019-09 · Stack Overflow · on “difficulty with imshow() numpy() and eager execution in tf2.0”
The critical insight for me was that running the tf.data.Dataset.map() function builds a graph, and the graph is executed later as part of a data pipeline. So it is more about code generation, and eager execution doesn't apply. Besides the lack of eager execution, building a graph has other restrictions, including that all inputs and outputs must be tensors. Tensors don't support item assignment operations such as T[0] += 1 . Item assignment is a fairly common use case, so there is a straightforward solution: tf.py_function (previously tf.py_func ). py_function works with numpy arrays as inputs and outputs, so you're free to make use of other numpy functions which have not yet been included
Recommends thiagocrepaldi · 2019-10 · Stack Overflow · on “How MapDataset captures user defined function from Python and inject into C++ implementation”
I am getting started with TF and I am interested in 1) learning how tf.data implementation works, in particular, on how the C++ backend captures/executes user defined-function by Python scripts. I was expecting that C++ backend would use Python embedded approach, but apparently this doesnt happen; maybe because of performance hits in instantiating python interpreters on c++ for each example/batch. Instead, tf.data seems to transform user-defined functions by Python scripts in graphs on the C++ side, as mentioned in this great video by TF folks at 5:15 [this video] . I have been trying to make sense of python/ops/gen_dataset_ops.py , tensorflow/core/ops/dataset_ops.cc , tensorflow/core/kernels/dat
Recommends AAudibert · 2020-06 · Stack Overflow · on “Does tf.data API make use of tf.placeholders under the hood?”
tf.data does not use tf.placeholders under the hood. tf.data.Dataset uses the same mechanisms as other Tensorflow ops. The graph it creates in graph mode is not separate. Check out the below video to learn more about how tf.data works under the hood. I linked to the part that explains what happens when you call next() on a dataset iterator. [this video]
Cites BBQuercus · 2020-04 · Stack Overflow · on “Loading segmentation data with tf.data as Dataset?”
[this video] I assume. If you want to add a short answer mentioning that map automatically unzips the dataset I can upvote it.