TensorFlow Datasets is turning 4! — The TensorFlow Site

Posted by the TensorFlow Datasets staff

Datasets landscape has improved a lot due to the fact TensorFlow Datasets (TFDS) was launched about 4 decades ago: TFDS created sharing or re-using a dataset appreciably a lot easier, and reworked the datasets landscape by inspiring other ML equipment, libraries and solutions.

Loading a dataset went from complicated scripts to:

import tensorflow_datasets as tfds ds = tfds.load('mnist', break up='train') for instance in ds: # case in point is `'image': tf.Tensor, 'label': tf.Tensor` print(listing(case in point.keys())) image = example["image"] label = instance["label"] print(image.condition, label)

Read through the documentation for a much more in depth introduction.

About the years, TFDS has grown to turn out to be a acknowledged way to load datasets. To celebrate our final 4.8.2 launch, we would like to acquire some time to replicate on the development and improvements manufactured more than those people previous many years and thank the group for their aid.

TFDS is however a library to aid down load, preparation and loading of datasets for ML pipelines, but it now supports hundreds of datasets and features the subsequent primary attributes:

1. A significant wide variety of features with encoding and decoding, ranging from text to visuals, video clips, audio and even RL-distinct forms (e.g. dataset of datasets).
2. Large datasets support: TFDS is correctly utilized inside of Google to put together and load huge datasets (PBs) applying high performance input pipelines.
3. Dataset collections, to arbitrarily team alongside one another a variety of present TFDS datasets, for instance applied in a benchmark.
4. Assist for all key ML Python frameworks: of course there is “TF” in “TFDS”, but moreover TensorFlow, one can use TFDS with Torch, Jax, NumPy, Keras and any other Python ML framework that can take in a tf.info.Dataset or a NumPy Iterator.
5. Global shuffling at preparation time: It is very good observe to shuffle training data, TFDS optionally does a global shuffling at preparation time in scenario the source of the info wasn’t previously shuffled.
6. Splits and slicing: datasets can specify their splits, and viewers can specify which break up(s) they want to go through, or slices of splits they want to go through, eg: exam[:10%] to “load the 10 initially % of the take a look at split”.
7. Versioning and determinism: TFDS datasets and collection are versioned, so it is achievable to reproduce experiments reliably. Loading a dataset pinned at a certain edition will always return the very same established of examples. This will work with slicing and worldwide shuffling as well, as those are deterministic.
8. Code-considerably less sharing: TFDS can go through TFDS geared up datasets even if the code made use of to get ready the dataset is not readily available. This facilitates sharing and versioning datasets.
9. Local community datasets and assistance for internal datasets within companies: TFDS enables organizations to handle distinct corpuses of datasets and make them accessible to their inner buyers.
10. Formats-specific builders: to conveniently define datasets primarily based on well known formats this sort of as CoNLL.
11. GCS integration: TFDS performs effectively with GCS.

Thank you to all of our contributors and consumers!

What is future?

TFDS is below lively growth to deliver you the greatest datasets to use as enter in your ML pipelines.

Notably, we perform on producing transformations seamless. Sometimes, a dataset is derived from an additional dataset by a several transformations (e.g., details augmentation or column renaming). We want all those transformations to be as uncomplicated to employ as attainable. This feature is already readily available experimentally, don’t be reluctant to give feedback on GitHub!

We are also working on creating the TensorFlow dependency optional. TFDS is a framework agnostic library that presents datasets and equipment to aid device mastering investigate. TFDS does not rely on any specific device studying framework, and we are doing the job to make the TensorFlow dependency optional.

We have other designs also, more compact types this sort of as the aid of partitioned datasets, and more time-term kinds that could durably affect the field. Stick to us on GitHub to get potential updates about these impending developments!