Add tutorials for using the training script and #196

alexander-soare · 2024-05-17T16:14:50Z

What this does

Adds a 4th basic tutorial that explains the training script, and particularly how to navigate the Hydra config system
Adds an advanced tutorial for training policies with different environments/datsets
Updates Makefile to test the advanced tutorial
Side mission: Some tweaks in the main README

Cadene

Will continue review tonight

Cadene · 2024-05-17T17:07:11Z

examples/4_train_policy_with_script.md

+
+Explaining the ins and outs of [Hydra](https://hydra.cc/docs/intro/) is beyond the scope of this document, but here we'll share the main points you need to know.
+
+First, consider that `lerobot/configs` might have a directory structure like this (this is the case at the time of writing):


Suggested change

First, consider that `lerobot/configs` might have a directory structure like this (this is the case at the time of writing):

First, `lerobot/configs` has a directory structure like this:

I'd like this to be more concrete as well. But being a little vague allows room for error as this is a very not-maintainable bit of documentation.

@aliberts do you think we can use a tool like this in CI? https://github.com/UmbrellaDocs/linkspector. That might help a little bit, and we can make it that all things like lerobot/configs have to include a link

@Cadene I made my sentence less apologetic, leaving just one "might".

I dont think we should use "might". We should tell exactly what is expected. No?

Did might -> will (everywhere), and used "like" or "something like". There may be minor refactors that have no bearing on the essentials that this document tries to convey, so I don't want the language to overly-commit.

Cadene · 2024-05-17T17:08:20Z

examples/4_train_policy_with_script.md

+
+**_For brevity, in the rest of this document we'll drop the leading `lerobot/configs` path. So `default.yaml` really refers to `lerobot/configs/default.yaml`._**
+
+When you run the training script, Hydra takes over via the `@hydra.main` decorator. If you take a look at the `@hydra.main`'s arguments you will see `config_path="../configs", config_name="default"`. This means Hydra looks for `default.yaml` in `../configs` (which resolves to `lerobot/configs`).


I would suggest to first give one or more examples, before delving into this complex details.

Hmm, I'm not sure what examples to add that would build the reader up to understanding this point any better. I've added a couple of lines that might help with train-of-thought. Let me know if you have a better approach.

Cadene · 2024-05-17T17:14:12Z

examples/4_train_policy_with_script.md

+Among regular configuration hyperparameters like `device: cuda`, `default.yaml` has a `defaults` section. It might look like this.
+
+```yaml
+defaults:
+  - _self_
+  - env: pusht
+  - policy: diffusion
+```
+
+So, Hydra will grab `env/pusht.yaml` and `policy/diffusion.yaml` and incorporate their configuration parameters (any configuration parameters already present in `default.yaml` are overriden).


Suggestion
"""
default.yaml is always the entry point. It begins by a defaults section likes this:

defaults: - _self_ - env: pusht - policy: diffusion

It means that Hydra will grab env/pusht.yaml and policy/diffusion.yaml and incorporate their configuration parameters (any configuration parameters already present in default.yaml are overriden).

After this defaults section, default.yaml also contains regular configuration hyperparameters like device: cuda, or .... TODO.
"""

Done, with some further tweaks.

examples/4_train_policy_with_script.md

Cadene · 2024-05-17T17:25:19Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


This in-depth tutorial is important, but it would be nice to have a specific markdown file called: 5_reproduce_sota.md with a structured list of training commands + pointers to our model page on the hub + evaluation commands of the pretrained models.

# Table of results TODO Table with link to model page on the hub + link to section in the markdown file ## Diffusion policy on Pusht Model card on the hub: Training command:

TODO

Eval Command

TODO

## Aloha ...

I don't think this is a good idea. It adds more not-maintainable documentation. These details live with the model cards and I don't think they should be repeated elsewhere. IMO, we should just have a zoo.md style file that points to the model cards.

What do you think of adding a 5_reproduce_sota.md which would be a tutorial on how to access the model card list, and how from the model card, access the training command?

Maybe long term with can avoid this maintainable documentation, but right now I have no idea how to access these training commands.

How about I put the commands on the model cards. Then you have command, configuration, and commit hash all in one place. Then I can add a section in the main README to talk about the model cards.

@alexander-soare Yes! Should we do this in this PR?

@alexander-soare A thought: ideally we should have a script to push model card in a standardize format to the hub

Added to this PR.

On your thought: I suppose... it's a one-liner right now, so a script doesn't add much value right?

How to have a script that generates this:

For your info, I will add a logic to automatically populate and tag the dataset card as well. For now they are empty

Ah for the README. Yeah that's a good idea :)

examples/advanced/train_act_pusht/train_act_pusht.sh

Cadene · 2024-05-17T21:09:45Z

examples/advanced/train_act_pusht/train_act_pusht.md

@@ -0,0 +1,62 @@
+In this tutorial we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.


I think this tutorial is too difficult to follow. It's a good start, but we should iterate to simplify as much as possible.

Ideally we should just provide the script to train act on pusht, and comment as much as possible.
And also to provide the script to train diffusion policy on aloha.

Unfortunately, the yaml file is necessary because the observation key changes via the CLI are challenging due to the "." separator and the nested lists. So we'd need two files at least.

I like the MD file as it allows for a more tutorial style voice and structure.

What do you think we could do to make it easier to follow?

Why should we also provide the script for DP / Aloha? What value does it add over one example that gets the point across?

I dont know exactly know, but we should try to compress the information.
When it's too long, people dont read.

I am just sharing high level thoughts sorry ^^'

I've had a read through again and couldn't find any obvious ways to cut it. How about we leave it to user testing (aka merge it and see how people interact with it).

Cadene

Thanks for iterating on this! it's super important :)

Cadene · 2024-05-20T09:10:46Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


What do you think of adding a 5_reproduce_sota.md which would be a tutorial on how to access the model card list, and how from the model card, access the training command?

Maybe long term with can avoid this maintainable documentation, but right now I have no idea how to access these training commands.

examples/4_train_policy_with_script.md

Cadene · 2024-05-20T09:16:19Z

examples/advanced/train_act_pusht/train_act_pusht.md

@@ -0,0 +1,62 @@
+In this tutorial we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.


I dont know exactly know, but we should try to compress the information.
When it's too long, people dont read.

I am just sharing high level thoughts sorry ^^'

Cadene

Nice work! Left some suggestions. Approve to unblock as this PR only needs minor changes.

examples/4_train_policy_with_script.md

Cadene · 2024-05-21T09:22:15Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


@alexander-soare Yes! Should we do this in this PR?

Cadene · 2024-05-21T09:22:47Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


@alexander-soare A thought: ideally we should have a script to push model card in a standardize format to the hub

Cadene · 2024-05-21T09:27:45Z

examples/4_train_policy_with_script.md

+Hydra takes over via the `@hydra.main` decorator. If you take a look at the `@hydra.main`'s arguments you will see `config_path="../configs", config_name="default"`. This means Hydra looks for `default.yaml` in `../configs` (which resolves to `lerobot/configs`).
+
+Therefore, `default.yaml` is the first configuration file that Hydra considers. At the top of the file, is a `defaults` section which looks likes this:


I would simplify

Suggested change

Hydra takes over via the `@hydra.main` decorator. If you take a look at the `@hydra.main`'s arguments you will see `config_path="../configs", config_name="default"`. This means Hydra looks for `default.yaml` in `../configs` (which resolves to `lerobot/configs`).

Therefore, `default.yaml` is the first configuration file that Hydra considers. At the top of the file, is a `defaults` section which looks likes this:

Hydra is setup to read the `default.yaml` (through the `@hydra.main` decorator in [`train.py`](https://github.com/huggingface/lerobot/blob/main/lerobot/scripts/train.py#L143)). At the top of the yaml file, is a `defaults` section which looks likes this:

Cadene · 2024-05-21T09:38:36Z

examples/4_train_policy_with_script.md

+  - policy: diffusion
+```
+
+So, Hydra then grabs `env/pusht.yaml` and `policy/diffusion.yaml` and incorporates their configuration parameters as well (any configuration parameters already present in `default.yaml` are overriden).


I would simplify

Suggested change

So, Hydra then grabs `env/pusht.yaml` and `policy/diffusion.yaml` and incorporates their configuration parameters as well (any configuration parameters already present in `default.yaml` are overriden).

This logic tells Hydra to incorporate configuration parameters from `env/pusht.yaml` and `policy/diffusion.yaml`.

I am hesitating to add this with an example but it should be self explanatory. When you see training.offline_steps: ??? you can guess it should be overriden. And besides this use case, which explicitly indicates an inheritance, we want to avoid override as much as possible.

Note: Be aware of the order as any configuration parameters with the same name will be overidden. Thus, `default.yaml` is overriden by `env/pusht.yaml` which is overidden by `policy/diffusion.yaml`.

@alexander-soare why is there a default for dataset_repo_id?

https://github.com/huggingface/lerobot/blob/main/lerobot/configs/default.yaml#L19

There is a default for dataset_repo_id in order to match the policy and env defaults (I'm assuming... I didn't actually set this up).

I will take your suggestions including the override disclaimer.

we want to avoid override as much as possible.

We can do this by removing a bunch of params in default.yaml. Let's make that a different PR though, this one's gone on long enough.

examples/4_train_policy_with_script.md

Cadene · 2024-05-21T10:08:17Z

examples/4_train_policy_with_script.md

+There's one new thing here: `hydra.run.dir=outputs/train/act_aloha_sim_transfer_cube_human`, which specifies where to save the training output.
+
+---


@alexander-soare Could we have a section on how to load from a config yaml file inside an experiment checkpoint?
Is such a thing even possible?

This would justify why we prefer to have these multi lines commands.

We might want to say that when we diverge too much from the original parameters from the yaml files, then it can be handy to write them down in a new yaml files.

I'm not sure what you mean by "load from a config yaml file inside an experiment checkpoint". I do have a PR in the works that does checkpointing and training resumption, and I plan to add a section here on how to resume training.

It would be nice to be able to reproduce an experiments by loading its config yaml inside its experiment directory.

Ahh, yeah that's a pain with Hydra because of the way that hydra.main decorator works. Maybe there's a way...

Cadene · 2024-05-21T10:11:19Z

examples/advanced/train_act_pusht/train_act_pusht.md

@@ -0,0 +1,64 @@
+In this tutorial we will learn how to adapt a policy configuration to be compatible with a new environment and dataset. As a concrete example, we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.


Could we do:

Could we have this?
advanced/0_train_act_pusht.md
advanced/1_calculate_validation_loss.py
advanced/ressources/act_pusht.yaml

I prefer not to number the advanced tutorials as they don't need to be done in any specific order. On the other hand, I think the basic tutorials have a nice order. What do you think?

I think it's clearer to bundle everything needed for one example into one directory (ptal at transformers https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification)

I dont think our examples are the same as in transformers. I would prefer to keep the indices at the begining.

examples/4_train_policy_with_script.md

Cadene · 2024-05-21T10:17:34Z

examples/advanced/train_act_pusht/train_act_pusht.md

+
+_Side note: technically we could override these via the CLI, but with many changes it gets a bit messy, and we also have a bit of a challenge in that we're using `.` in our observation keys which is treated by Hydra as a hierarchical separator_.
+
+For your convenience, we provide [`act_pusht.yaml`](./act_pusht.yaml) in this directory. It contains the diff above, plus some other (optional) ones that are explained within. Please copy it into `lerobot/configs/policy` (remember from a [previous tutorial](../4_train_policy_with_script.md) that Hydra will look in the `lerobot/configs` directory). Now try running the following.


Suggestion:

"""
For your convenience, we provide act_pusht.yaml in the ressources directory. It contains the diff above, plus some other (optional) ones that are explained within. Please copy it into lerobot/configs/policy with:

cp examples/advanced/ressources/act_pusht.yaml lerobot/configs/policy/act_pusht.yaml

Note: We need to perform this copy because Hydra is set to look in the lerobot/configs directory (see previous tutorial if needed).

Now try running the following:

I thought you wanted to cut this file down :P I prefer not to add a copy command (especially one that needs maintenance because of the paths in it).

It's much clearer than "copy this to this directory". We will need to add unit tests to these tutorials anyway.

Fair enough. Done.

…l_act_pusht

Cadene

Really cool, back to you and we are good.

README.md

Cadene · 2024-05-21T14:44:23Z

examples/4_train_policy_with_script.md

+There's one new thing here: `hydra.run.dir=outputs/train/act_aloha_sim_transfer_cube_human`, which specifies where to save the training output.
+
+---


It would be nice to be able to reproduce an experiments by loading its config yaml inside its experiment directory.

Cadene · 2024-05-21T14:46:55Z

examples/advanced/train_act_pusht/train_act_pusht.md

@@ -0,0 +1,64 @@
+In this tutorial we will learn how to adapt a policy configuration to be compatible with a new environment and dataset. As a concrete example, we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.


I dont think our examples are the same as in transformers. I would prefer to keep the indices at the begining.

Cadene · 2024-05-21T14:47:45Z

examples/advanced/train_act_pusht/train_act_pusht.md

+
+_Side note: technically we could override these via the CLI, but with many changes it gets a bit messy, and we also have a bit of a challenge in that we're using `.` in our observation keys which is treated by Hydra as a hierarchical separator_.
+
+For your convenience, we provide [`act_pusht.yaml`](./act_pusht.yaml) in this directory. It contains the diff above, plus some other (optional) ones that are explained within. Please copy it into `lerobot/configs/policy` (remember from a [previous tutorial](../4_train_policy_with_script.md) that Hydra will look in the `lerobot/configs` directory). Now try running the following.


It's much clearer than "copy this to this directory". We will need to add unit tests to these tutorials anyway.

examples/4_train_policy_with_script.md

Cadene · 2024-05-21T14:50:11Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


How to have a script that generates this:

Cadene · 2024-05-21T14:51:12Z

examples/4_train_policy_with_script.md

@@ -0,0 +1,157 @@
+This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.


For your info, I will add a logic to automatically populate and tag the dataset card as well. For now they are empty

Co-authored-by: Remi <re.cadene@gmail.com>

ready for review

4e9b484

alexander-soare added the 📝 Documentation Improvements or additions to documentation label May 17, 2024

Cadene assigned alexander-soare May 17, 2024

Cadene requested review from aliberts and Cadene and removed request for aliberts May 17, 2024 17:05

Cadene reviewed May 17, 2024

View reviewed changes

Cadene requested changes May 17, 2024

View reviewed changes

alexander-soare added 3 commits May 20, 2024 07:43

Update the README to reflect WandB disabled by default

9691ddd

Merge branch 'update_readme_on_wandb' into tutorial_act_pusht

dbed2ee

revision

43ebb30

alexander-soare force-pushed the tutorial_act_pusht branch from 68ee968 to 43ebb30 Compare May 20, 2024 07:42

Merge remote-tracking branch 'upstream/main' into tutorial_act_pusht

6e439c1

Cadene requested changes May 20, 2024

View reviewed changes

alexander-soare added 4 commits May 20, 2024 12:14

revision

45c6a67

Merge branch 'main' into tutorial_act_pusht

b166040

Merge branch 'main' into tutorial_act_pusht

5d1a498

Merge branch 'main' into tutorial_act_pusht

69a9332

Cadene approved these changes May 21, 2024

View reviewed changes

alexander-soare added 3 commits May 21, 2024 13:00

Merge remote-tracking branch 'upstream/main' into tutorial_act_pusht

2440902

revision

aa4d0d0

Merge remote-tracking branch 'origin/tutorial_act_pusht' into tutoria…

71ba9b5

…l_act_pusht

Cadene approved these changes May 21, 2024

View reviewed changes

alexander-soare and others added 3 commits May 21, 2024 15:54

Update README.md

7e12323

Co-authored-by: Remi <re.cadene@gmail.com>

revision

557ecc4

add file name changes

82c18fc

alexander-soare merged commit e67da1d into huggingface:main May 21, 2024
5 checks passed

alexander-soare deleted the tutorial_act_pusht branch May 21, 2024 15:47


		Explaining the ins and outs of [Hydra](https://hydra.cc/docs/intro/) is beyond the scope of this document, but here we'll share the main points you need to know.

		First, consider that `lerobot/configs` might have a directory structure like this (this is the case at the time of writing):

	First, consider that `lerobot/configs` might have a directory structure like this (this is the case at the time of writing):
	First, `lerobot/configs` has a directory structure like this:


		_For brevity, in the rest of this document we'll drop the leading `lerobot/configs` path. So `default.yaml` really refers to `lerobot/configs/default.yaml`._

		When you run the training script, Hydra takes over via the `@hydra.main` decorator. If you take a look at the `@hydra.main`'s arguments you will see `config_path="../configs", config_name="default"`. This means Hydra looks for `default.yaml` in `../configs` (which resolves to `lerobot/configs`).

		@@ -0,0 +1,157 @@
		This tutorial will explain the training script, how to use it, and particularly the use of Hydra to configure everything needed for the training run.

		@@ -0,0 +1,62 @@
		In this tutorial we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.

		Hydra takes over via the `@hydra.main` decorator. If you take a look at the `@hydra.main`'s arguments you will see `config_path="../configs", config_name="default"`. This means Hydra looks for `default.yaml` in `../configs` (which resolves to `lerobot/configs`).

		Therefore, `default.yaml` is the first configuration file that Hydra considers. At the top of the file, is a `defaults` section which looks likes this:

	So, Hydra then grabs `env/pusht.yaml` and `policy/diffusion.yaml` and incorporates their configuration parameters as well (any configuration parameters already present in `default.yaml` are overriden).
	This logic tells Hydra to incorporate configuration parameters from `env/pusht.yaml` and `policy/diffusion.yaml`.

		There's one new thing here: `hydra.run.dir=outputs/train/act_aloha_sim_transfer_cube_human`, which specifies where to save the training output.

		---

		@@ -0,0 +1,64 @@
		In this tutorial we will learn how to adapt a policy configuration to be compatible with a new environment and dataset. As a concrete example, we will adapt the default configuration for ACT to be compatible with the PushT environment and dataset.


		_Side note: technically we could override these via the CLI, but with many changes it gets a bit messy, and we also have a bit of a challenge in that we're using `.` in our observation keys which is treated by Hydra as a hierarchical separator_.

		For your convenience, we provide [`act_pusht.yaml`](./act_pusht.yaml) in this directory. It contains the diff above, plus some other (optional) ones that are explained within. Please copy it into `lerobot/configs/policy` (remember from a [previous tutorial](../4_train_policy_with_script.md) that Hydra will look in the `lerobot/configs` directory). Now try running the following.

Add tutorials for using the training script and #196

Add tutorials for using the training script and #196

Conversation

alexander-soare commented May 17, 2024

What this does

Cadene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-soare May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cadene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cadene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cadene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-soare May 20, 2024 •

edited