Add new tutorial: Multi-Objective NAS with Ax #2006
Conversation
…ed nbformat as dependency; removed weights_format argument to allow newer version of lightning
Fixes to Ax tutorial
✅ Deploy Preview for pytorch-tutorials-preview ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
There was a problem hiding this comment.
Thanks, @Balandat! Looks great! I added a few editorial suggestions and there seems to be a problem with data loading. From the log:
Aug 14 22:51:14 ax.exceptions.core.NoDataError: Observed data is required for generation step #1 (model MOO), but fetched data was empty. Something is wrong with experiment setup -- likely metrics do not implement fetching logic (check your metrics) or no data was attached to experiment for completed trials.
Hmm this is interesting. I've never ran into this issue before, this seems to either a problem with running the training jobs or at least with accessing their logs. A few different folks are able to run this locally, maybe it's an issue with the environment / permissions of the circleci runner? Is there a way to run this as a docker image or ssh into the job to understand what's going on? |
Co-authored-by: Svetlana Karslioglu <[email protected]>
|
@danielrjiang just pointed out to me that the trials are completing in one second, which is not good - it means that the training actually fails. I'll dig into why that may be the case (likely some issue with the environment or the relative location of the script file...). @d4l3k is there a straightforward way to view the logs kept by TorchX in a programmatic way (without using the command line tool |
Co-authored-by: Nikita Shulga <[email protected]>
|
Our CI system (and .py to -> .ipynb converter) is currently implemented around the idea, that single file corresponds to single tutorial. As this tutorial comprises of two files, it:
Both problems can be easily fixed by blocklisting stripping code/rendering of auxilary file, but I'm curious how Jupyther notebook will look like in this case, and whether it is a good experience from user point of view, when they can not see part of the tutorial code on a single page. |
|
Thanks for catching the issue! I'm linking the training script in the tutorial file, so it should be pretty straightforward for the user to view it. But I do take your point about that potentially being confusing / not as clear as it could be if it's not shown on the same page, so I'd be happy to consider changing this if it makes sense. Note that we can just pass the source code of the training script as a string to the TorchX runner. So one option could be to define the code directly in the notebook. The issue with that is that if it's a string it won't have syntax highlighting etc. and so it would be hard to read. Maybe there is some fancy way to extract the source code of a specific cell into a string (though since this is a .py file I guess the concept of a cell doesn't even exist yet prior to the parsing)? I'm not super familiar with what one could do here using rst functionality, maybe there is a way to render the separate py file in some minipage as part of the tutorial? |
Add option (to be used by #2006) to specify extra files particular tutorial depends on Make sure those files will not get deleted Add a bit more type annotation to the code
Adds a new tutorial for how to perform multi-objective Neural Architecture Search with Ax. The tutorial walks the user through a case study that shows how to use Bayesian optimization in order to find Pareto-efficient tradeoffs between model complexity and performance in a sample-efficient fashion. The tutorial uses PyTorch lightning and TorchX to run a number of actual training jobs in the background, and uses Ax+BoTorch to automate the multi-objective optimization.
Built website output: Multi-Objective NAS with Ax — PyTorch Tutorial [BUILD].zip
cc @dme65, @d4l3k