Questions tagged [fb-hydra]
Questions about Hydra, an open source Python framework by Facebook
169
questions
0
votes
1
answer
12
views
Python Hydra - Is this possible to list config and yaml file?
I have below file hirarchy from which i want to list config path and yaml files
Main_dir:
conf:
project:
test_project1:
env:
source:
src.yaml
dest:
...
0
votes
0
answers
9
views
Python Hydra- Is there any way to print config path and file using hydra compose api?
I need to get the list of config directory/path name and files inside the config path from below directory
Main_dir:
conf:
project:
test_project1:
env:
source:
src....
1
vote
0
answers
14
views
Use Hydra to populate nested dataclass objects
I have a yaml config file that has the following structure
paths:
log: ./runs
data: ./data
data:
downloads: [
{
name: "1",
url: "...
1
vote
1
answer
60
views
How to use hydra compose API to read yaml from a nested directory?
I am trying to print the src.yaml from a nested directory below
Main_dir:
conf:
project:
test_project1:
env:
source:
src.yaml
src.yaml
table: ...
0
votes
1
answer
10
views
Hydra defaults using YAML: How to handle optional user configuration?
I'm distributing an application which includes a several commands.
Some commands require no user configuration as sane defaults can be packaged with the app
Some commands require additional user ...
0
votes
0
answers
23
views
Easiest way to override PosixPath in hydra
Consider the following yaml file for hydra config:
a:
b: !!python/object/apply:pathlib.PosixPath
- /my/path/to/dir
How would I override a.b so that is stays PosixPath after providing a new ...
0
votes
1
answer
19
views
refined selection of log-level of modules in Hydra
I'm trying to debug a Hydra application, setting hydra.verbose=true will set the logging level of all modules to Debug.
Is there a way to get rid of some of the modules debug messages that produce ...
0
votes
0
answers
26
views
How to gather several datasets (dataset configurations) in a list with hydra?
I am using hydra for the configuration of deep learning projects. I want to put together several datasets for the training. As the number of datasets is a priori not defined, I want to get the ...
0
votes
0
answers
13
views
Running Hydra from IDE
That may seem a naive question, but is there a way to run use Hydra from an IDE?
I am interested in using it within anaconda spyder, but if there is way to run experiments with hydra from another IDE ...
0
votes
1
answer
25
views
Where does hydra find its keys in Python?
I'm working in the nuPlan devkit (https://github.com/motional/nuplan-devkit/tree/master/tutorials), specifically in the nuplan_framework tutorial, and I reach this error:
MissingConfigException: In '...
0
votes
1
answer
109
views
Hydra omegaconf resolvers on dictionaries and lists
I'm trying to use omegaconf resolvers on dictionaries and lists but I keep getting an error.
for example the following code gives a hydra.errors.OverrideParseException
(taken from here Hydra combine ...
0
votes
2
answers
159
views
OmegaConf - how to delete a single parameter
I have a code that looks something like this:
def generate_constraints(c):
if c.name == 'multibodypendulum':
con_fnc = MultiBodyPendulum(**c)
where c is an OmegaConf object containing a ...
0
votes
1
answer
111
views
Overriding Hydra config with experiment
I'm working through this Hydra doc example to override the main config with an experiment config. I'm trying to override main with experiment and I'm not sure where I'm going wrong.
The error I get is:...
1
vote
1
answer
51
views
How can I organize Hydra 1.3 configuration files and import them into each other?
I am trying to use Hydra 1.3 contrive a simple, but apparently not trivial, configuration that maps endpoints of a given API to their corresponding processing functions.
So far, I came up with a ...
0
votes
0
answers
108
views
Is there a way to use Ray Tune in combination with ML Flow and Hydra in Python?
I want to do hyperparameter tuning for a neural net, created with keras. For this project I handle my config.yaml files with hydra, use mlflow to store the metrics and parameters from the optimization ...
0
votes
0
answers
89
views
How to customize hydra run dir and sweep subdir based on values of a sweep?
I am using hydra to organize my configurations and output folders for a project. This project involves first specifying a particular objective that can differ in a few parameters. Then that objective ...
0
votes
1
answer
75
views
Key 'optuna_config' not in 'OptunaSweeperConf' full_key: hydra.sweeper.optuna_config object_type=OptunaSweeperConf
I tried this tutorial but gives me this error Key 'optuna_config' not in 'OptunaSweeperConf' full_key: hydra.sweeper.optuna_config object_type=OptunaSweeperConf
why is this error?
this is my config....
0
votes
1
answer
95
views
In hydra, can I interpolate config from a file without using the defaults list?
Say I have a directory utility_configs that has a bunch of different configurations for different things that are useful in different situations. And then I want to be able to use these different ...
0
votes
1
answer
130
views
Hydra - accessing parent of parent directory
is it possible to have a 2-degree sibling file as a default configuration?
Files hierarchy:
dir_1
├── dir_1_1
│ └── dir_1_1_1
│ └── conf.yaml
└── dir_1_2
└── dir_1_2_1
└── ...
1
vote
1
answer
163
views
Set Hydra enviroment with uvicorn/fastapi
I have a FastAPI application that is going to production soon, however i am facing some problems with hydra integration.
First i could not run the @hydra.main() decorator on fastAPI endpoints. This ...
1
vote
1
answer
207
views
extend list from default config in Hydra
I have a list that is defined in my defaults
configuration file base_list:
list:
- 1
- 2
I know I can override the list values in the config file:
defaults:
- base_list
list:
- 3
- 4
which ...
1
vote
1
answer
573
views
How to access the Hydra config object at runtime
I need to change the output/working directory of the hydra config framework in such a way that it lies outside of my project directory. According to my understanding and the doc, config.yaml would ...
1
vote
2
answers
72
views
How can I access to a help subsection from a hydra config file in a command line?
I am currently using the hydra config file as a part of a deep learning framework, in order to declare several parameters.
I wrote a help section in my config.yaml file to declare all the parameters ...
0
votes
1
answer
354
views
Hydra config: using another config as an input to multiple keys within a config
Suppose I have the following directory structure for my hydra configs:
config
|_config.yml
operations
|_subconfig.yml
subconfig.yml is
param_1: foo
param_2: bar
and config.yml is:
...
0
votes
1
answer
360
views
Access Hydra configuration parameters from different files
I'm trying to integrate hydra into my reinforcement learning code and I would like to access configuration parameters from different files, containing various functions and classes.
Short Example:
...
0
votes
1
answer
117
views
Hydra - Specifying config group with a keyword other than the directory
Say I have the following file structure:
|--configs
| |--config.yaml
| |--A
| |--conf1.yaml
| |--conf2.yaml
| |--conf3.yaml
|--main_app.py
and my base config.yaml looks like
defaults:
...
0
votes
0
answers
243
views
AttributeError: __aenter__ on asynchronous websocket python
I try to resolve this problem about asynchronous WebSocket using python. I don't know why when I used the asynchronous process I get the error log bellow. But my code is totally fine when I didn't ...
0
votes
1
answer
71
views
Schema validation in Hydra not working when configuration path is parent folder
I have the following project setup:
configs/
├── default.yaml
└── trainings
├── data_config
│ └── default.yaml
├── simple.yaml
└── schema.yaml
The content of the files are as ...
1
vote
1
answer
217
views
is there a way to modify log directory in hydra?
I am trying to modify log dir in hydra.
In default, output dir in hydra is like below.
outputs
└── 2022-11-12
├── 18-17-28
│ └── train.log
├── 18-18-37
│ └── train.log
└── 18-...
2
votes
0
answers
48
views
Hydra: disable permutation of few hyperparameters
I am trying to do a basic hyperparameter tuning. By default Hydra creates a permutation of each hyperparameter.
hydra:
mode: MULTIRUN
sweeper:
params:
+n: 5,10,15
+...
1
vote
1
answer
333
views
Conditional intializations of parameters in hydra
I'm pretty new to hydra and was wondering if the following thing is was possible: I have the parameter num_atom_feats in the model section which I would like to make dependent on the feat_type ...
0
votes
2
answers
94
views
Using multiple models with repeated keywords in `hydra`
I'm pretty new to hydra and I'm trying to better understand the config.yaml file. I'm undertaking a deep learning experiment where I have two separate models, an embedding network and a simple fully ...
1
vote
1
answer
356
views
How to dynamically define project root path in config file?
I want to define project root path in Hydra config file without hardcoding it. 'til now couldn't find any feature of Hydra that can imply the path!
E.g., suppose we have the following project ...
0
votes
0
answers
282
views
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of size: : [50] Unet integration
I modified this repository to use hydra and add U-net model
https://github.com/kuangliu/pytorch-cifar
I run this:
python3 main.py --config-name=unet_train params.epoch_count=5
I got this error, but ...
3
votes
2
answers
2k
views
What are Hydra advantages vs using a regular configuration file
I wonder what are the advantages of using Hydra to manage my configuration files,
versus loading .yaml configuration file directly (using import yaml)?
1
vote
1
answer
165
views
How to create cli to access multi cli commands using hydra?
I want to package the repo and provide a cmd-line entry point for all the separate tasks managed by hydra. Let me explain with an example.
I have multiple tasks that are all powered by hydra. Each ...
0
votes
1
answer
162
views
How to access all file names in hydra config
I have a directory contains a bunch of txt files:
dir/train/[train1.txt, train2.txt, train3.txt]
I'm able to read a single file, if I define following in a config.yaml
file_name: ${paths.data_dir}/...
1
vote
0
answers
3k
views
ModuleNotFoundError: No module named 'hydra'
I'm trying to run a Python script but it gives an error immediately at the first line of code while trying importing hydra library via:
import hydra
The error i get is the following:
Traceback (most ...
1
vote
1
answer
118
views
Handling high number of configuration parameters in python
In my Machine Learning project I have a high number of parameters that are loaded from a configuration file, e.g. a YAML file. I wonder, is there any best practice on how to integrate them in the ...
1
vote
1
answer
131
views
Python - Hydra - Obtaining configuration in sub-module
Question
I am currently having trouble with getting hydra to work when not using @hydra.main in the main.py script (refer to error section). I have tried the compose API, but that prevents me from ...
0
votes
1
answer
158
views
How to use a config group multiple times, while overriding each instance
Here is my current config structure
hydra/
pipeline/
common/
feature.yaml
stage/
train.yaml
with the following files:
train.yaml
# @package _global_
defaults:
- _self_
- ../...
0
votes
1
answer
722
views
How to use Hydra Config alongside user-defined parameters?
Is there a way to use Hydra Config alongside other parameters?
I want some arguments to be defined in the code, and others to be read from the config file.
@hydra.main(config_path="conf", ...
0
votes
2
answers
271
views
Using SageMaker with Hydra
I have a question about SageMaker and Hydra.
TL;DR
Is there a way to pass arguments from SageMaker estimator to a Hydra script? Currently it passes parameters in a very strict way.
Full Question
I use ...
1
vote
1
answer
363
views
define a value dynamically using hydra for yaml files
Let's say I have an app.py like this
class myClassA :
def __init__(self):
self.id = 100
class myClassB :
def __init__(self, objA, id):
pass
Is there a way to use hydra to have a ...
2
votes
2
answers
191
views
Is there any way to log 'git hash' in hydra?
I want to control version of experiment configuration files with hydra and dvc without uploading original config files to git.
Hydra does control config, and dvc controls version. But Hydra does not ...
0
votes
1
answer
319
views
Can't create a multiprocessing Manager when using hydra
I am developing a code where I need to create a Manager to synchronize a Queue. In the project I am using hydra and everything works fine until I create a multiprocessing Manager Object. This is the ...
2
votes
1
answer
119
views
How to allow the user to override a subset of the configuration using their own yaml file?
Let's say I have this basic app:
from dataclasses import dataclass
import hydra
from hydra.core.config_store import ConfigStore
@dataclass
class MyAppConfig:
req_int: int
opt_str: str = &...
1
vote
1
answer
2k
views
Hydra install on python 3.10 fails due to VS build tools
I'm trying to install Hydra 2.5 on a Windows 10 system. I have Visual Studio Build Tools 2022 installed with the desktop C++ development option. When I use pip I get the error attached below. I've ...
2
votes
2
answers
279
views
Passing argument/key pair to vscode Python debugger separated with an equal sign
For management of my config files, I'm using Hydra which requires passing additional arguments using a plus and then an equal sign between the argument and its value, e.g.
python evaluate.py '+...
2
votes
2
answers
880
views
Use a parameter multiple times in hydra config file
I am currently trying to replace the usage of argparse with hydra files to set the hyperparameters of a deep learning neural network.
I succeeded in using a config.yaml file linked to a hydra main ...