Questions tagged [clearml]
The clearml tag has no usage guidance.
48
questions
0
votes
1
answer
38
views
Replacing IPs in MongoDB
I have a nested schema for mongoDB collection. It looks smth like this:
{
"_id":"61d99bf5544f4822bd963bda0a9c213b",
"execution": {
"test_split":0,
...
0
votes
1
answer
58
views
Automating a Script with ClearML
I am using ClearML to automate a script. I'm new to the tool and I haven't found any tutorials for this. The goal I'm trying to accomplish is automate several python scripts so that a server is spun ...
0
votes
1
answer
129
views
No preview images for dataset in ClearML web UI
After creating a new dataset via CLI for a bunch of images, I closed that dataset and thus uploaded it to our own, newly installed ClearML server. Now in the web UI the new dataset has been created ...
0
votes
1
answer
125
views
ClearML webapp is slow
We manage our own ClearML server, on an EC2 instance AWS cloud.
Instance type: t3.xlarge (4 vCPUs, 16 GiB Memory).
Data disk: gp3 (size: 200 GB, IOPS: 3,000, Throughput: 125).
We have 3 ClearML ...
0
votes
2
answers
272
views
How to run the ClearML Cleanup?
I've installed ClearML test manager solution using ClearML Docker-Compose. So now the whole thing is running using 6 containers (webserver,apiserver,redis,elasticsearch,fileserver and mongodb). I'm ...
1
vote
1
answer
377
views
Experiment tracking for multiple ML independent models using WandB in a single main evaluation
Can you recommend from your experience about choosing a convenient tracking experiment tool and versioning only "Multi independent models, but one input->multi-models->one output" in ...
1
vote
2
answers
87
views
ClearML: How to merge 2 datasets which the 2 datasets inherited from a main dataset?
First I create a root (A) dataset. Then I created 2 dataset branches derived from root dataset.
How do I "merge" these 2 branches to form another dataset ?
Basically, the graph looks like ...
0
votes
1
answer
77
views
How to make ClearML not upload annotations twice when they have the same ID?
The following uploads two annotations, though I expected there to be only one
from typing import List
from allegroai import Dataset, DatasetVersion, SingleFrame, DataView
from allegroai.dataframe....
0
votes
1
answer
367
views
ClearML, how to query the best performing model for a specific project and metric
I want to download the best performing model for a certain ClearlML project. I have the following content in my ClearML experiment platform:
According to: https://clear.ml/docs/latest/docs/...
0
votes
2
answers
123
views
ClearML - dynamically updating Plotly plots?
I have a question related to ClearML plot logging. We are currently using:
self.task_logger.report_table("TableSpaceName", "Some Info", iteration=0, table_plot=df)
To report ...
1
vote
1
answer
76
views
Does ClearML have accounting of information security events
ClearML is one of the most famous MLOps tools existing. It has logging of machine learning processes, however I couldn't find any information regarding its system of accounting of information security ...
2
votes
1
answer
491
views
[Catboost][ClearML] Error: if loss-function is Logloss, then class weights should be given for 0 and 1 classes
Having recently started using ClearML to manage the MLOps, I am facing the following problem:
When running a script that trains a CatBoost in a binary classification problem using different class ...
0
votes
1
answer
37
views
Where do augmentations in ClearML run?
In ClearML Dataviews, it is possible to add augmentations.
Where do these augmentations run?
Options
Original data gets downloaded to local, then runs (on which device? How is multiprocessing handled?...
0
votes
1
answer
162
views
clearml-serving-triton | 2022-06-18 16:21:19,790 - clearml.Task - INFO - No repository found, storing script code instead
i found this error when running this command "docker-compose --env-file example.env -f docker-compose-triton.yml up".
Actually, when i run this command for the first time, it worked. And ...
1
vote
1
answer
235
views
ModuleNotFoundError: No module named 'allegroai'
I am trying to use ClearML.
The only line in my file is
from allegroai import Dataset, DatasetVersion
which yields
ModuleNotFoundError: No module named 'allegroai'
Looks like some pip package is ...
0
votes
1
answer
195
views
Remotely execute ClearML task using local-only repo
I want to execute ClearML task remotely. According to docs there are 2 options: 1) execute single python file; 2) ClearML would identify that script is part of repo, that repo will be cloned and ...
-1
votes
1
answer
413
views
make clearml agent do not install envs for every task
I want make my clearml agent do not install python envs for every task, it take too long.
I tried setup config like: package_manager.system_size_packages=true, but it doesn't work.
clearml agent won't ...
0
votes
1
answer
315
views
ALB host based routing without domain name
I'm trying to configure host based routing in AWS ALB for ClearML server using this tutorial.
However, I don't have a domain name. So can I only use alb's dns for this routing?
For example, I will ...
1
vote
0
answers
196
views
clearml hyperparameter optimization task fails on creating python environment
Trying to use clearml HyperparametersOptimizers. Launching task, but each experiment fails while trying to create python environment. Specificaly the error I get is related to pytorch
clearml_agent: ...
0
votes
1
answer
236
views
How to Deploy a ClearML Agent in a k8s setting?
I want to deploy a ClearML agent in the kubernetes environment while using the ClearML's Free Tier Demo server.
I was able to deploy the Agent pod in the k8s cluster with the allegroai/clearml-agent ...
0
votes
1
answer
190
views
Export metrics of ClearML to Prometheus and show them in Grafana
Are there any metrics I can get from the API server? or any docker image I can point to the backend and get some metrics?
Most important is the see how many tasks running in real-time (like we can see ...
2
votes
1
answer
567
views
What would stop credentials from validation on a ClearML server?
I've set up a ClearML server in GCP using the sub-domain approach. I can access all three domains (https://app.clearml.mydomain.com, https://api.clearml.mydomain.com and https://files.clearml.mydomain....
0
votes
1
answer
280
views
how to capture logger values using clearml
I am using clearml for testing algorithms and it works well with library Stable Baselines 3, in which clearml automatically captures all the output and plot them in the Scalars tab.
However, when I ...
0
votes
1
answer
233
views
ClearML How to get configurable hyperparameters?
How do I get args like epochs to show up in the UI configuration panel under hyperparameters? I want to be able to change number of epochs and learning rate from within the UI.
1
vote
2
answers
1k
views
Mounting an S3 bucket in docker in a clearml agent
What is the best practice for mounting an S3 container inside a docker image that will be using as a ClearML agent? I can think of 3 solutions, but have been unable to get any to work currently:
Use ...
3
votes
1
answer
443
views
How to manage datasets in ClearML Web UI?
Using a self-deployed ClearML server with the clearml-data CLI, I would like to manage (or view) my datasets in the WebUI as shown on the ClearML webpage (https://clear.ml/mlops/clearml-feature-store/)...
3
votes
1
answer
150
views
ClearML get max value from logged values
I use ClearML to track my tensorboard logs (from PyTorch Lightning) during training.
At a point later I start another script which connects to existing task and do some testing.
But unfortenautly I do ...
1
vote
1
answer
75
views
ClearML Web UI custom column not persistent
I'm using the experiments page of a project in ClearML Web UI to visualize some custom metrics. Therefore I've customized my table vie (https://allegro.ai/clearml/docs/docs/webapp/webapp_exp_table....
1
vote
1
answer
529
views
ClearML multiple tasks in single script changes logged value names
I trained multiple models with different configuration for a custom hyperparameter search. I use pytorch_lightning and its logging (TensorboardLogger).
When running my training script after Task.init()...
2
votes
1
answer
681
views
ClearML how to change clearml.conf file in AWS Sagemaker
I am working in AWS Sagemaker Jupyter notebook.
I have installed clearml package in AWS Sagemaker in Jupyter.
ClearML server was installed on AWS EC2.
I need to store artifacts and models in AWS S3 ...
1
vote
1
answer
251
views
ClearML SSH port forwarding fileserver not available in WEB Ui
Trying to use clearml-server on own Ubuntu 18.04.5 with SSH Port Forwarding and not beeing able to see my debug samples.
My setup:
ClearML server on hostA
SSH Tunnel connections to access Web App ...
3
votes
1
answer
539
views
ClearML server IP address not used with localhost and SSH port forwarding
Trying to use clearml-server on own Ubuntu 18.04.5.
I use env variables to set the IP Address of my clearml-server.
export CLEARML_HOST_IP=127.0.0.1
export TRAINS_HOST_IP=127.0.0.1
But it still is ...
6
votes
1
answer
2k
views
Can ClearML (formerly Trains) work a local server?
I am trying to start my way with ClearML (formerly known as Trains).
I see on the documentation that I need to have server running, either on the ClearML platform itself, or on a remote machine using ...
2
votes
1
answer
54
views
How to fix trainserver empty server? [closed]
Im trying to install an allegroai trains-server on a k8s cluster.
I tried the following 3 methods
bare linux installtion
k8s manifest installation
helm installation
I followed the linux installation ...
1
vote
1
answer
306
views
Trains: reusing previous task id
I am using reuse_last_task_id=True to overwrite an existing task (with same project and task name). But the experiment contains the torch model and therefore does not overwrite the existing task but ...
1
vote
3
answers
1k
views
pip install trains fails
upon running pip install trains in my virtual env
I am getting
ERROR: Command errored out with exit status 1:
command: /home/epdadmin/noam/code/venv_linux/bin/python -u -c 'import sys, ...
2
votes
1
answer
145
views
How should Trains be used with hyper-param optimization tools like RayTune?
What could be a reasonable setup for this? Can I call Task.init() multiple times in the same execution?
1
vote
1
answer
174
views
Will Trains automagically log Tensorboard HParams?
I know that it's possible to send hyper-params as a dictionary to Trains.
But can it also automagically log hyper-params that are logged using the TF2 HParams module?
Edit: This is done in the HParams ...
-1
votes
1
answer
54
views
How resilient is reporting to Trains server?
How would Trains go about sending any missing data to the server in the following scenarios?
Internet connection breaks temporarily while running an experiment
Internet connection breaks and doesn't ...
2
votes
1
answer
420
views
Trains: Can I reset the status of a task? (from 'Aborted' back to 'Running')
I had to stop training in the middle, which set the Trains status to Aborted.
Later I continued it from the last checkpoint, but the status remained Aborted.
Furthermore, automatic training metrics ...
0
votes
1
answer
32
views
Does clone experiment work on sklearn functions?
I'm trying to run a script and I'm constantly getting this while cloning experiment in allegro.ai
AttributeError: 'Namespace' object has no attribute 'get'
Can anybody help?
1
vote
1
answer
211
views
Can Trains config file be specified dynamically or relative to the running script path?
Suppose I have a server where many users run different experiments, possibly with different Trains Servers.
I know about the TRAINS_CONFIG_FILE environment variable, but I wonder if this can be made ...
2
votes
2
answers
144
views
Tracking separate train/test processes with Trains
In my setup, I run a script that trains a model and starts generating checkpoints. Another script watches for new checkpoints and evaluates them. The scripts run in parallel, so evaluation is just a ...
2
votes
1
answer
66
views
Parallel Coordinates Plot in TRAINS
Is there a way to create a parallel coordinates plot in TRAINS (https://github.com/allegroai/trains) package to compare several hyper-parameters in respect to a specific metric?
1
vote
2
answers
134
views
trains with grid search
I would like to test trains usage during grid search and it not clear how to do so.
from trains import Task
Task.init(project_name="project name", task_name='name')
creates an experiment in the ...
1
vote
1
answer
3k
views
How to Backup/Restore TRAINS-server when moving from AMI to local machine
I recently started using TRAINS, with the server in AWS AMI. We are currently using v0.9.0.
I would like to move the TRAINS-server to run on our on-premises kubernetes cluster. However, I don't want ...
1
vote
1
answer
169
views
How to manually register a sci-kit model with TRAINS python auto-magical experiment manager?
I'm working mostly with scikit-learn, as far as I understand, the TRAINS auto-magic doesn't catch scikit-learn model store/load automatically.
How do I manually register the model after I have '...
1
vote
1
answer
436
views
Is there a way to create a graph comparing hyper-parameters vs model accuracy with TRAINS python package?
I would like to run multiple experiments, then report model accuracy per experiment.
I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the ...