55
questions
0
votes
0
answers
66
views
using clearml to explore learning rates in PyTorch. Tuning(experiments list) task is now DRAFT and data is not showing up on loss monitoring screen
I am using clearml to explore learning rates in PyTorch. Tuning task is now DRAFT and data is not showing up on loss monitoring screen.
step1:(This has been a success.)
import torch
import torchvision
...
0
votes
1
answer
54
views
How to filter clearML jobs by started date in specific time range
from allegroai.backend_api.services.v2_13.tasks import GetAllRequest
task_query_qwargs = {"started": ["2024-04-30T21:00:00", "2024-05-02T21:00:00"]}
GetAllRequest(...
0
votes
1
answer
37
views
What is the process for installing custom .whl packages inside ClearML-Agent?
I'm trying to run my code, which imports a custom Python package. How can I run this code on the agent with the default queue on ClearML? Additionally, could you also guide me through the process of ...
1
vote
1
answer
463
views
setup clearml serving after seting up self-hosted clearml server
I built clearml server by self-hosted
I have followed official tutorial to set up clearml-serving
My config:
CLEARML_WEB_HOST="http://localhost:8080"
CLEARML_API_HOST="http://localhost:...
0
votes
1
answer
23
views
How to save hidden/visible scalars layout in "Compare Experiments" tab?
I am logging many scalars (hundreds, for every class, grouped by metric name) after the validation step. It is really tedious to select the desired scalars after every page reload. It seems like there ...
1
vote
1
answer
123
views
How can I view log in ElasticSearch In Clearml?
I followed the guide at https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac/ to deploy local server
I have implemented successfully
enter image description here
Now, I want ...
0
votes
0
answers
112
views
ClearML Change Debug Samples output destination after moving task to another project and renaming it
I start a ClearML Task with .init() and then start my application, which is tracked by the automatic logging.
However, I get my project and task name when the application starts because my application ...
0
votes
1
answer
50
views
Replacing IPs in MongoDB
I have a nested schema for mongoDB collection. It looks smth like this:
{
"_id":"61d99bf5544f4822bd963bda0a9c213b",
"execution": {
"test_split":0,
...
0
votes
1
answer
243
views
Automating a Script with ClearML
I am using ClearML to automate a script. I'm new to the tool and I haven't found any tutorials for this. The goal I'm trying to accomplish is automate several python scripts so that a server is spun ...
1
vote
1
answer
260
views
No preview images for dataset in ClearML web UI
After creating a new dataset via CLI for a bunch of images, I closed that dataset and thus uploaded it to our own, newly installed ClearML server. Now in the web UI the new dataset has been created ...
0
votes
1
answer
284
views
ClearML webapp is slow
We manage our own ClearML server, on an EC2 instance AWS cloud.
Instance type: t3.xlarge (4 vCPUs, 16 GiB Memory).
Data disk: gp3 (size: 200 GB, IOPS: 3,000, Throughput: 125).
We have 3 ClearML ...
0
votes
2
answers
714
views
How to run the ClearML Cleanup?
I've installed ClearML test manager solution using ClearML Docker-Compose. So now the whole thing is running using 6 containers (webserver,apiserver,redis,elasticsearch,fileserver and mongodb). I'm ...
1
vote
1
answer
624
views
Experiment tracking for multiple ML independent models using WandB in a single main evaluation
Can you recommend from your experience about choosing a convenient tracking experiment tool and versioning only "Multi independent models, but one input->multi-models->one output" in ...
1
vote
2
answers
236
views
ClearML: How to merge 2 datasets which the 2 datasets inherited from a main dataset?
First I create a root (A) dataset. Then I created 2 dataset branches derived from root dataset.
How do I "merge" these 2 branches to form another dataset ?
Basically, the graph looks like ...
0
votes
1
answer
112
views
How to make ClearML not upload annotations twice when they have the same ID?
The following uploads two annotations, though I expected there to be only one
from typing import List
from allegroai import Dataset, DatasetVersion, SingleFrame, DataView
from allegroai.dataframe....
0
votes
1
answer
668
views
ClearML, how to query the best performing model for a specific project and metric
I want to download the best performing model for a certain ClearlML project. I have the following content in my ClearML experiment platform:
According to: https://clear.ml/docs/latest/docs/...
0
votes
2
answers
227
views
ClearML - dynamically updating Plotly plots?
I have a question related to ClearML plot logging. We are currently using:
self.task_logger.report_table("TableSpaceName", "Some Info", iteration=0, table_plot=df)
To report ...
1
vote
1
answer
101
views
Does ClearML have accounting of information security events
ClearML is one of the most famous MLOps tools existing. It has logging of machine learning processes, however I couldn't find any information regarding its system of accounting of information security ...
2
votes
1
answer
1k
views
[Catboost][ClearML] Error: if loss-function is Logloss, then class weights should be given for 0 and 1 classes
Having recently started using ClearML to manage the MLOps, I am facing the following problem:
When running a script that trains a CatBoost in a binary classification problem using different class ...
0
votes
1
answer
55
views
Where do augmentations in ClearML run?
In ClearML Dataviews, it is possible to add augmentations.
Where do these augmentations run?
Options
Original data gets downloaded to local, then runs (on which device? How is multiprocessing handled?...
0
votes
1
answer
259
views
clearml-serving-triton | 2022-06-18 16:21:19,790 - clearml.Task - INFO - No repository found, storing script code instead
i found this error when running this command "docker-compose --env-file example.env -f docker-compose-triton.yml up".
Actually, when i run this command for the first time, it worked. And ...
2
votes
1
answer
593
views
ModuleNotFoundError: No module named 'allegroai'
I am trying to use ClearML.
The only line in my file is
from allegroai import Dataset, DatasetVersion
which yields
ModuleNotFoundError: No module named 'allegroai'
Looks like some pip package is ...
0
votes
1
answer
656
views
Remotely execute ClearML task using local-only repo
I want to execute ClearML task remotely. According to docs there are 2 options: 1) execute single python file; 2) ClearML would identify that script is part of repo, that repo will be cloned and ...
-1
votes
1
answer
756
views
make clearml agent do not install envs for every task
I want make my clearml agent do not install python envs for every task, it take too long.
I tried setup config like: package_manager.system_size_packages=true, but it doesn't work.
clearml agent won't ...
0
votes
1
answer
396
views
ALB host based routing without domain name
I'm trying to configure host based routing in AWS ALB for ClearML server using this tutorial.
However, I don't have a domain name. So can I only use alb's dns for this routing?
For example, I will ...
1
vote
0
answers
288
views
clearml hyperparameter optimization task fails on creating python environment
Trying to use clearml HyperparametersOptimizers. Launching task, but each experiment fails while trying to create python environment. Specificaly the error I get is related to pytorch
clearml_agent: ...
0
votes
1
answer
333
views
How to Deploy a ClearML Agent in a k8s setting?
I want to deploy a ClearML agent in the kubernetes environment while using the ClearML's Free Tier Demo server.
I was able to deploy the Agent pod in the k8s cluster with the allegroai/clearml-agent ...
0
votes
1
answer
276
views
Export metrics of ClearML to Prometheus and show them in Grafana
Are there any metrics I can get from the API server? or any docker image I can point to the backend and get some metrics?
Most important is the see how many tasks running in real-time (like we can see ...
2
votes
1
answer
1k
views
What would stop credentials from validation on a ClearML server?
I've set up a ClearML server in GCP using the sub-domain approach. I can access all three domains (https://app.clearml.mydomain.com, https://api.clearml.mydomain.com and https://files.clearml.mydomain....
0
votes
1
answer
536
views
how to capture logger values using clearml
I am using clearml for testing algorithms and it works well with library Stable Baselines 3, in which clearml automatically captures all the output and plot them in the Scalars tab.
However, when I ...
0
votes
1
answer
398
views
ClearML How to get configurable hyperparameters?
How do I get args like epochs to show up in the UI configuration panel under hyperparameters? I want to be able to change number of epochs and learning rate from within the UI.
1
vote
2
answers
2k
views
Mounting an S3 bucket in docker in a clearml agent
What is the best practice for mounting an S3 container inside a docker image that will be using as a ClearML agent? I can think of 3 solutions, but have been unable to get any to work currently:
Use ...
3
votes
1
answer
600
views
How to manage datasets in ClearML Web UI?
Using a self-deployed ClearML server with the clearml-data CLI, I would like to manage (or view) my datasets in the WebUI as shown on the ClearML webpage (https://clear.ml/mlops/clearml-feature-store/)...
3
votes
1
answer
184
views
ClearML get max value from logged values
I use ClearML to track my tensorboard logs (from PyTorch Lightning) during training.
At a point later I start another script which connects to existing task and do some testing.
But unfortenautly I do ...
1
vote
1
answer
97
views
ClearML Web UI custom column not persistent
I'm using the experiments page of a project in ClearML Web UI to visualize some custom metrics. Therefore I've customized my table vie (https://allegro.ai/clearml/docs/docs/webapp/webapp_exp_table....
1
vote
1
answer
892
views
ClearML multiple tasks in single script changes logged value names
I trained multiple models with different configuration for a custom hyperparameter search. I use pytorch_lightning and its logging (TensorboardLogger).
When running my training script after Task.init()...
2
votes
1
answer
1k
views
ClearML how to change clearml.conf file in AWS Sagemaker
I am working in AWS Sagemaker Jupyter notebook.
I have installed clearml package in AWS Sagemaker in Jupyter.
ClearML server was installed on AWS EC2.
I need to store artifacts and models in AWS S3 ...
1
vote
1
answer
359
views
ClearML SSH port forwarding fileserver not available in WEB Ui
Trying to use clearml-server on own Ubuntu 18.04.5 with SSH Port Forwarding and not beeing able to see my debug samples.
My setup:
ClearML server on hostA
SSH Tunnel connections to access Web App ...
2
votes
1
answer
768
views
ClearML server IP address not used with localhost and SSH port forwarding
Trying to use clearml-server on own Ubuntu 18.04.5.
I use env variables to set the IP Address of my clearml-server.
export CLEARML_HOST_IP=127.0.0.1
export TRAINS_HOST_IP=127.0.0.1
But it still is ...
7
votes
1
answer
2k
views
Can ClearML (formerly Trains) work a local server?
I am trying to start my way with ClearML (formerly known as Trains).
I see on the documentation that I need to have server running, either on the ClearML platform itself, or on a remote machine using ...
2
votes
1
answer
63
views
How to fix trainserver empty server? [closed]
Im trying to install an allegroai trains-server on a k8s cluster.
I tried the following 3 methods
bare linux installtion
k8s manifest installation
helm installation
I followed the linux installation ...
1
vote
1
answer
469
views
Trains: reusing previous task id
I am using reuse_last_task_id=True to overwrite an existing task (with same project and task name). But the experiment contains the torch model and therefore does not overwrite the existing task but ...
1
vote
3
answers
1k
views
pip install trains fails
upon running pip install trains in my virtual env
I am getting
ERROR: Command errored out with exit status 1:
command: /home/epdadmin/noam/code/venv_linux/bin/python -u -c 'import sys, ...
2
votes
1
answer
177
views
How should Trains be used with hyper-param optimization tools like RayTune?
What could be a reasonable setup for this? Can I call Task.init() multiple times in the same execution?
1
vote
1
answer
201
views
Will Trains automagically log Tensorboard HParams?
I know that it's possible to send hyper-params as a dictionary to Trains.
But can it also automagically log hyper-params that are logged using the TF2 HParams module?
Edit: This is done in the HParams ...
0
votes
1
answer
63
views
How resilient is reporting to Trains server?
How would Trains go about sending any missing data to the server in the following scenarios?
Internet connection breaks temporarily while running an experiment
Internet connection breaks and doesn't ...
2
votes
1
answer
697
views
Trains: Can I reset the status of a task? (from 'Aborted' back to 'Running')
I had to stop training in the middle, which set the Trains status to Aborted.
Later I continued it from the last checkpoint, but the status remained Aborted.
Furthermore, automatic training metrics ...
0
votes
1
answer
38
views
Does clone experiment work on sklearn functions?
I'm trying to run a script and I'm constantly getting this while cloning experiment in allegro.ai
AttributeError: 'Namespace' object has no attribute 'get'
Can anybody help?
1
vote
1
answer
322
views
Can Trains config file be specified dynamically or relative to the running script path?
Suppose I have a server where many users run different experiments, possibly with different Trains Servers.
I know about the TRAINS_CONFIG_FILE environment variable, but I wonder if this can be made ...
2
votes
2
answers
207
views
Tracking separate train/test processes with Trains
In my setup, I run a script that trains a model and starts generating checkpoints. Another script watches for new checkpoints and evaluates them. The scripts run in parallel, so evaluation is just a ...