Questions tagged [amazon-sagemaker]
Amazon SageMaker is a fully-managed AWS service that enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale.
2,910
questions
0
votes
0
answers
4
views
Opencv Build Wheel Timeout in SageMaker Endpoint Deployment
I am trying to deploy an AWS SageMaker Notebook endpoint. In my requirements.txt file, I include opencv_python. When I try to deploy my endpoint, I get the following error:
"...
-4
votes
0
answers
30
views
How do I work through the code in this example? [closed]
I would like to begin playing with some machine learning models from AWS. I see that they recently released a Bedrock service, and I would like to setup an environment to do that. I would also need to ...
0
votes
0
answers
6
views
unable to extract text using tika in sagemaker, what is the reason for this?
code I've used:
from tika import parser
headers = {
'X-Tika-PDFextractInlineImages': 'true',
}
text = parser.from_file("pdffiles/BG1309020268.pdf",headers=headers)
print(text)
text = parser....
0
votes
0
answers
8
views
How to write a lambda function for the Map task in the sagemaker?
I have a machine learning model deployed in SageMaker, and I'm currently working on a process to handle multiple inputs for predictions. I understand that AWS Lambda can be used to implement a Map ...
0
votes
0
answers
19
views
Invoking SageMaker endpoint failed since timm library was not found in the environment
I deployed a Hugging Face facebook/detr-resnet-50 model using the official code provided here:
https://huggingface.co/facebook/detr-resnet-50?sagemaker_deploy=true
Then, I tried to use the predictor ...
0
votes
0
answers
7
views
How to deploy multiple models in amazon sagemaker on various endpoints
I'm recently interested to Amazon Sagemaker and Rekognition.
I would like to try AWS instances and deploy ML models.
I am going to train and deploy models for diagnoising diseases(Image ...
1
vote
0
answers
29
views
How do I use Sagemaker HyperparameterTuner on a SKLearn Estimator?
I'm following an Amazon Sagemaker workshop to try and leverage several of Sagemaker's utilities instead of running everything off a Notebook as I'm currently doing.
The thing is, in the workshop they ...
0
votes
0
answers
18
views
Save pyspark dataframe as RecordIO protobuf
I want to save my pyspark dataframe in RecordIO protobuf format. I am using Amazon EMR to run my pyspark scripts, and I want to use AWS SageMaker to train a machine learning model.
SageMaker pipe mode ...
0
votes
0
answers
14
views
Path Structure for Processing inputs to step function "AppManaged" and not app managed for a sagemaker Job
I have a use case where i am running a sagemaker processing job in a step function , now for this use case input is :
"ProcessingInputs": [
{
"...
1
vote
1
answer
24
views
Sagemaker Studio Notebook - How to access OS Environ
I am trying to set (at the terminal or in the notebook) an environment variable. I am running the following in the terminal or (with %%bash in the cell) notebook cell:
echo "export MY_KEY='BLAH-...
0
votes
0
answers
19
views
AWS SageMaker Real-Time inference configure storage size
i'm currently using SageMaker real-time inference (and async inference) since September.
Now my endpoint workload increased such that i need to process a couple hundreds of HD photos (each about 30-40 ...
2
votes
1
answer
100
views
Sagemaker serverless ValidationError: oversize body
I have an image classification model stored in s3 location.
If I deploy the model as real-time inference, then I am able to predict any image size, however, if I deploy the model as serverless then I ...
0
votes
0
answers
21
views
+100
Building ML Model in AWS Sagemaker autopilot with Time Series feature
I am building a classification model in AWS Sagemaker using Autopilot. I have imported a dataset which has a time series feature. This features includes a list of values in each cell. The data set is ...
1
vote
0
answers
29
views
Sagemaker giving 'OSError: [Errno 28] No space left on device' when separate model and code zip is given
This looks like an interesting problem.
When I am zipping my model and code together(like below), things work fine.
huggingface_model = HuggingFaceModel(
model_data="s3://abc/xyz/model.tar.gz&...
-4
votes
0
answers
46
views
Reading local excel files on AWS without uploading it to S3 bucket
How to Read local excel files on AWS Sagemaker notebook without uploading it to S3 bucket.
Basically I want to read excel/csv files from local drive, and I don't want to duplicate the same file on s3 ...
0
votes
0
answers
23
views
Using outputs from TrainingStep in Sagemaker Pipeline
My TrainingStep works as needed in my sagemaker pipeline, however I can't seem to access it in any way in the future steps of my sagemaker pipeline.
I am using ProcessingInputs but I can't seem to ...
0
votes
0
answers
33
views
+50
Can't train sagemaker model using recordio protobuf data saved in S3
I want to preprocess data in Amazon EMR using PySpark and train a machine learning model in SageMaker using pipe mode. The issue I am having now is saving data in S3 and feeding it to the model.
...
0
votes
0
answers
7
views
Do I have to send all trainging data columns when requesting a prediction from AWS SageMaker?
I'm using the Javascript library @aws-sdk/client-sagemaker-runtime to interface with AWS SageMaker. I'm able to successfully request predictions from our model like this:
const params = {
...
0
votes
0
answers
13
views
`arguments` method of `ProcessingStep` results in unwanted upload to S3
I want to register a model made via SageMaker Pipelines. Following this as an example. Here is the code for model registration:
import logging
from sagemaker.drift_check_baselines import ...
0
votes
1
answer
16
views
Are instances used in sagemaker pipeline destroyed automatically after the pipeline completion?
I saw that for each step in sagemaker pipeline we define instances on which the step will be run. So, after the completion of each step is the instance destroyed, or does it keep on running?
0
votes
0
answers
28
views
How to setup pytesseract path in amazon sagemaker notebook (cloud)?
I have installed pytesseract using pip install pytesseract in sagemaker notebook now I need to setup tesseract path pytesseract.pytesseract.tesseract_cmd = 'home/studio-lab-user/.conda/envs/studiolab/...
1
vote
1
answer
45
views
Loading a pretrained model from torch.hub in Sagemaker
I'm getting an import error when trying to load a pretrained resnet50 model in Sagemaker Studio. I'm using the provided Pytorch 2.0.0 Python 3.10 GPU optimized image on a t3.medium. My code:
import ...
0
votes
1
answer
19
views
In AWS Sagemaker Pipeline, handing off CreateModelStep to TransformStep results in ModuleNotFoundError: No module named 'inference'
In AWS Sagemaker Pipeline, I have a CreateModelStep that connects to a TransformStep. When I get to the TransformStep, I see the following error thrown.
Traceback (most recent call last):
File "/...
0
votes
1
answer
23
views
Can we use sagemaker model python CLI without using s3 bucket?
I'm trying to train the Anthropic model to understand the table of data and it's meta data(table about how the tables are related). Then use it to generate an SQL to fetch the data from database. I ...
-1
votes
1
answer
39
views
JumpStartEstimator.fit ERROR: UnboundLocalError: local variable 'dataset_train' referenced before assignment
I have a simple model tuning python code, when I run I get the fallowing error, appreciate your help.
ERROR: sagemaker.exceptions.UnexpectedStatusException: Error for
Training job meta-textgeneration-...
0
votes
0
answers
33
views
how to download output artifacts from AWS sagemaker (Python)
I have a code similar to this one
'''
from sagemaker.experiments import Experiment
from sagemaker.experiments import Run
import matplotlib.pyplot as plt
from sagemaker import Session
...
0
votes
0
answers
44
views
SageMaker Pytorch Job Not Finding Custom Dependency Files
I've been trying to get SageMaker to run my job for a week and keep running into the same issue no matter what I try.
Folder Structure
Root:
__init__.py
train.py
/scripts (contains my_dataloader.py ...
0
votes
1
answer
61
views
how to fine-tune stable diffusion in sagemaker jumpstart with different descriptions per image
I'm trying to fine-tune stable diffusion in sagemaker using this method: https://aws.amazon.com/blogs/machine-learning/fine-tune-text-to-image-stable-diffusion-models-with-amazon-sagemaker-jumpstart/
...
0
votes
0
answers
27
views
SageMaker batch transform error Requested instance type cannot work with all containers of the model
I'd appreciate your help with this error: botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTransformJob operation: Requested instance type cannot work ...
0
votes
0
answers
31
views
Sagemaker - build your own docker container from a poetry vm
I am learning sagemaker and would like to build my own Docker container from my code. I looked at examples here: https://github.com/aws/sagemaker-training-toolkit, but I have no experience with Docker ...
0
votes
2
answers
98
views
Deploy LLM using Sagemaker and Langchain
I am trying to deploy Generative AI solution built using Langchain (obviously with LLM at it's core) and Sagemaker. So, the code is not just an inference script but inference pipeline (challenge is ...
0
votes
0
answers
32
views
How do I define SM_CHANNEL_TRAIN for a training step in AWS Sagemaker Pipelines?
I am using AWS Sagemaker Pipelines, and apparently I cannot correctly define the value for SM_CHANNEL_TRAIN. Here is my pipeline definition.
import boto3
import sagemaker
import sagemaker.session
from ...
-1
votes
0
answers
25
views
How to output variable from ProcessingStep to use in the Sagemaker pipeline script?
I'm trying to use factorization machines in sagemaker which need a value for feature_dim which I want for it to be the same number as columns from the training dataset. The problem is that I only ...
0
votes
0
answers
25
views
AWS Sagemaker Endpoint inference "async-invocations" endpoint does not log of my log / print statements
I've exposed a Sagemaker async-invocations endpoint using Python and Flask as explained here. When I invoke the endpoint it gets triggered with correct inputs sent from the invocation and I can see ...
0
votes
1
answer
40
views
ValueError: math domain error in AutoARIMA model of StatsForecast
I am trying to use statsforecast AutoArima for forecasting on below type of data:
zip_code
product_family
week_date
rma_count
12198
ABC
2021-01-03
6.0
61022
DEF
2021-01-03
1.0
43106
GHI
2021-01-03
...
0
votes
0
answers
11
views
Aws neo - RuntimeError('PytorchStreamReader failed locating file constants.pkl: file not found')")
I have a checkpoint.pth file . This is working fine in sagemaker and giving results.
However when i try to use aws-neo-compilation-job i am seeing below error
ClientError: InputConfiguration: We are ...
0
votes
0
answers
37
views
Retry for Sagemaker ModelError
When querying my sagemaker inference endpoint using the following code
config = Config(retries={"max_attempts": 4, "mode": "standard"})
sagemaker = boto3.client("...
0
votes
0
answers
9
views
Sagemaker Clarify Job keeps returning ClientError: Threshold values must be provided for continuous features
Data Config
from sagemaker import clarify
data_config_unbalanced = clarify.DataConfig(
s3_data_input_path='s3://a/b.csv', # S3 object path containing the unbalanced dataset
s3_output_path='s3:...
0
votes
0
answers
49
views
Is It Possible to Automatically Enable Code Whisperer Extension Using SageMaker Lifecycle Configurations?
I've been following the AWS blog post on using SageMaker Lifecycle Configurations for customizing SageMaker Studio. According to the post, the second use case involves the installation of custom ...
0
votes
1
answer
48
views
Sagemaker LLama2 endpoint without huggingface token
I have a model.tar.gz archive with the following files in it:
LICENSE.txt
README.md
Responsible-Use-Guide.pdf
USE_POLICY.md
checklist.chk
consolidated.00.pth
params.json
tokenizer.model
...
0
votes
2
answers
32
views
How to read large text file in Amazon SageMaker?
I want to read multiple text file from s3 bucket which will make up a data frame of 10M records and 900 columns.
But it is taking too long to read data using ml.p3.2xlarge instance. Moreover, I would ...
0
votes
1
answer
47
views
Managing Multiple AWS SageMaker Lifecycle Configurations
I've been exploring SageMaker Lifecycle Configurations, particularly in reference to the setting default LCC in the AWS documentation . According to the documentation, it's possible to attach multiple ...
-1
votes
1
answer
64
views
Sagemaker-huggingface An error occurred (ValidationException) when calling the CreateModel operation: Requested image not found
I'm trying to create an asyncronous sagemaker endpoint as shown here shown here but using a huggingface model, so I need to use one of the huggingface images and for that I check the official list of ...
0
votes
0
answers
22
views
I've trained an autopilot model using AutoML tool in Sagemaker. I want to load this model outside of sagemaker and score data in local machine
I have trained an aws autopilot model on Sagemaker. I am trying to load this model in local and score data. But I'm getting error when I'm trying to load model using pickle.
And when I'm trying to ...
1
vote
1
answer
36
views
How do I detach the HuggingFace SageMaker training?
I am training a HuggingFace model remotely on SageMaker integration. My training job takes more than two hours, and I would like to shut my computer off during training.
I use the following snippet to ...
0
votes
1
answer
22
views
Sagemaker Monitor - MonitoringDatasetFormat as gz
I have created a monitoring schedule to monitor predictions from a Batch Transform job. The schedule runs fine when the input dataset_format in BatchTransformInput is csv. However, my batch job is ...
0
votes
0
answers
25
views
Is there a way to know the structure and code of lamda provided by aws?
I am using aws-sagemaker. Follow the workflow and provide by aws during use
You are trying to use Lambda with
arn:aws:lambda:ap-southeast-2:454466003867:function:PRE-ImageMultiClass
ARN.
But I want to ...
0
votes
1
answer
164
views
Error in deploying Falcon-7B after fine-tuning to AWS SageMaker endpoint using SageMaker Python SDK
I am currently running into a problem with AWS SageMaker where I cannot deploy my fine-tuned Falcon-7B model to a SageMaker endpoint after training it using an AWS training job. I am roughly following ...
0
votes
1
answer
20
views
AWS Sagemaker. How to pass data_capture_config parameter to Pipeline?
I'm trying to monitorize my data so I need to use the DataCaptureConfig object in Sagemaker. The point is that I'm using a Pipeline object that include all my steps. In the documentation it seems that ...
0
votes
1
answer
111
views
Pytorch-Dataloader throws multiprocessing exception when deployed inside gunicorn/flask server
I am deploying my pytorch model in aws-sagemaker container and using gunicorn server for inference
imports i have
Dependency versions
matplotlib = 3.0;
pandas = 2.0;
...