Questions tagged [mesos]

Apache Mesos is a cluster manager that provides an efficient, scalable, and robust way to share resources (CPUs, memory, disk, ports, et.) across frameworks or applications that are distributed across a cluster of machines. Mesos can be thought of as a kernel for the data center that provides a unified view of resources on all nodes and seamless access to these resources.

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Trying to use log4j configuration sent with --files param on spark-submit

I am trying to pass a custom log4j configuration file to the spark executor. So far i am using the --files param to spark-submit and can indeed attest that the respective file is transported in the ...
omu_negru's user avatar
  • 4,682
0 votes
1 answer
34 views

deregistered instance showing unreachable in mesos , how to clear this?

i am using mesos-marathon cluster for my setup and i have 40 instances registered with mesos . i want to remove 1 instance from the cluster. To do that , i just stopped that ec2 instance and reduce ...
Sandeep Grover's user avatar
1 vote
1 answer
298 views

Configuring a self hosted GitLab-CE CI/CD server in parallel with a GitHub Team account

I am looking to perform code reviews as part of a trunk based development approach with my backend team on a self hosted gitlab server. I also wish to use it to deploy to internally hosted ...
Toby Derrum's user avatar
0 votes
1 answer
51 views

How to set log level in chronos

I'm trying to change the loglevel in chronos from the default INFO to TRACE. I have gone through the docs and they seem to be outdated as per this issue. I have done a local setup using docker but to ...
James Dube's user avatar
0 votes
0 answers
63 views

configure: error: Mesos requires Python < 3.0

I want to install Mesos on Oracle Linux 8.3 which does not have Internet access. I followed the process exactly in its site in part CentOS 7.1 and installed some packages before building mesos. I ...
M_Gh's user avatar
  • 1,070
0 votes
0 answers
1k views

Exception in thread "main" org.apache.spark.SparkException: Driver cores must be a positive number

I have problem with submit task in Mesos with cluster mode, first I using this syntax to running cluster mode on Mesos $ cd spark $ ./sbin/start-mesos-dispatcher.sh --master mesos://10.2.3.95:5050 ...
Calvin Valentino Gosal's user avatar
0 votes
0 answers
49 views

Task/Job Failed on Mesos

enter image description here Anyone know why every task/job that I run using spark submit on my apache Mesos cluster is always STATE FAILED? For your information, I using spark mesos dispatcher to run ...
Calvin Valentino Gosal's user avatar
0 votes
1 answer
51 views

Integrating hadoop yarn with mesos infra

I have created a hdfs cluster . I have to configure yarn so as to allow yarn application master to be able to create containers for job processing on the mesos cluster on demand . how can i integrate ...
Pun Master's user avatar
1 vote
1 answer
53 views

Once tasks are finished does Mesos Agent kill the Mesos Executor that was running the tasks or does the Mesos Executor persist to receive new tasks?

I'm trying to better understand when Mesos Executors are killed by Mesos Agents once tasks that are running in the Mesos Executors finish. Does the Mesos Agent automatically kill Mesos Executors if ...
user18561105's user avatar
2 votes
1 answer
233 views

Where can I find binary packages for Apache Mesos?

I have been trying to install mesos on my VM. To be specific on Ubuntu Focal Fossa. However, I googled this and didn't saw any repository/binary packages for mesos. Can someone help me with this? I ...
Abhishek Sinha's user avatar
0 votes
1 answer
18 views

Standalone mesos master behind load balancer EC2

I have a mesos master behind a load balancer and a mesos agent that tries to connect to the mesos master via the load balancer Everything is good when the agent directly connects to the master by ...
ffff's user avatar
  • 2,884
0 votes
1 answer
145 views

I wonder why many resource managers (Yarn, Mesos, Kubernetes etc.) do not support setting the disk I/O resource usage of containers

In most resource managers, we can set the container's CPU usage and memory usage. But I'm curious about the technical reasons for not supporting disk I/O resource allocation for containers.
김민우's user avatar
2 votes
1 answer
217 views

Is Celery a "real" scheduler like PBS, MESOS or YARN?

I have an existing application that is using Celery. Clients submit tasks to Celery, and Celery's workers unstack those tasks and run it, accross different physical hosts. Each Celery worker run a ...
Klun's user avatar
  • 104
4 votes
4 answers
1k views

Installing Mesos on ubuntu 20.04 causing makefile issue

I was trying to install mesos latest version(1.9.0) on the ubuntu 20.04 using Dockefile. FROM ubuntu:20.04 ENV MESOS_VERSION 1.9.0 ENV MESOS_ARTIFACT_FILENAME mesos-${MESOS_VERSION}.tar.gz # Install ...
Bruce wayne - The Geek Killer's user avatar
0 votes
1 answer
283 views

Apache Mesos/Chronos task status is not getting updated and stuck as RUNNING status

I am using Mesos 1.3.1 and Chronos in my local. I currently have 100 jobs scheduled every 30 minutes for testing. Sometimes the tasks get stuck in RUNNING status forever until I restart the Mesos ...
mye1102's user avatar
  • 41
1 vote
0 answers
63 views

Mesos/Chronos How are the tasks getting updated to FINISHED task status?

I am new to Mesos and trying to understand the flow when the task is successfully finished and how it is updated to Mesos with Chronos. (in version 2.4.0) When I have a job where the task finished ...
mye1102's user avatar
  • 41
0 votes
1 answer
41 views

Mesos implementation

I have two Django websites that create a Spark Session to a Cluster which is running on Mesos. The problem is that whatever Django starts first will create a framework and take 100% the resources ...
Alejandro A's user avatar
  • 1,170
0 votes
1 answer
99 views

API - GET TASKS - custom filters

Is there a way to filter results of GET_TASKS in Mesos API ? http://mesos.apache.org/documentation/latest/operator-http-api/#get_tasks There is no info about filters, like for example get TASKS that ...
Dawid PR's user avatar
  • 117
0 votes
1 answer
59 views

mesos master with single quorum

during the test with 2 master nodes(2 quorum), the source and frameworks are not visible normally. so I changed the quorum from 2 to 1 and it works normally. problems occur in all cases with quorum ...
HelloRacoon's user avatar
0 votes
1 answer
50 views

Marathon Getting appId: Specify a path

I was Deploying Mesos Marathon cluster on ! Master server and 2 slave server. I Set up everything Fine with the load balancer but after setup everythinh i am getting this error from marathon. I was ...
damagebrain's user avatar
-1 votes
1 answer
104 views

appId: error.pattern when create application in Marathon

I'm trying to create an application in Marathon using web console. Here is the JSON file: { "id": "TestSpringApplication", "cmd": "/usr/bin/java -jar -...
Chuong Nguyen's user avatar
1 vote
0 answers
163 views

Spark BlockManager listening to localhost

Not sure why and already looked into this answer, with no luck. My executors are listening to their localhost address, runining any kinf of shuffle operation as they can't talk to eacother. I am ...
Alejandro A's user avatar
  • 1,170
1 vote
1 answer
838 views

Pyspark, executors lost connectivity when performing a join

My mesos-spark cluster: Executors are crashing every time I try to do a .count() after a join, the count without the join works perfectly, not sure why but in failed queries I see: And in the ...
Alejandro A's user avatar
  • 1,170
0 votes
1 answer
64 views

Does a Mesos slave needs to be contacted by a Mesos master?

Can Apache Mesos 'slave' nodes be located on a separate network than the Mesos 'master' nodes? Similarly (for high-availability (HA) deploys), can the Apache Zookeeper nodes used in Mesos 'master' ...
Dimitri Kopriwa's user avatar
0 votes
1 answer
180 views

Spark+Mesos Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

I have a master on the cloud with 3 masters and 10 slaves. All the slaves appear active but 0 resources alocated in the Mesos Master UI: In the home page I can see 10 activated Agents but 9 are ...
Alejandro A's user avatar
  • 1,170
0 votes
1 answer
89 views

Can't access Mesos UI

Can't access Mesos UI on the master:5050. Running a small cluster 1 master-3 slaves on a Linux dist. Master seems to be picked OK: I0902 09:43:23.234127 19479 zookeeper.cpp:262] A new leading master (...
Alejandro A's user avatar
  • 1,170
0 votes
1 answer
50 views

Modifying number of tasks executed on mesos slave

In a Mesos ecosystem(master + scheduler + slave), with the master executing tasks on the slaves, is there a configuration that allows modifying number of tasks executed on each slave? Say for example, ...
Tushar's user avatar
  • 528
0 votes
0 answers
215 views

Mesos failed with Error: Failed to load class ... Command exited with status 101 Spark3, Mesos 1.9.0, Jdk headless 11 RHEL 7

We current project works with jdk8, Cent7, Spark 2.3.1 and MESOS 1.4.1. We are upgrading to jdk11 headless and RHEL7, Spark 3.0.0 and Mesos 1.9.0. On a new cluster that I created with the upgrades, ...
Susi's user avatar
  • 1
1 vote
2 answers
168 views

Gracefully exiting akka cluster node during rolling deployment

We are running a akka cluster in docker and running in Mesos. The structure is such that 3 different applications (each having 4 instances) talk to each other within the cluster When we want to do a ...
slowhandblues's user avatar
0 votes
1 answer
77 views

Scheduling spark job via marathon

I want to schedule spark job to run on daily basis via marathon. I am using mesos as cluster manager. How to schedule a job to run only once a day via marathon. Right now the job keeps on running ...
TJJ's user avatar
  • 31
0 votes
0 answers
36 views

How to use "Revocable resource" in cluster (Spark + Mesos)

We tried to use "Revocable resource" option but no success due didn't find any information how to use it. So our question is: How to create a task in Spark (on Mesos) in right way to use "Revocable ...
zh_'s user avatar
  • 95
0 votes
1 answer
171 views

Mesos parameters for enabling communication over SSL with Zookeeper-3.5.7

Recently, I've been working on securing communication between Mesos master and zookeeper by setting up SSL authentication. I followed the steps as listed here to have client-server communication ...
Tushar's user avatar
  • 528
1 vote
0 answers
29 views

dcos package install - resource request declined due to no port need?

I am following https://docs.konghq.com/install/dcos to install kong on my newly setup DCOS cluster. However I am blocked at the step Deploy Marathon-LB. The service cannot be deployed because there ...
wxh's user avatar
  • 688
1 vote
0 answers
29 views

spark run on top of mesos - cluster mode - aws instances EC2 (1 master 1 slave)

i am facing issue , i am trying to deploy a try out spark app (wordcount, written in java) i would run spark on top of mesos cluster (1 master, 1 agent slave) task failed, it seems master cant ...
Offset sigorski's user avatar
0 votes
0 answers
348 views

apche mesos - install ubuntu 18.04 - start master failed - mesos version 1.6.0

I tried to install Apache Mesos 1.4.0 on Ubuntu 18.04, but start master failed error ./bin/mesos-master.sh: line 24: /home/hdpu00/mesos-1.4.0/build/src/mesos-master: No such file or directory ...
Offset sigorski's user avatar
2 votes
3 answers
459 views

Marathon (Mesos) - Stuck in "Loading applications"

I am building a mesos cluster from scratch (using Vagrant, which is not relevant for this issue). OS: Ubuntu 16.04 (trusty) Setup: Master -> Runs ZooKeeper, Mesos-master, Marathon and Chronos Slave ->...
Cristian's user avatar
  • 369
0 votes
1 answer
86 views

Mesos masters behind load balancer

I have 3 Mesos masters running and all are registers with Zookeepers. 3 Mesos URL :- http://mesos1.test.com:5050 http://mesos2.test.com:5050 http://mesos3.test.com:5050 Now, All 3 URLs can be ...
user1578872's user avatar
  • 8,268
0 votes
1 answer
237 views

How do I understand that caching is used in Spark?

In my Scala/Spark application, I create DataFrame. I plan to use this Dataframe several times throughout the program. For that's why I decided to used .cache() method for that DataFrame. As you can ...
Nurzhan Nogerbek's user avatar
0 votes
1 answer
392 views

Where can I locate the 'stdout' and 'stderr' log files on Mesos Slave machines?

I am trying to access the echo and print logs of my cron tasks in the Mesos Slave machines. I can see the stdout and stderr logs from the GUI but can't seem to locate them on the machine. So far I ...
Vishakha Lall's user avatar
0 votes
1 answer
55 views

Can VM's be added to Mesos cluster?

I'm trying to understand what Mesos can (and can't) do. It looks to be similar to Kubernetes except that it manages resources other than just containers. My question is, could one of those resources ...
Ramy's user avatar
  • 20.7k
0 votes
1 answer
77 views

Single instance of a marathon application creates multiple docker containers

I have followed the sample json app cfg @ https://mesosphere.github.io/marathon/docs/native-docker.html#bridged-networking-mode (modified instance count to 1 instead of 2) on the following local setup ...
ZeroGraviti's user avatar
  • 1,047
0 votes
1 answer
28 views

Error while running make check for mesos-1.9.0 on Mac os

While running make check for mesos-1.9.0 on mac os, I get an error for 'ulimit -u`. This is the error message on Terminal Detected low process count ulimit (2837 vs 4096). Increase 'ulimit -u' to ...
ZeroGraviti's user avatar
  • 1,047
4 votes
1 answer
7k views

How to set docker command --log.level=DEBUG in a marathon deployment file?

reading at this docker-compose file : (Source: https://docs.traefik.io/user-guides/docker-compose/acme-dns/#setup) version: "3.3" services: traefik: image: "traefik:v2.1.0" container_name: ...
Dimitri Kopriwa's user avatar
0 votes
2 answers
306 views

Controlling the number of spark drivers running on Mesos

We have a cluster with 6 EC2 nodes in AWS (16 cpu, 64 gb per node - 1 node running mesos master and 5 nodes running mesos slave). The mesos dispatcher is running on the master node. We use this ...
Srinivas K's user avatar
0 votes
1 answer
57 views

Mesos agent memory control with resource flag

When I was trying to limit memory on mesos agent with --resource flag, it seems doesn't work at all. The slave can still use all the cpus and memory on virtual machine. Did I use it wrong?
Bo Cai's user avatar
  • 1
0 votes
1 answer
184 views

How to expose mongo containers to outside of the overlay network?

I am running a mongodb replica set as containers in Mesos cluster ( My question also applied to k8s/swarm). The replica set can be initialized using DNS or container IPs. But, both DNS or container ...
Haoyuan Ge's user avatar
  • 3,459
1 vote
1 answer
140 views

Broken garbage collection in Mesos

I hit a strange problem with potentially broken stale frameworks garbage collection on mesos slaves (1.9.0-2.0.1.debian8). As per documentation stale frameworks should be automatically deleted if ...
elxakpe's user avatar
  • 11
2 votes
0 answers
2k views

Apache Flink Resource Manager (App Master) fails allocating new Task Managers after the first one is killed after idle timeout

I am using Apache Flink 1.9.1 on Apache Mesos 1.8.1 using more or less standard settings for both on a 3 nodes cluster. The first node (192.168.10.11) runs the Mesos master and the Flink App Master. ...
Jerome Kehrli's user avatar
0 votes
1 answer
149 views

What's the difference between Apache's Mesos infrastructure and Virtual Machine Cluster

What exactly is the difference between Apache's Mesos and Virtual Machine Cluster? I understand both are server cluster management software. Can anyone elaborate where the main differences? Is it ...
Sandeep's user avatar
  • 225
2 votes
1 answer
72 views

How are frameworks in Mesos notified when tasks scheduled by them finish?

My question is a combination of this and this question on stackoverflow, however, answers there don't help me. I want to know that when in a Mesos cluster a task corresponding to a framework finishes ...
PHcoDer's user avatar
  • 1,176

1
2 3 4 5
25