8

Frequently last few days onwards our GitLab(CE) running slowly. We have a hook for the CI with Jenkins. We had installed the GitLab by OmniAuth. I don't have any more ideas regarding this because we didn`t do anything new in our instances.

We are the newbie to GitLab environment. We are working in the GitLab since December 2016 and also we never faced this kind of issue before. I hope that I will fix this problem with you people. Kindly help me to fix the issue.

Follow the below image for our Gitlab details. enter image description here

enter image description here

How could I overcome from this issue?

1
  • note to future users, if only your "git pull/push/clone" commands are getting slow then it is highly likely that something may also be wrong with the ssh server configuration. for me, it was the GSSAPIAuthentication. check out this thread: superuser.com/a/191624
    – emrekgn
    May 9, 2018 at 9:21

2 Answers 2

3

These are just some as-is suggestions offered without warranty, but they may help guide you to solving the problem.

Occam's Razor

You mentioned that these issues appear to have just started most recently. This means that the VERY FIRST place to look is what may have changed around the time that these issues occurred. If you have change control for your infrastructure, start there. Make absolutely sure nobody has changed anything around the time these issues started happening. Check your logs for any warnings that may have started showing up. If your OS has a security log or logs configuration changes, check those. If you don't have good visibility/audit-ability into your environment, this may be hard, but if you can identify something that changed around the same time as these issues started occurring, that is most often going to be your problem.

Specificity

It may be helpful for you to describe what you mean by it getting slow. Is it a specific operation that is slow? Or is it all activity? If it's something specific, like triggering a Jenkins job, then you can start to isolate your search there.

It can also help to run top on your server to get a picture of what might be causing the issues. There might be a specific process running on the machine that is dominating everything else and eating all of the resources.

Hardware

First thing I would check is to make sure your hardware configuration matches the 'Hardware requirements' guidelines on gitlab's website: https://docs.gitlab.com/ce/install/requirements.html#requirements Based on what you've posted, the CPU and memory on your system seem adequate for several thousand users, so I'm going to assume this isn't a problem, but in case you do have thousands of users, I will add some brief information on this. Your disk configuration (other than size) is not presented in the information above, so we don't know if that is sufficient or not.

I would recommend running vmstat on the server (since it's GitLab, I am assuming this is running on Linux, since they do not recommend Windows installations) to get some basic information about what is going on. The vmstat command will give you several columns of information. To the very left there should be a column 'r'. This is the 'run queue', or the number of processes that are waiting to be run on a CPU. If the value in that column is large compared to the number of cores the system has, you probably have a CPU bottleneck. The next column, 'b', is processes that are blocked. If this is large, you probably don't have a CPU bottleneck. To the right, there are CPU columns: us, sy, id, or something along these lines. These columns are a breakdown of where the CPU is spending its time, either in the application code (us), in the OS code (sy), or waiting (id). High percentage numbers in us generally indicate that you either are running healthily or have a CPU bottleneck. High percentage numbers in sy are usually going to indicate some kind of contention, possibly a configuration issue like having too many worker threads configured for the number of CPUs you have. A high percentage number in id usually indicates that the system either isn't doing much, or can't do much because it's waiting on something like disk or an external database.

So if the 'b' and/or 'id' columns in your vmstat output have high numbers, we may want to consider the possibility of there being an I/O bottleneck. Here are a couple introductory articles on evaluating Linux IO for bottlenecks that might help you determine if this is the case: https://bartsjerps.wordpress.com/2011/03/04/io-bottleneck-linux/ http://www.linux-mag.com/id/2001/ These articles should get you pointed in the right direction to help you decide if your disks aren't fast enough.

One thing to note, if you're seeing what appears to be a CPU bottleneck (high r values, high us values), make sure that situation makes sense for the number of users you have. The CPU bottleneck may be caused a virtualization issue, or some OS issue causing the CPU to perform poorly, not just by the CPU hardware itself being insufficient.

Topology

One thing mentioned in the gitlab requirements I linked to above is that it is not recommended to run GitLab runner on the same box as GitLab itself. This is something I would say is true for any CI software working with GitLab. If you're running GitLab Runner or Jenkins on the same box as GitLab itself, you should consider moving those to their own hardware.

If you have thousands of users, you may wish to get in contact with GitLab themselves and have consulting on how to get an enterprise-grade cluster stood up and what that looks like. There are people who are experts in the specific hardware configurations that make sense for a very large GitLab installation, and I am not one of them. However, if you don't have a large number of users, the hardware you have is probably not the issue.

Software

If you're running things like vmstat and iostat and you're not finding any specific hardware bottleneck, there may be a configuration issue. Make sure you have a good number of Unicorn Workers configured, so that the box can properly utilize your hardware.

External bottlenecks

Make sure things like network speed on the server are sufficient for its needs. Make sure users trying to reach the server aren't being bottlenecked by a misconfigured network. If you're using OmniAuth, make sure the provider is performing correctly. For example, if you're using some external authentication, and that isn't scaling/performing well, you'll get bad performance in GitLab as well. These are especially important to look at if you're not seeing much hardware utilization using the methods above.

5
  • I have following doubts. Kindly clarify it
    – user6874415
    Apr 6, 2017 at 6:32
  • 1. I tried iostat in my instance. But it says The program 'iostat' is currently not installed. You can install it by typing: apt install sysstat What is the use of the iostat? 2. I noticed that the cache(free 21669960, buff 509120, cache 6877532) memory is high when I run vmstat. How could I clear this? Is anything affect if I clear the cache?
    – user6874415
    Apr 6, 2017 at 6:44
  • The cache is good and you want it. You don't want to try to get rid of the cache.
    – Dogs
    Apr 7, 2017 at 0:24
  • Thank you, What about iostat?
    – user6874415
    Apr 7, 2017 at 4:05
  • 1
    @Arunkumar Regarding iostat, it says you need to install it through the sysstat package. Simply run apt-get install sysstat then run iostat
    – Jawad
    Apr 13, 2017 at 8:46
0

Two aspects which can help accelerate GitLab are, in the latest April 2020 12.10 version:

The last point is:

When fetching changes from a Git repository, the server advertises a list of all the branches and tags in the repository, known as refs.

In some instances, we have observed up to 75% of all requests to the GitLab web server are requests for the refs.
In the best case, when all the refs are packed, this is an inexpensive operation.

However, when there are unpacked refs, Git must iterate over the unpacked refs. This causes additional disk I/O, which is slow when using high latency storage like NFS.

In GitLab 12.10, info/refs are cached to improve the performance of ref advertisement and decrease the pressure on Gitaly in situations where refs are fetched very frequently.

In testing this feature on GitLab.com, we observed read operations outnumber write operations 10 to 1, and saw median latency decrease by 70%.
For GitLab instances using NFS for Git storage, we expect even greater improvements.

See Documentation and Issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.