Slow Gitlab Integration connection

Slow Gitlab Integration connection - Klaros Test Management - Forum - Klaros Test Management

Christopher Huber, modified 4 Years ago.

Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

Hello everybody,

I'm working with the community edition of the Klaros-Testmanagement and I want to integrate it with Gitlab to create Issues.

I've added a Gitlab Integration in the config menu and then I linked it to a project. Until now, everything is fine.

BUT:
If I now want to create a new Issue (it doesn't matter if I create it after a test or directly in the testcase), then this takes up to 40 seconds to finish the request. The first slow down starts if I click on "New Issue". Then the UI doesn't response for 10 seconds and then I see the UI to add a new Issue. After filling out everything I hit "Save" and it takes 30 seconds to response "Issue successfully created".

If I take a look on the gitlab server, then I see, that there is a high CPU workload. (ca. 50%). So now I'm wondering, whats happen there.
The problem is, that I can't see how many request are send from Klaros to Gitlab or back. So can you help me to understand the communication between Klaros and Gitlab, to find out if the problem is on the Klaros side or on the Gitlab side.

Kind regards
Christopher

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

Hello Christopher,

can you tell me how many projects are hosted on this server (roughly)?

Regards,

Torsten

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

Hello Torsten,

I've 260 Projects in 3 different groups.
The AccessToken I've added in Klaros is from a user which has only access to ONE project. This is also the project I've added in the configuration of Klaros.

Regards,
Christopher

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

If I try to call the Gitlab API with a CURL command, than I can create an issue in under 2seconds. Also when I try to GET some informtion about the group or project. (i.e. the members or the milestones). This is also done in a few seconds.

So please, can you tell me, how you retrieve the information from gitlab and how you create a new issue?

Thank you very much.

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

Could it happen, that the request will be send multiple times?? I mean, after I've send a new Issue to gitlab, then Klaros retrieves the new information about the inserted issue. But could it happen, that this information request will be triggered multiple times?

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection (Answer)

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

To answer your question: Yes, there are multiple requests going on when creating an issue on to the GitHub server, mostly for retrieving project metadata (Milestones, Priorities, etc.). Some of them may be repeated.

You did not state the version of Klaros you are using, this is of importance here.

We are leveraging a third party software to connect to Gitlab. Before version 4.12.5, this implementation connected to all projects on the GitLab server, we patched that behavior in 4.12.5, so only the configured project will be retrieved. I am suspecting that this may be the case here.

So if you are using a version below 4.12.5 please update and retry.

On the downside we also had to remove connection caching in 4.12.5 as it was interfering with the ability to select an assignee during issue creation. This may as well have a detrimental influence here, but I wouldn't expect this in this magnitude.

If you are experienced with using Wireshark or a logging http/https proxy I would be interested to see what is going on at comunication with your Gitlab server.

Regards,

Torsten

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

I'm using the latest Klaros Version 4.12.5. running in a docker container. And also gitlab is running in a docker container.

Currently I've no option for wireshark or something like that. Because of the dockerized structure.

But I will try to get the HTTP communication, so that we can see what happens.

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

I've installed Klaros 4.12.5 on my local machine and captured the network traffic with Wireshark. Sadly I'm using https, so I can't see the complete request but I found some interesing packages:

First I got a request to gitlab with a Hello packets and some data, than after some seconds I got an reset signal:

	No. Time Protocol Length Info

	89 4.652329 TLSv1.2 647 Client Hello
	93 5.174111 TLSv1.3 140 Application Data
	94 5.174495 TLSv1.3 379 Application Data

	139 8.234023 TCP 54 60523 → 443 [RST] Seq=1049 Win=0 Len=0

Then the request repeats with the same Data and length. So there are again packets with data (same as above).

	No. Time Protocol Length Info

	270 11.264539 TLSv1.3 647 Client Hello TCP 54
	274 11.301455 TLSv1.3 140 Application Data
	275 11.301778 TLSv1.3 379 Application Data

	339 14.119048 TCP 54 60556 → 443 [RST] Seq=1049 Win=0 Len=0

This looks a littlebit strange for me.... Is there a way to log the messages from Klaros? Maybe with another log level in any property file?

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

Let me have a look ...

The file to edit is: webapps/klaros-web/WEB-INF/classes/logback.xml

We are using logback with slf4j. jul-to-slf4j is already present. The GitLab API uses the standard Java HttpURLConnection class.

You will probably have to change the default log level and add the correct log level settings.

I was not yet able to try this out.

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

I've now logged the following action:

Open a Testcase, hit "Issue" and select "new". (execution time untill the UI opens: ca. 30 seconds)
Then I got the following output: (see attachment)

You can see that there are 5 Messages which are repeated 5 times.

One round tooks 3 seconds. So if this is repeated 5 times with some delay between the requests, then we got this 20 till 30 seconds.

But what can we do now?? I'm totally fine with the 3 seconds for one round... but it is not so nice that this occurs 5 times.

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

Thank you for the analysis, this is interesting.

First of all, I honestly think that 3 seconds for a single request is unacceptable in any case.

I am unsure if this is based in our changes regarding connection pooling applied, so I added a modified version of the connector to this post. Please replace this file (located in webapps/klaros-web/WEB-INF/lib) restart the server and retry.

If this still does not significantly better your situation then I strongly suspect a cause outside of Klaros.

Sadly, this will reintroduce issue KLAROS-4298 (Unable to select an assignee when creating/editing GitLab issues) so this is not a permanent solution.

Please let me know your results.

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

Hello again,

you are right. 3 seconds is to long to wait. but it is better than 30 emoticon

Thank you for that modified version... but it is still not working. Now I got another issue:

The Authentication is wrong.

I looked into the logs and there is a strange behavior to check if the gitlab server is available.

1. A request is send to: /api/v4/projects/182/issues/2147483647 with an API Token (which is valid).
This is ok, but I'm confused about the issue-id 2147483647. This issue doesn't exists so the server get an 404 not found.

2. Then a second request is send to: /api/v4/user which looks good, BUT there is NO ApiToken in the request. So the Server got an 401 Unauthorized

More requests are not possible because of an exception, that the username/token is invalid, which is not the case.

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

Thanks for the detailed report.

The /api/v4/projects/182/issues/2147483647 request is a probe for validating the connection parameters.
It should only be done once per user session to a Gitlab project and is intentional.

I am unsure about the error message regarding the authentication error you receive, I am going to validate this and get back to you.

Christopher Huber, modified 4 Years ago.

RE: Slow Gitlab Integration connection

Youngling Posts: 9 Join Date: 2/26/20 Recent Posts

Did you found anything related this topic?
or was you able to reproduce the issue?

Torsten Stolpmann, modified 4 Years ago.

RE: Slow Gitlab Integration connection (Answer)

Jedi Council Member Posts: 755 Join Date: 2/12/09 Recent Posts

Hello Christopher,

our automated tests are running a login, issue creation and retrieval in GitLab in roughly 2.5 sec (with GitLab running in a virtualized environment on crap hardware). I am unsure on what we can do on our side to remedy your situation.

Can you rule out SSL-negotiation problems with your GitLab installation? If it is exposed to the internet https://www.ssllabs.com/ssltest/analyze.html might help to diagnose.

We will keep an eye on this, but I am missing the right idea here.

Torsten

P.S.: I am travelling this week, so do not expect fast responses.