Issues

Slow Gitlab Integration connection

Threads [ Previous | Next ]
RSS (Opens New Window)
Slow Gitlab Integration connection
issue management integration docker gitlab slow api
Answer
2/26/20 11:38 AM
Hello everybody,

I'm working with the community edition of the Klaros-Testmanagement and I want to integrate it with Gitlab to create Issues.

I've added a Gitlab Integration in the config menu and then I linked it to a project. Until now, everything is fine.

BUT:
If I now want to create a new Issue (it doesn't matter if I create it after a test or directly in the testcase), then this takes up to 40 seconds to finish the request. The first slow down starts if I click on "New Issue". Then the UI doesn't response for 10 seconds and then I see the UI to add a new Issue. After filling out everything I hit "Save" and it takes 30 seconds to response "Issue successfully created".

If I take a look on the gitlab server, then I see, that there is a high CPU workload. (ca. 50%). So now I'm wondering, whats happen there.
The problem is, that I can't see how many request are send from Klaros to Gitlab or back. So can you help me to understand the communication between Klaros and Gitlab, to find out if the problem is on the Klaros side or on the Gitlab side.

Kind regards
Christopher
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 11:51 AM as a reply to Christopher Huber.
Hello Christopher,

can you tell me how many projects are hosted on this server (roughly)?

Regards,

Torsten
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 11:53 AM as a reply to Torsten Stolpmann.
Hello Torsten,

I've 260 Projects in 3 different groups.
The AccessToken I've added in Klaros is from a user which has only access to ONE project. This is also the project I've added in the configuration of Klaros.

Regards,
Christopher
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 12:45 PM as a reply to Christopher Huber.
If I try to call the Gitlab API with a CURL command, than I can create an issue in under 2seconds. Also when I try to GET some informtion about the group or project. (i.e. the members or the milestones). This is also done in a few seconds.

So please, can you tell me, how you retrieve the information from gitlab and how you create a new issue?

Thank you very much.
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 12:58 PM as a reply to Torsten Stolpmann.
Could it happen, that the request will be send multiple times?? I mean, after I've send a new Issue to gitlab, then Klaros retrieves the new information about the inserted issue. But could it happen, that this information request will be triggered multiple times? 
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 1:17 PM as a reply to Christopher Huber.
To answer your question: Yes, there are multiple requests going on when creating an issue on to the GitHub server, mostly for retrieving project metadata (Milestones, Priorities, etc.). Some of them may be repeated.

You did not state the version of Klaros you are using, this is of importance here.

We are leveraging a third party software to connect to Gitlab. Before version 4.12.5, this implementation connected to all projects on the GitLab server, we patched that behavior in 4.12.5, so only the configured project will be retrieved. I am suspecting that this may be the case here.

So if you are using a version below 4.12.5 please update and retry.

On the downside we also had to remove connection caching in 4.12.5 as it was interfering with the ability to select an assignee during issue creation. This may as well have a detrimental influence here, but I wouldn't expect this in this magnitude.

If you are experienced with using Wireshark or a logging http/https proxy I would be interested to see what is going on at comunication with your Gitlab server.

Regards,

Torsten
 
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 1:24 PM as a reply to Torsten Stolpmann.
I'm using the latest Klaros Version 4.12.5. running in a docker container. And also gitlab is running in a docker container.

Currently I've no option for wireshark or something like that. Because of the dockerized structure. 

But I will try to get the HTTP communication, so that we can see what happens.
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 3:11 PM as a reply to Torsten Stolpmann.
I've installed Klaros 4.12.5 on my local machine and captured the network traffic with Wireshark. Sadly I'm using https, so I can't see the complete request but I found some interesing packages:

First I got a request to gitlab with a Hello packets and some data, than after some seconds I got an reset signal:
1234567
No.  Time      Protocol    Length    Info

89   4.652329  TLSv1.2     647       Client Hello
93   5.174111  TLSv1.3     140       Application Data
94   5.174495  TLSv1.3     379       Application Data

139  8.234023  TCP         54        60523 → 443 [RST] Seq=1049 Win=0 Len=0


Then the request repeats with the same Data and length. So there are again packets with data (same as above). 

1234567
No.  Time      Protocol    Length    Info

270  11.264539  TLSv1.3     647       Client Hello TCP 54
274  11.301455  TLSv1.3     140       Application Data
275  11.301778  TLSv1.3     379       Application Data

339  14.119048  TCP         54        60556 → 443 [RST] Seq=1049 Win=0 Len=0

This looks a littlebit strange for me.... Is there a way to log the messages from Klaros? Maybe with another log level in any property file?
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/26/20 3:32 PM as a reply to Christopher Huber.
Let me have a look ...

The file to edit is: webapps/klaros-web/WEB-INF/classes/logback.xml

We are using logback with slf4j. jul-to-slf4j is already present. The GitLab API uses the standard Java HttpURLConnection class.

You will probably have to change the default log level and add the correct log level settings.

I was not yet able to try this out.
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/27/20 10:16 AM as a reply to Torsten Stolpmann.
I've now logged the following action:

Open a Testcase, hit "Issue" and select "new". (execution time untill the UI opens: ca. 30 seconds)
Then I got the following output: (see attachment)

You can see that there are 5 Messages which are repeated 5 times.

One round tooks 3 seconds. So if this is repeated 5 times with some delay between the requests, then we got this 20 till 30 seconds.

But what can we do now?? I'm totally fine with the 3 seconds for one round... but it is not so nice that this occurs 5 times.

Attachments:
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/27/20 2:40 PM as a reply to Christopher Huber.
Thank you for the analysis, this is interesting.

First of all, I honestly think that 3 seconds for a single request is unacceptable in any case.

I am unsure if this is based in our changes regarding connection pooling applied, so I added a modified version of the connector to this post. Please replace this file (located in webapps/klaros-web/WEB-INF/lib) restart the server and retry.

If this still does not significantly better your situation then I strongly suspect a cause outside of Klaros.

Sadly, this will reintroduce issue KLAROS-4298 (Unable to select an assignee when creating/editing GitLab issues) so this is not a permanent solution.

Please let me know your results.

Attachments:
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/28/20 8:37 AM as a reply to Torsten Stolpmann.
Hello again,

you are right. 3 seconds is to long to wait. but it is better than 30 emoticon


Thank you for that modified version... but it is still not working. Now I got another issue:

The Authentication is wrong.

I looked into the logs and there is a strange behavior to check if the gitlab server is available.

1. A request is send to: /api/v4/projects/182/issues/2147483647 with an API Token (which is valid).
This is ok, but I'm confused about the issue-id 2147483647. This issue doesn't exists so the server get an 404 not found.

2. Then a second request is send to: /api/v4/user which looks good, BUT there is NO ApiToken in the request. So the Server got an 401 Unauthorized

More requests are not possible because of an exception, that the username/token is invalid, which is not the case.

Attachments:

Attachment

0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
2/28/20 5:05 PM as a reply to Christopher Huber.
Thanks for the detailed report.

The  /api/v4/projects/182/issues/2147483647 request is a probe for validating the connection parameters.
It should only be done once per user session to a Gitlab project and is intentional.

I am unsure about the error message regarding the authentication error you receive, I am going to validate this and get back to you.
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
3/9/20 9:02 AM as a reply to Torsten Stolpmann.
Did you found anything related this topic?
or was you able to reproduce the issue?
0 (0 Votes)
Flag

RE: Slow Gitlab Integration connection
Answer
3/9/20 10:38 PM as a reply to Christopher Huber.
Hello Christopher,

our automated tests are running a login, issue creation and retrieval in GitLab in roughly 2.5 sec (with GitLab running in a virtualized environment on crap hardware). I am unsure on what we can do on our side to remedy your situation.

Can you rule out SSL-negotiation problems with your GitLab installation? If it is exposed to the internet https://www.ssllabs.com/ssltest/analyze.html might help to diagnose.

We will keep an eye on this, but I am missing the right idea here.

Torsten

P.S.: I am travelling this week, so do not expect fast responses.
0 (0 Votes)
Flag