Slow Gitlab Integration connection - Klaros Test Management - Forum - Klaros Test Management
Slow Gitlab Integration connection
Christopher Huber, modified 4 Years ago.
Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
Hello everybody,
I'm working with the community edition of the Klaros-Testmanagement and I want to integrate it with Gitlab to create Issues.
I've added a Gitlab Integration in the config menu and then I linked it to a project. Until now, everything is fine.
BUT:
If I now want to create a new Issue (it doesn't matter if I create it after a test or directly in the testcase), then this takes up to 40 seconds to finish the request. The first slow down starts if I click on "New Issue". Then the UI doesn't response for 10 seconds and then I see the UI to add a new Issue. After filling out everything I hit "Save" and it takes 30 seconds to response "Issue successfully created".
If I take a look on the gitlab server, then I see, that there is a high CPU workload. (ca. 50%). So now I'm wondering, whats happen there.
The problem is, that I can't see how many request are send from Klaros to Gitlab or back. So can you help me to understand the communication between Klaros and Gitlab, to find out if the problem is on the Klaros side or on the Gitlab side.
Kind regards
Christopher
I'm working with the community edition of the Klaros-Testmanagement and I want to integrate it with Gitlab to create Issues.
I've added a Gitlab Integration in the config menu and then I linked it to a project. Until now, everything is fine.
BUT:
If I now want to create a new Issue (it doesn't matter if I create it after a test or directly in the testcase), then this takes up to 40 seconds to finish the request. The first slow down starts if I click on "New Issue". Then the UI doesn't response for 10 seconds and then I see the UI to add a new Issue. After filling out everything I hit "Save" and it takes 30 seconds to response "Issue successfully created".
If I take a look on the gitlab server, then I see, that there is a high CPU workload. (ca. 50%). So now I'm wondering, whats happen there.
The problem is, that I can't see how many request are send from Klaros to Gitlab or back. So can you help me to understand the communication between Klaros and Gitlab, to find out if the problem is on the Klaros side or on the Gitlab side.
Kind regards
Christopher
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
Hello Christopher,
can you tell me how many projects are hosted on this server (roughly)?
Regards,
Torsten
can you tell me how many projects are hosted on this server (roughly)?
Regards,
Torsten
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
Hello Torsten,
I've 260 Projects in 3 different groups.
The AccessToken I've added in Klaros is from a user which has only access to ONE project. This is also the project I've added in the configuration of Klaros.
Regards,
Christopher
I've 260 Projects in 3 different groups.
The AccessToken I've added in Klaros is from a user which has only access to ONE project. This is also the project I've added in the configuration of Klaros.
Regards,
Christopher
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
If I try to call the Gitlab API with a CURL command, than I can create an issue in under 2seconds. Also when I try to GET some informtion about the group or project. (i.e. the members or the milestones). This is also done in a few seconds.
So please, can you tell me, how you retrieve the information from gitlab and how you create a new issue?
Thank you very much.
So please, can you tell me, how you retrieve the information from gitlab and how you create a new issue?
Thank you very much.
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
Could it happen, that the request will be send multiple times?? I mean, after I've send a new Issue to gitlab, then Klaros retrieves the new information about the inserted issue. But could it happen, that this information request will be triggered multiple times?Â
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection (Answer)
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
To answer your question: Yes, there are multiple requests going on when creating an issue on to the GitHub server, mostly for retrieving project metadata (Milestones, Priorities, etc.). Some of them may be repeated.
You did not state the version of Klaros you are using, this is of importance here.
We are leveraging a third party software to connect to Gitlab. Before version 4.12.5, this implementation connected to all projects on the GitLab server, we patched that behavior in 4.12.5, so only the configured project will be retrieved. I am suspecting that this may be the case here.
So if you are using a version below 4.12.5 please update and retry.
On the downside we also had to remove connection caching in 4.12.5 as it was interfering with the ability to select an assignee during issue creation. This may as well have a detrimental influence here, but I wouldn't expect this in this magnitude.
If you are experienced with using Wireshark or a logging http/https proxy I would be interested to see what is going on at comunication with your Gitlab server.
Regards,
Torsten
Â
You did not state the version of Klaros you are using, this is of importance here.
We are leveraging a third party software to connect to Gitlab. Before version 4.12.5, this implementation connected to all projects on the GitLab server, we patched that behavior in 4.12.5, so only the configured project will be retrieved. I am suspecting that this may be the case here.
So if you are using a version below 4.12.5 please update and retry.
On the downside we also had to remove connection caching in 4.12.5 as it was interfering with the ability to select an assignee during issue creation. This may as well have a detrimental influence here, but I wouldn't expect this in this magnitude.
If you are experienced with using Wireshark or a logging http/https proxy I would be interested to see what is going on at comunication with your Gitlab server.
Regards,
Torsten
Â
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
I'm using the latest Klaros Version 4.12.5. running in a docker container. And also gitlab is running in a docker container.
Currently I've no option for wireshark or something like that. Because of the dockerized structure.Â
But I will try to get the HTTP communication, so that we can see what happens.
Currently I've no option for wireshark or something like that. Because of the dockerized structure.Â
But I will try to get the HTTP communication, so that we can see what happens.
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
I've installed Klaros 4.12.5 on my local machine and captured the network traffic with Wireshark. Sadly I'm using https, so I can't see the complete request but I found some interesing packages:
First I got a request to gitlab with a Hello packets and some data, than after some seconds I got an reset signal:
Then the request repeats with the same Data and length. So there are again packets with data (same as above).Â
This looks a littlebit strange for me.... Is there a way to log the messages from Klaros? Maybe with another log level in any property file?
First I got a request to gitlab with a Hello packets and some data, than after some seconds I got an reset signal:
No. Time Protocol Length Info | |
89 4.652329 TLSv1.2 647 Client Hello | |
93 5.174111 TLSv1.3 140 Application Data | |
94 5.174495 TLSv1.3 379 Application Data | |
139 8.234023 TCP 54 60523 → 443 [RST] Seq=1049 Win=0 Len=0 |
Then the request repeats with the same Data and length. So there are again packets with data (same as above).Â
No. Time Protocol Length Info | |
270 11.264539 TLSv1.3 647 Client Hello TCP 54 | |
274 11.301455 TLSv1.3 140 Application Data | |
275 11.301778 TLSv1.3 379 Application Data | |
339 14.119048 TCP 54 60556 → 443 [RST] Seq=1049 Win=0 Len=0 |
This looks a littlebit strange for me.... Is there a way to log the messages from Klaros? Maybe with another log level in any property file?
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
Let me have a look ...
The file to edit is: webapps/klaros-web/WEB-INF/classes/logback.xml
We are using logback with slf4j. jul-to-slf4j is already present. The GitLab API uses the standard Java HttpURLConnection class.
You will probably have to change the default log level and add the correct log level settings.
I was not yet able to try this out.
The file to edit is: webapps/klaros-web/WEB-INF/classes/logback.xml
We are using logback with slf4j. jul-to-slf4j is already present. The GitLab API uses the standard Java HttpURLConnection class.
You will probably have to change the default log level and add the correct log level settings.
I was not yet able to try this out.
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
I've now logged the following action:
Open a Testcase, hit "Issue" and select "new". (execution time untill the UI opens: ca. 30 seconds)
Then I got the following output: (see attachment)
You can see that there are 5 Messages which are repeated 5 times.
One round tooks 3 seconds. So if this is repeated 5 times with some delay between the requests, then we got this 20 till 30 seconds.
But what can we do now?? I'm totally fine with the 3 seconds for one round... but it is not so nice that this occurs 5 times.
Open a Testcase, hit "Issue" and select "new". (execution time untill the UI opens: ca. 30 seconds)
Then I got the following output: (see attachment)
You can see that there are 5 Messages which are repeated 5 times.
One round tooks 3 seconds. So if this is repeated 5 times with some delay between the requests, then we got this 20 till 30 seconds.
But what can we do now?? I'm totally fine with the 3 seconds for one round... but it is not so nice that this occurs 5 times.
Attachments:
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
Thank you for the analysis, this is interesting.
First of all, I honestly think that 3 seconds for a single request is unacceptable in any case.
I am unsure if this is based in our changes regarding connection pooling applied, so I added a modified version of the connector to this post. Please replace this file (located in webapps/klaros-web/WEB-INF/lib) restart the server and retry.
If this still does not significantly better your situation then I strongly suspect a cause outside of Klaros.
Sadly, this will reintroduce issue KLAROS-4298 (Unable to select an assignee when creating/editing GitLab issues) so this is not a permanent solution.
Please let me know your results.
First of all, I honestly think that 3 seconds for a single request is unacceptable in any case.
I am unsure if this is based in our changes regarding connection pooling applied, so I added a modified version of the connector to this post. Please replace this file (located in webapps/klaros-web/WEB-INF/lib) restart the server and retry.
If this still does not significantly better your situation then I strongly suspect a cause outside of Klaros.
Sadly, this will reintroduce issue KLAROS-4298 (Unable to select an assignee when creating/editing GitLab issues) so this is not a permanent solution.
Please let me know your results.
Attachments:
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
Hello again,
you are right. 3 seconds is to long to wait. but it is better than 30Â
Thank you for that modified version... but it is still not working. Now I got another issue:
The Authentication is wrong.
I looked into the logs and there is a strange behavior to check if the gitlab server is available.
1. A request is send to:Â /api/v4/projects/182/issues/2147483647 with an API Token (which is valid).
This is ok, but I'm confused about the issue-id 2147483647. This issue doesn't exists so the server get an 404 not found.
2. Then a second request is send to:Â /api/v4/user which looks good, BUT there is NO ApiToken in the request. So the Server got an 401Â Unauthorized
More requests are not possible because of an exception, that the username/token is invalid, which is not the case.
you are right. 3 seconds is to long to wait. but it is better than 30Â
Thank you for that modified version... but it is still not working. Now I got another issue:
The Authentication is wrong.
I looked into the logs and there is a strange behavior to check if the gitlab server is available.
1. A request is send to:Â /api/v4/projects/182/issues/2147483647 with an API Token (which is valid).
This is ok, but I'm confused about the issue-id 2147483647. This issue doesn't exists so the server get an 404 not found.
2. Then a second request is send to:Â /api/v4/user which looks good, BUT there is NO ApiToken in the request. So the Server got an 401Â Unauthorized
More requests are not possible because of an exception, that the username/token is invalid, which is not the case.
Attachments:
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
Thanks for the detailed report.
The  /api/v4/projects/182/issues/2147483647 request is a probe for validating the connection parameters.
It should only be done once per user session to a Gitlab project and is intentional.
I am unsure about the error message regarding the authentication error you receive, I am going to validate this and get back to you.
The  /api/v4/projects/182/issues/2147483647 request is a probe for validating the connection parameters.
It should only be done once per user session to a Gitlab project and is intentional.
I am unsure about the error message regarding the authentication error you receive, I am going to validate this and get back to you.
Christopher Huber, modified 4 Years ago.
RE: Slow Gitlab Integration connection
Youngling Posts: 9 Join Date: 2/26/20 Recent Posts
Did you found anything related this topic?
or was you able to reproduce the issue?
or was you able to reproduce the issue?
Torsten Stolpmann, modified 4 Years ago.
RE: Slow Gitlab Integration connection (Answer)
Jedi Council Member Posts: 759 Join Date: 2/12/09 Recent Posts
Hello Christopher,
our automated tests are running a login, issue creation and retrieval in GitLab in roughly 2.5 sec (with GitLab running in a virtualized environment on crap hardware). I am unsure on what we can do on our side to remedy your situation.
Can you rule out SSL-negotiation problems with your GitLab installation? If it is exposed to the internet https://www.ssllabs.com/ssltest/analyze.html might help to diagnose.
We will keep an eye on this, but I am missing the right idea here.
Torsten
P.S.: I am travelling this week, so do not expect fast responses.
our automated tests are running a login, issue creation and retrieval in GitLab in roughly 2.5 sec (with GitLab running in a virtualized environment on crap hardware). I am unsure on what we can do on our side to remedy your situation.
Can you rule out SSL-negotiation problems with your GitLab installation? If it is exposed to the internet https://www.ssllabs.com/ssltest/analyze.html might help to diagnose.
We will keep an eye on this, but I am missing the right idea here.
Torsten
P.S.: I am travelling this week, so do not expect fast responses.