RE: Feature request for using Klaros with Hudson - Klaros Test Management - Forum - Klaros Test Management
Hi Gaetan,
please find our comments inline, your feedback is of course welcome:
I agree that this should be the job of the Hudson plug-in. Currently the implementation works as follows:
For every test result file matching the file name pattern specified in the Hudson plugin a new test run result is created, containing the individual test results. Currently you cannot browse these in the UI but at least we store them with the correct associations.
I think this is not the optimal solution in your case, since the relation between the individual test run results is not persisted, so we need to store at least e.g. the build id of the Hudson build as well.
This is something we like to implement in the future, but not in the upcoming release 2.6 (KLAROS-701 - Allow the generation of accumulated test run reports).
One the Single Test Run Report view, I see a list of all test run sorted chronologically. I can also see how many test have failed for each run.. That's very good. I'd like to have more information about all the test run for a given line. I can only generate PDF, XML HTML and co of the report, but not list easilly all tests for a given Test Run Report.
Request 2 : have a link on the "TRU0000001" (for example) listing all test run on this Test Run.
This is definitly missing and something we like to implement in the upcoming release 2.6 (KLAROS-700 - Allow the online browsing of test runs and their results).
Ah - the nesting of test suites - this as well comes up in our discussions every now and then
.
Let me make a short digression on our view on test plans:
We see test plans as well defined, long lived lists of revisionable test cases which are grouped together for a certain purpose. On the other hand, this is not what your average JUnit tests would be like.
The most import difference is, that the Klaros can not control what tests are executed by Hudson, so any plan definition would not make sense here anyway.
I am more inclined to allow a multi-level hierarchical categorization of test cases, giving more of a logical, structured view on the pool of defined test cases. I filed a feature request for a release in 2.7 or later for this, as I think we need to do some more impact investigation on this before implementation (KLAROS-703 - Allow logical categorization of test cases).
My automatically generated tests have ugly names like "My_Test_Suite./path/to/hudson/build/path/to/unittest/applications/test/myTestsuite.Friendly name".
Request 4 : Is it possible to add a "friendly name" field to rename the test when it is displayed in the reports? Of course, the idea behind that is to keep the ability for the test case to be feeded by the hudson plugin (using this "ugly" name as key) and display the nice name in the reports.
Are you using Hudson slaves? I am a bit suprised about the names you are getting here (should be: myTestsuite.Friendly name).
The names given by default are directly taken from the JUnit result files. To make them non-ambiguous we need to concatenate the package name as well, I agree that this is not the most readable solution.
I see your point, but I am unsure if adding another display name really is the best solution here. I think that we have to track external ids of test cases like myTestsuite.Friendly separatly anyway, so this would free the name field again to have 'nicer' names. Expect some improvements here in the future.
I agree completly, this can and must be improved. Please note that the result icon already shows the result description in its tooltip. I filed a feature request scheduled for the upcoming 2.6 release for this (KLAROS-702 - Show more details of a test result).
Our pleasure, thanks for sharing your thoughts this has been a really helpful post. We are looking forward to any further ideas.
Best Regards,
Torsten
please find our comments inline, your feedback is of course welcome:
Gaetan SEMET:
Hello.
I'm starting using Klaros and it seems a very promise product, well built and you seem to be very reactive at the support. I have a couple of special request on this product, since I don't know it very well maybe these feature are present and I don't know how to use it.
I give you a typical use case that may be quite generic, so could be really useful for everybody.
My setup is quite simple, I have a bunch or automatic tests generating JUnit xml files, sent to klaros with the Hudson plugin.
Request 1 : I have several XML files generated after each run. How could I concatenate them into one since Test Run Report? Should I contatenate the XML files themselve and have hudson send only one file to Klaros? This may be the job of the hudson to klaros plugin.
I'm starting using Klaros and it seems a very promise product, well built and you seem to be very reactive at the support. I have a couple of special request on this product, since I don't know it very well maybe these feature are present and I don't know how to use it.
I give you a typical use case that may be quite generic, so could be really useful for everybody.
My setup is quite simple, I have a bunch or automatic tests generating JUnit xml files, sent to klaros with the Hudson plugin.
Request 1 : I have several XML files generated after each run. How could I concatenate them into one since Test Run Report? Should I contatenate the XML files themselve and have hudson send only one file to Klaros? This may be the job of the hudson to klaros plugin.
I agree that this should be the job of the Hudson plug-in. Currently the implementation works as follows:
For every test result file matching the file name pattern specified in the Hudson plugin a new test run result is created, containing the individual test results. Currently you cannot browse these in the UI but at least we store them with the correct associations.
I think this is not the optimal solution in your case, since the relation between the individual test run results is not persisted, so we need to store at least e.g. the build id of the Hudson build as well.
This is something we like to implement in the future, but not in the upcoming release 2.6 (KLAROS-701 - Allow the generation of accumulated test run reports).
One the Single Test Run Report view, I see a list of all test run sorted chronologically. I can also see how many test have failed for each run.. That's very good. I'd like to have more information about all the test run for a given line. I can only generate PDF, XML HTML and co of the report, but not list easilly all tests for a given Test Run Report.
Request 2 : have a link on the "TRU0000001" (for example) listing all test run on this Test Run.
This is definitly missing and something we like to implement in the upcoming release 2.6 (KLAROS-700 - Allow the online browsing of test runs and their results).
Ideally, I have a matrix of all modules (see next request !) with the same three colunm view ("pass/fail/error"). When I click on each module, and submodules, I arrive at the end to the list of all test run for this module with their results (same as in the Test Case Result, but with the result (passed/failed/error)).
The Test Case results can display the result of all my test, but I'll have hundreds of tests automatically generated. Using a flat list of all test result is not possible.
Request 3 : Is it possible to use the name of the test to group them into modules/submodules? Or give the ability to define into our own modules/grouping (new tests are not added in a specific group, and we can move them from one group to another one easilly). Test plan may fit this request, but they are not organised by tree. For example, I wish I could define a hierarchy with "Codec/Quality tests/Mpeg2 Encoding/". Maybe we could think test plan as a list of test, and organise test plan in test suites. Test suites may themselve be organised in meta test suites, and meta-meta test suites,...
The Test Case results can display the result of all my test, but I'll have hundreds of tests automatically generated. Using a flat list of all test result is not possible.
Request 3 : Is it possible to use the name of the test to group them into modules/submodules? Or give the ability to define into our own modules/grouping (new tests are not added in a specific group, and we can move them from one group to another one easilly). Test plan may fit this request, but they are not organised by tree. For example, I wish I could define a hierarchy with "Codec/Quality tests/Mpeg2 Encoding/". Maybe we could think test plan as a list of test, and organise test plan in test suites. Test suites may themselve be organised in meta test suites, and meta-meta test suites,...
Ah - the nesting of test suites - this as well comes up in our discussions every now and then
.Let me make a short digression on our view on test plans:
We see test plans as well defined, long lived lists of revisionable test cases which are grouped together for a certain purpose. On the other hand, this is not what your average JUnit tests would be like.
The most import difference is, that the Klaros can not control what tests are executed by Hudson, so any plan definition would not make sense here anyway.
I am more inclined to allow a multi-level hierarchical categorization of test cases, giving more of a logical, structured view on the pool of defined test cases. I filed a feature request for a release in 2.7 or later for this, as I think we need to do some more impact investigation on this before implementation (KLAROS-703 - Allow logical categorization of test cases).
My automatically generated tests have ugly names like "My_Test_Suite./path/to/hudson/build/path/to/unittest/applications/test/myTestsuite.Friendly name".
Request 4 : Is it possible to add a "friendly name" field to rename the test when it is displayed in the reports? Of course, the idea behind that is to keep the ability for the test case to be feeded by the hudson plugin (using this "ugly" name as key) and display the nice name in the reports.
Are you using Hudson slaves? I am a bit suprised about the names you are getting here (should be: myTestsuite.Friendly name).
The names given by default are directly taken from the JUnit result files. To make them non-ambiguous we need to concatenate the package name as well, I agree that this is not the most readable solution.
I see your point, but I am unsure if adding another display name really is the best solution here. I think that we have to track external ids of test cases like myTestsuite.Friendly separatly anyway, so this would free the name field again to have 'nicer' names. Expect some improvements here in the future.
I am in the Test Case Result page. I see my test that have failed, let's say TCR0000001. How could I have more information about the result of this execution (why it has failed, the log output,...)
Request 5 : a link in the Test case result page, around "TCR0000001" for example, or in an "Action" icon, give the ability to display a page with more information about the execution for this very test. For instance, displaying the Junit raw xml file, the log output of the program, generated pictures,...
Request 5 : a link in the Test case result page, around "TCR0000001" for example, or in an "Action" icon, give the ability to display a page with more information about the execution for this very test. For instance, displaying the Junit raw xml file, the log output of the program, generated pictures,...
I agree completly, this can and must be improved. Please note that the result icon already shows the result description in its tooltip. I filed a feature request scheduled for the upcoming 2.6 release for this (KLAROS-702 - Show more details of a test result).
I really hope this could help you improving your product.
Best Regards
Gaetan
Best Regards
Gaetan
Our pleasure, thanks for sharing your thoughts this has been a really helpful post. We are looking forward to any further ideas.
Best Regards,
Torsten
Thank you so much of taking time to study my comment.
Thank of taking it into account.
This is definitly missing and something we like to implement in the upcoming release 2.6 (KLAROS-700 - Allow the online browsing of test runs and their results).
That would be so useful, enabling online analysis of the test results.
Ah - the nesting of test suites - this as well comes up in our discussions every now and then .
Let me make a short digression on our view on test plans:
We see test plans as well defined, long lived lists of revisionable test cases which are grouped together for a certain purpose. On the other hand, this is not what your average JUnit tests would be like.
The most import difference is, that the Klaros can not control what tests are executed by Hudson, so any plan definition would not make sense here anyway.
I am more inclined to allow a multi-level hierarchical categorization of test cases, giving more of a logical, structured view on the pool of defined test cases. I filed a feature request for a release in 2.7 or later for this, as I think we need to do some more impact investigation on this before implementation (KLAROS-703 - Allow logical categorization of test cases).
I do agree that multi-level of categorization of test cases would be the best solution. I don't know it would be difficult to reflect the result of one automatic test run into the test plan, but to my humble opinion, this would be barely maintainable throught all life of the product (automatic test change often,...)
My automatically generated tests have ugly names like "My_Test_Suite./path/to/hudson/build/path/to/unittest/applications/test/myTestsuite.Friendly name".
Request 4 : Is it possible to add a "friendly name" field to rename the test when it is displayed in the reports? Of course, the idea behind that is to keep the ability for the test case to be feeded by the hudson plugin (using this "ugly" name as key) and display the nice name in the reports.
Are you using Hudson slaves? I am a bit suprised about the names you are getting here (should be: myTestsuite.Friendly name).
Actually I'm using C++ Boost.Test library that generates very clean XML file:
but they are translated into junit by a hudson plugin that put some ugly names:
Personnaly, I see the my as a hierarchy of test suite. I much prefere the boost.test log (many level of nested TestSuite as we want) that the Junit format (wich seem to only want one test suit by xml file).
I see your point, but I am unsure if adding another display name really is the best solution here. I think that we have to track external ids of test cases like myTestsuite.Friendly separatly anyway, so this would free the name field again to have 'nicer' names. Expect some improvements here in the future.
ID means unit tests should be tracked over time, and this ID would be well defined (for instance by setting the ID matching the Klaros Test Case ID directly in the junit file. That would be quite difficult to track) I think relying on the name would be enought, given that it doesn't change. That why a "friendly" name would be quite useful.
I agree completly, this can and must be improved. Please note that the result icon already shows the result description in its tooltip. I filed a feature request scheduled for the upcoming 2.6 release for this (KLAROS-702 - Show more details of a test result).
Torsten
Thanks for you quick feedback.
Torsten Stolpmann:
Hi Gaetan,
This is something we like to implement in the future, but not in the upcoming release 2.6 (KLAROS-701 - Allow the generation of accumulated test run reports).
This is something we like to implement in the future, but not in the upcoming release 2.6 (KLAROS-701 - Allow the generation of accumulated test run reports).
Thank of taking it into account.
This is definitly missing and something we like to implement in the upcoming release 2.6 (KLAROS-700 - Allow the online browsing of test runs and their results).
That would be so useful, enabling online analysis of the test results.
Ah - the nesting of test suites - this as well comes up in our discussions every now and then
.Let me make a short digression on our view on test plans:
We see test plans as well defined, long lived lists of revisionable test cases which are grouped together for a certain purpose. On the other hand, this is not what your average JUnit tests would be like.
The most import difference is, that the Klaros can not control what tests are executed by Hudson, so any plan definition would not make sense here anyway.
I am more inclined to allow a multi-level hierarchical categorization of test cases, giving more of a logical, structured view on the pool of defined test cases. I filed a feature request for a release in 2.7 or later for this, as I think we need to do some more impact investigation on this before implementation (KLAROS-703 - Allow logical categorization of test cases).
I do agree that multi-level of categorization of test cases would be the best solution. I don't know it would be difficult to reflect the result of one automatic test run into the test plan, but to my humble opinion, this would be barely maintainable throught all life of the product (automatic test change often,...)
My automatically generated tests have ugly names like "My_Test_Suite./path/to/hudson/build/path/to/unittest/applications/test/myTestsuite.Friendly name".
Request 4 : Is it possible to add a "friendly name" field to rename the test when it is displayed in the reports? Of course, the idea behind that is to keep the ability for the test case to be feeded by the hudson plugin (using this "ugly" name as key) and display the nice name in the reports.
Are you using Hudson slaves? I am a bit suprised about the names you are getting here (should be: myTestsuite.Friendly name).
Actually I'm using C++ Boost.Test library that generates very clean XML file:
<TestLog> | |
<TestSuite name="My Suite"> | |
<TestSuite name="Nested test suite"> | |
<TestCase name="First test case"> | |
<Info file="/home/hudson/build/trunk/file.cpp" line="577">check result == true passed</Info> | |
... | |
</TestCase> | |
</TestSuite> | |
</TestSuite> | |
</TestLog> |
but they are translated into junit by a hudson plugin that put some ugly names:
<?xml version="1.0" encoding="UTF-8"?> | |
<testsuite tests="10" errors="1" failures="8" name="Nested test suite" skipped="0"> | |
<testcase classname="Nested test suite." name="First test case" time="2.76"> | |
<system-out/> | |
<system-err> | |
=== [Exception] - ::signal: SIGALRM (timeout while executing function)</system-err> | |
</testcase> | |
</testsuite> |
Personnaly, I see the my as a hierarchy of test suite. I much prefere the boost.test log (many level of nested TestSuite as we want) that the Junit format (wich seem to only want one test suit by xml file).
I see your point, but I am unsure if adding another display name really is the best solution here. I think that we have to track external ids of test cases like myTestsuite.Friendly separatly anyway, so this would free the name field again to have 'nicer' names. Expect some improvements here in the future.
ID means unit tests should be tracked over time, and this ID would be well defined (for instance by setting the ID matching the Klaros Test Case ID directly in the junit file. That would be quite difficult to track) I think relying on the name would be enought, given that it doesn't change. That why a "friendly" name would be quite useful.
I agree completly, this can and must be improved. Please note that the result icon already shows the result description in its tooltip. I filed a feature request scheduled for the upcoming 2.6 release for this (KLAROS-702 - Show more details of a test result).
Torsten
Thanks for you quick feedback.