{"id":4486,"date":"2018-10-23T10:25:15","date_gmt":"2018-10-23T08:25:15","guid":{"rendered":"https:\/\/zen-cori.138-201-132-86.plesk.page\/?p=4486"},"modified":"2022-09-28T16:32:06","modified_gmt":"2022-09-28T07:32:06","slug":"continuous-delivery-3-best-practices-to-get-maximum-benefit-from-automated-testing","status":"publish","type":"post","link":"https:\/\/www.btc-embedded.jp\/ja\/continuous-delivery-3-best-practices-to-get-maximum-benefit-from-automated-testing\/","title":{"rendered":"Continuous Delivery: 3 best practices to get maximum benefit from automated testing"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"4486\" class=\"elementor elementor-4486\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-3b5f0493 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"3b5f0493\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-559a247a\" data-id=\"559a247a\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-d83c847 elementor-widget elementor-widget-text-editor\" data-id=\"d83c847\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.20.0 - 26-03-2024 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p>Agile Development, Continuous Integration (CI) and Continuous Delivery (CD) have become more than just buzzwords. They are key success factors in software industry. The idea is, to have a software pipeline covering all stages from concept to deployment while ensuring repeatability and reliability in order to enable frequent releases with constant feedback. This concept is a perfect fit for the increasing demands on cost and quality in automotive software development.<\/p><p>There is a catch of course. When I first got in contact with software development in the automotive domain I was frustrated to see how slow things were moving. I really needed some time getting used to the fact that the diverse landscape of tools and processes and the extensive dependencies to hardware make the transition to CD extremely complex in automotive (compared to other domains like app or web development). Additionally, developers in automotive often got the feeling that testing was slowing them down.<\/p><p>The following three aspects explain best practices for efficiently integrating your testing activities into an automated development process while connecting modern CD principles with automotive and model-based development.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7c099a8 elementor-widget elementor-widget-heading\" data-id=\"7c099a8\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.20.0 - 26-03-2024 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">1. Define automatic verdicts for your tests<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-45e1cbc elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"45e1cbc\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-a4a77d0\" data-id=\"a4a77d0\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-af21f71 elementor-widget elementor-widget-text-editor\" data-id=\"af21f71\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>I still see a lot of cases where a test is performed like this:<\/p><ol><li>Specify some inputs for the SUT (System Under Test)<\/li><li>Simulate the SUT with these inputs<\/li><li>Inspect the outputs<\/li><li>Manually decide on passed or failed<\/li><\/ol>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-7307992\" data-id=\"7307992\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-d12e251 elementor-widget elementor-widget-image\" data-id=\"d12e251\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.20.0 - 26-03-2024 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"250\" height=\"174\" src=\"https:\/\/www.btc-embedded.jp\/wp-content\/uploads\/2018\/10\/automatic-verdicts-250x174-9e2.png\" class=\"attachment-large size-large wp-image-4491\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-a1ff07f elementor-widget elementor-widget-text-editor\" data-id=\"a1ff07f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Usually this process is done because of missing requirements or a combination of extremely low-level requirements for algorithms which realize complex mathematical calculations. Especially Model-based Development with Matlab\/Simulink makes it tempting to implement a rather manual and interactive test workflow. The problem with the approach described above is that it requires manual effort to choose a verdict (passed\/failed). Like this, with increasingly frequent iterations the manual effort increases as well. In addition, the risk of human error is very high with the developer having to repeat the same task over and over (with the pressure of the project deadline). In general, this approach is not transparent and does just not scale at all, thereby counteracting the benefits you want to achieve from your CI\/CD process.<\/p><p>How to solve this? In addition to the inputs that you specify, your Requirements-based test cases should always contain information about the desired output behavior of the SUT coming from the requirements, either in the form of expected values (usually considering tolerances), formulas or even scripts which analyze the simulated output. From this point on, your requirements-based test cases are very low-maintenance. In general, they only have to be adapted if the related requirements change or if structural changes of the SUT impact the interface. Having an automatic verdict for all of your tests is vital for efficiency and highly increases the transparency while eliminating the chances for human error. In addition, automatic verdicts ensure scalability with iterations becoming more and more frequent.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27e953f elementor-widget elementor-widget-heading\" data-id=\"27e953f\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">2. Failed Tests should impact your process<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6cdaf06 elementor-widget elementor-widget-text-editor\" data-id=\"6cdaf06\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>\u00a0What becomes apparent is that the idea of a failed test is associated with bad feelings. Often the only reaction to failed test cases is that the failure appears in a report. Downstream steps sometimes continue as if nothing happened because of the pressure of producing a deliverable piece of software:<\/p><p><i>\u201cWe can still judge if the failed test case really is an issue later on.\u201d<\/i><\/p><p>This approach which is motivated by the belief that delivering\u00a0<i>something<\/i>\u00a0is better than delivering\u00a0<i>nothing<\/i>\u00a0ignores a few facts, especially regarding unit tests:<\/p><ul><li>The bulk of the resource and execution time intensive part comes after the unit tests (Hardware-in-the-Loop, Prototypes, etc.)<\/li><li>A fix can reveal other defects or introduce regressions.<\/li><li>The sooner you fix an issue the cheaper it is to do so. Since your unit tests should finish very soon after introducing a change, applying a fix immediately does not cost much time.<\/li><li>Debugging is easier for a small number of issues. Waiting until a lot goes wrong will make it extremely hard to isolate individual issues.<br \/><br \/><\/li><\/ul><p>With the\u00a0<i><a href=\"https:\/\/devopsnet.com\/2011\/08\/04\/continuous-delivery\/\" target=\"_blank\" rel=\"noopener\">Fail Fast Principle<\/a><\/i>\u00a0in mind, developers, testers, team leaders and managers should aim for a paradigm shift: Test failures are good and they should stop your pipeline.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-fdfe927 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"fdfe927\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-373966a\" data-id=\"373966a\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-0dcfaf3 elementor-widget elementor-widget-text-editor\" data-id=\"0dcfaf3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Regardless of standards which require certain test activities, you should embrace not only testing itself but also failed tests. Why? Because they prove that your quality assurance process works. Try to convince an auditor that your process is solid if you cannot show him any failed tests \u2013 not that easy, right? Furthermore, each run that was blocked by failed tests saves valuable resources and time which would otherwise be wasted and prevents costly and highly damaging recalls.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-5e6f192\" data-id=\"5e6f192\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-ddf21f5 elementor-widget elementor-widget-image\" data-id=\"ddf21f5\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"689\" height=\"212\" src=\"https:\/\/www.btc-embedded.jp\/wp-content\/uploads\/2018\/10\/pipeline-1920x1920-cd9.png\" class=\"attachment-large size-large wp-image-4495\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-6ec190f elementor-widget elementor-widget-text-editor\" data-id=\"6ec190f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>For those of you who still struggle with the thought of stopping a precious pipeline run: Don\u2019t worry. Stopping the pipeline does not mean giving up. See the next section for how to deal with failures.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-da6f93a elementor-widget elementor-widget-heading\" data-id=\"da6f93a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">3. Have a next-step plan for failures<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fa217fc elementor-widget elementor-widget-text-editor\" data-id=\"fa217fc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Often software development pipelines (not only automotive) are optimized for the\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Happy_path\" target=\"_blank\" rel=\"noopener\">happy path<\/a>. To show that the pipeline manages to transfer a change into a new deliverable of the software is a natural first step when setting up an automated process. If something fails along the line it is treated as an exception.<\/p><p>However, as we have established above, test failures are to be expected. The definition of a successfully implemented software delivery pipeline is not that it produces a deliverable for every change but that it produces a deliverable for every change that passed all tests. Since failed tests should block the pipeline, an important aspect towards efficiency is to prepare everything that developers and testers need to know in order to react on failures. This includes:<\/p><ul><li>Links to requirements<\/li><li>Reports, Logs, Stack Traces<\/li><li>Data for debugging (SUT + test cases)<\/li><\/ul><p><br \/>Having the next steps prepared enables your teams to\u00a0<i><a href=\"https:\/\/martinfowler.com\/articles\/continuousIntegration.html#FixBrokenBuildsImmediately\" target=\"_blank\" rel=\"noopener\">fix broken builds immediately<\/a><\/i>\u00a0which is a vital part of the CD concept. Developers should not add new functionality before the existing features meet the expected quality.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3810d50 elementor-widget elementor-widget-heading\" data-id=\"3810d50\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d8961b6 elementor-widget elementor-widget-text-editor\" data-id=\"d8961b6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The idea of testing activities \u2013 especially unit testing \u2013 must shift from being a hassle to being a valuable resource. In order to achieve this, you should aim for the highest possible degree of automation which includes automated passed-failed criteria of your tests and the automated preparation of analysis- and debugging environments. With these things handled there is no reason to fear a failed test: your pipeline will stop, saving valuable resources and at the same time providing all you need to identify the cause of the issue.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Agile Development, Continuous Integration (CI) and Continuous Delivery (CD) have become more than just buzzwor [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"elementor_theme","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[55,56,54],"product":[],"use_cases":[],"class_list":["post-4486","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-ci-cd","tag-cloud","tag-jenkins"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/posts\/4486","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/comments?post=4486"}],"version-history":[{"count":12,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/posts\/4486\/revisions"}],"predecessor-version":[{"id":12549,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/posts\/4486\/revisions\/12549"}],"wp:attachment":[{"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/media?parent=4486"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/categories?post=4486"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/tags?post=4486"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/product?post=4486"},{"taxonomy":"use_cases","embeddable":true,"href":"https:\/\/www.btc-embedded.jp\/ja\/wp-json\/wp\/v2\/use_cases?post=4486"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}