2
0
mirror of https://github.com/xcat2/xcat-core.git synced 2024-11-21 09:02:00 +00:00
14 Wish_List_for_xCAT_FVT
Weihua Hu edited this page 2017-11-17 14:36:39 +08:00
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Stabilize the automation environment

  • Resolve unstable problem of VMs in automation environment. (one possible solution is using RHEV). (p1) DONE
  • Modify unstable test cases. (such like nodeset_cmdline and so on). (p1) DONE
  • Make sure the same build run 7 times with the same regression result. (redhat+ppc64le has the highest priority, then SLES, ubuntu) (p1) Almost done, now the timing issue focus on xCAT product itself
  • Make sure all automation hardware are in good condition and can be monitored. DONE
    • Check the status (CPU/memory/disk space) of hypervisor by script. (p1)

Increase test cases

  • Work out a rule about how to write a test case, such like must clean up environment and how to handle multiple platforms. (p1) DONE
  • Add test case depending on customers feedback. (p1) in progress
    • Go through github or mail list, add cases for the issues have been resolved.
  • Add test case for new feature. (p1) in progress
    • How to plan test case set.
    • Work out a rule to measure the case automated rate and pass rate.
  • Summarize the number of new case and its pass rate in the end of each sprint. (p1) in progress
  • How to manage the manual test cases. (one option is github). (p2) DONE
  • How to involve developer to contribute test cases. (p2) in progress
  • Go through existed cases and refine (p3)
    • Offer a ideal test scope by developer
      • Command line : *def , hardware control command, nodeset, makedns , makehosts, makedhcp, updatenode, xdsh , xdcp, pping, ppping, prsync…
      • OS provision process : copycds, rinstall, postscript, postbootscript, confignics, confignetwork, genimage,packimage
      • DB operation : tab*, DB backup/restore.
      • xCAT installation : go_xcat, xcatconfig, xcat deamon control, lsxcatd, migration
      • Hardware discovery : mknb, bmcsetup,
      • REST api
      • KIT
    • Analyse how many existed cases are valid.
    • Compare the valid cases and the ideal test scope to work out how many cases need to add.

Enhance release process

  • Figure out the coverage of release cycle.
    • Daily regression DONE
    • Manual test : against specific hardware, os provision against physical machine, discovery. (p1)
    • Release process DONE
  • Work out a check list for 3.1, including acceptance criteria. (p1) DONE
  • Automated the 3.1.3 release process (p3)

Enhance automation framework

  • Work out a step by step to debug problem in automation environment. (p1) DONE
  • How to verify a new test case in non-production environment. (p1) DONE
  • Reduce the running time of regression project. (p1)
  • Make it easier to prepare a xcattest configuration file (p1) DONE
    • Make it easier to find out the error caused by xcattest configuration file or new build in automation environment.
  • Make it easier for developer to do unit test (p1) in progress
  • Using Jenkins API to add /delete/switch projects. (p1) DONE
  • Refine test report to including more useful information to debug. (p2)
    • To highlight the people who checked in code during last day. DONE
  • How to maintain and categorize bundle files. Make it easier to add new test case and to invoke bundles. (p2)
    • Using “#INCLUDE” label. DONE
    • Support bundle priority.
    • Manage the case not existed in bundle.
    • Generate bundle for specific purpose. Such like for specific hardware device or for specific business project. in progress
  • Enhance automation infrastructure to support more test requirement, including hardware and network. (p2)
    • To support discovery test.
    • To support specific hardware test.
  • Make it easier to add/delete server into/from automation platform (p2) in progress
  • Make it easier to reset up a new automation platform (p2) in progress
  • Find the root cause of provision failure quickly (p3)
  • Report auto test result to slack quickly. (p3)
  • Learning hot automation tools (p3)

Leverage CI to do quick verification for pull request (p3)

  • To verify if a pull request follow expected process. DONE
  • To check syntax DONE
  • To do build DONE
  • To check xcat basic function, such like checking xcatd daemon, xcat DB and so on. DONE
  • Set up a build service to build xCAT package depending on specific requirement, such like for specific repo.