When your action makes a change to object attributes, the prior values should be preserved, so that a new action could "undo" the change. You could either stack the prior values, or allow the undo to reference a specific change.
If test coverage for default driver config can be developed, consider a mode of storing the Validator config in an eDir object (DirXML-Resource class, with DirXML-ContentType=mime/Validator-test-suite) so that when you import drivers from Vault in Validator, you also get a default test suite. Would allow developers to ship drivers with test suites included.
An On Failure action could potentially be useful.
For example: Run Cleanup.
Echo a message.
Maybe even start a workflow to go do some cleanup/notification/whatever
For each test step in the report, include the description. This is what is seen when you collapse the test steps.
We create users for testing different things. At one point we were getting an error on the creation so we had to back out the attributes one by one and then re-add. It would be a huge help to be able to just disable them. Same with if you are "Asserting attributes and Values in eDirectory" if you have several items here you can enable/disables items.
Store passwords in a way that:
Ties the serialized value to the local Validator install
Allows tests to be imported with "clear text passwords" that are then stored securely.
Separates the password storage from the test definitions.
Perhaps reuse the "named password" features in IDM to implement the storage?
Instead of literal strings we use GCVs to deal with environment specific values in policy. Being able to import existing driver/driverset GCVs as variables would eliminate the step to manually redo them in validator.
Put the Description of the test into the results tab. Looking at many very similar tests having the Description can make it so much easier.
And on a failed test, particularly one where values are compared. Show the value found. This is a standard feature of Junit where it shows both expected and actual values. Right now you show only expected value in the report.
We use git for version control on our test cases. But Validator makes this a bit cumbersome because it stores the actual test case logic combined with state information in one file. Bot layout state (if a group is expanded or not) and variable values should be stored separately.
Since many workflows are started automatically, it would be helpful In the UserApp connector if there was a way to get a list of workflow approvals pending for a particular user. This would allow us to know if the workflow was automatically started correctly.
Implement LDAP query function and store the query result in a multi-valued variable.
- Allow the stored format to be JSON, delimited text, xml.
- Allow max number of results to be variable (1...1000 or more)
Allow looping over values in a variable.
Add a parameter to select a specific test in a suite file to run from command line.
I would like to create a suite with two tests, one basic for Swift verification and one for mor thorough tests.
Then I want to run these from command line.
It will be a maintenence problem to put these tests in diffrent suite files, since they will be based on the same set of templates.
We would like to put quite a lot of information in variables. It would simplify the overview if there was a built in namespace/grouping functionality. This would make it possible to introduce folding for each namespace/Group.
In the UI one could unfold only the section one is interested in.
To make validator tests more maintainable, it would be nice if one could reference other sets of variables/connectors or templates from a suite.
It will be easier to 'divide and conquer' testing if some common parts could be reused instead of having to copy/paste between suites.