Certification and Verification
Overview
Much has been said recently about "out of the box interoperability" and many wanting to implement SIF have wondered "if two SIF agents are certified, then shouldn't they automatically work together and why wouldn’t we always be happy with the results?"
What SIF Certification Tests
The SIF certification process tests the behavior of the SIF agent that is associated with an application, and does it thoroughly. It makes sure that the agent implements security models correctly, registers in a zone correctly, formats messages correctly, sends acknowledgements correctly and does exactly what it advertises it does in its Conformance Statement.
For providers, it tests to make sure that the agent can respond to requests, even if those requests are unusual and when it generates events, that those messages are formed correctly. The certification test also tests to make sure that the agent can provide every object specified in the Conformance Statement.
For subscribers, the process makes sure that the agent can receive and process event messages and make requests and properly handle the responses. The certification test also checks to make sure that the agent can handle every object specified in the Conformance Statement.
These tests are numerous and exhaustive, but they only go as far as testing the agent’s functionality and not the application itself.
What SIF Certification Doesn’t Test
There are a few things that the certification test does not test:
- Although mandatory element values are required, the test process doesn’t require that a providing agent always give values for all the optional elements that it says it provides. For example, a Student Management System supports a learner MiddleName field and the value is optional. Every published record does not need to have this value. Another example might be ULN (Unique Learner Number), however. It is optional, but some subscribing applications may not work properly if it is not published. Problem.
- When the providing agent runs the test, it is using test data of its choice, which might (and probably is) cleaner and more complete than the data that one would find at the typical school. So, the provider SIF agent might be “on its best behavior” during the test and may run different when confronted with real world data. Problem.
- If a ZIS has validation turned on, publishing agent code translation errors come to light quickly because the publisher sends bad information in the field and the ZIS rejects the message because of the bad value. A subscribing agent, however, will store a SIF code value into a field in the application’s database where a translated value should be if a code value is not properly handled in its SIF agent. On the subscribing agent side, all you will notice is an application that doesn’t appear to work correctly. The certification test does not check to make sure that a subscribing application properly translates the incoming code value to the correct local value. Problem.
- SIF Extended Elements are a very useful way to add to the SIF specification between releases of the specification or if a particular requirement is so specialized that it will never likely make its way into the official specification. If a subscribing application requires a certain SIF Extended Element and there is no provider, the subscribing application will not work properly. The certification process does not do anything with Extended Elements.
The SIF certification process is intended to be generic – to thoroughly test all the way through the application would require a custom designed test for each certification.
What is an SIF Application Verification Test
A verification test is different from the SIF certification test in that it tests pairs of applications that are connected through a SIF infrastructure.
The end result of the test is to verify that the subscribing application continues to work properly when it receives its input through its SIF agent instead of its normal user interface.
The Verification Test Ledger
If the subscribing application successfully works and all functionality works correctly, the results of the tests will be made available on a web site for future reference by the two companies. This will include (if acceptable to the two companies):
- A description of the application and the SIF agent
- A description of the tests performed
- The ZIS audits and access to the messages (it will be required that depersonalized test information only be used in these tests)
- A section where two agent providers may add reference sites where the combination of applications is in use (assuming permission to do so has been granted)
The Verification Test Process
The testing process will typically involve one or more agent providers connecting to the ZIS testing facility located at our office. The connections can be HTTP and/or HTTPS – if HTTPS, we have a certificate server and can issue certificates for use in testing.
The Test Lab
A test lab has been up that contains a four server high-availability ZIServer ZIS cluster and two agent machines for testing, although the agents would normally be connected from user sites.
The Tests
The tests would be determined by the functionality of the subscribing agent. If the two agents passed information back and forth, then each agent would construct tests to validate the information it receives from the other. In either case, the receiving party would be the controller of the test.
The determination of the success of the test is if the application behaved correctly given the information passed through the SIF interface.
Dispute Resolution
It is bound to occur – both ends of the connection believe that their agent is behaving properly, yet the subscriber is not working properly. This is one of the reasons why this testing is done at a site where a team of experts is available to analyze the message stream, compare it with the specification and provide advice.
The plan is to not only schedule only one or two of these at any given time so that a reasonable amount of attention can be paid to the testing session by local staff.
The Test Results
If the test ends up in a successful test, the results are published on a public site, including, the information about the test, but also the detailed information about the test, so that others can look at how the test was performed.
Testing like this gives those wanting to implement this application pair confidence that the two applications work together successfully.