Much has been said recent­ly about "out of the box inter­op­er­abil­i­ty" and many want­i­ng to imple­ment SIF have won­dered "if two SIF agents are cer­ti­fied, then shouldn't they auto­mat­i­cal­ly work togeth­er and why wouldn’t we always be hap­py with the results?"

What SIF Certification Tests

The SIF cer­ti­fi­ca­tion process tests the behav­ior of the SIF agent that is asso­ci­at­ed with an appli­ca­tion, and does it thor­ough­ly. It makes sure that the agent imple­ments secu­ri­ty mod­els cor­rect­ly, reg­is­ters in a zone cor­rect­ly, for­mats mes­sages cor­rect­ly, sends acknowl­edge­ments cor­rect­ly and does exact­ly what it adver­tis­es it does in its Con­for­mance State­ment.

For providers, it tests to make sure that the agent can respond to requests, even if those requests are unusu­al and when it gen­er­ates events, that those mes­sages are formed cor­rect­ly. The cer­ti­fi­ca­tion test also tests to make sure that the agent can pro­vide every object spec­i­fied in the Con­for­mance State­ment.

For sub­scribers, the process makes sure that the agent can receive and process event mes­sages and make requests and prop­er­ly han­dle the respons­es. The cer­ti­fi­ca­tion test also checks to make sure that the agent can han­dle every object spec­i­fied in the Con­for­mance State­ment.

These tests are numer­ous and exhaus­tive, but they only go as far as test­ing the agent’s func­tion­al­i­ty and not the appli­ca­tion itself.

What SIF Certification Doesn’t Test

There are a few things that the cer­ti­fi­ca­tion test does not test:

• Although manda­to­ry ele­ment val­ues are required, the test process doesn’t require that a pro­vid­ing agent always give val­ues for all the option­al ele­ments that it says it pro­vides. For exam­ple, a Stu­dent Man­age­ment Sys­tem sup­ports a learn­er Mid­dle­Name field and the val­ue is option­al. Every pub­lished record does not need to have this val­ue. Anoth­er exam­ple might be ULN (Unique Learn­er Num­ber), how­ev­er. It is option­al, but some sub­scrib­ing appli­ca­tions may not work prop­er­ly if it is not pub­lished. Prob­lem.
• When the pro­vid­ing agent runs the test, it is using test data of its choice, which might (and prob­a­bly is) clean­er and more com­plete than the data that one would find at the typ­i­cal school. So, the provider SIF agent might be “on its best behav­ior” dur­ing the test and may run dif­fer­ent when con­front­ed with real world data. Prob­lem.
• If a ZIS has val­i­da­tion turned on, pub­lish­ing agent code trans­la­tion errors come to light quick­ly because the pub­lish­er sends bad infor­ma­tion in the field and the ZIS rejects the mes­sage because of the bad val­ue. A sub­scrib­ing agent, how­ev­er, will store a SIF code val­ue into a field in the application’s data­base where a trans­lat­ed val­ue should be if a code val­ue is not prop­er­ly han­dled in its SIF agent. On the sub­scrib­ing agent side, all you will notice is an appli­ca­tion that doesn’t appear to work cor­rect­ly. The cer­ti­fi­ca­tion test does not check to make sure that a sub­scrib­ing appli­ca­tion prop­er­ly trans­lates the incom­ing code val­ue to the cor­rect local val­ue. Prob­lem.
• SIF Extend­ed Ele­ments are a very use­ful way to add to the SIF spec­i­fi­ca­tion between releas­es of the spec­i­fi­ca­tion or if a par­tic­u­lar require­ment is so spe­cial­ized that it will nev­er like­ly make its way into the offi­cial spec­i­fi­ca­tion. If a sub­scrib­ing appli­ca­tion requires a cer­tain SIF Extend­ed Ele­ment and there is no provider, the sub­scrib­ing appli­ca­tion will not work prop­er­ly. The cer­ti­fi­ca­tion process does not do any­thing with Extend­ed Ele­ments.

The SIF cer­ti­fi­ca­tion process is intend­ed to be gener­ic – to thor­ough­ly test all the way through the appli­ca­tion would require a cus­tom designed test for each cer­ti­fi­ca­tion.

What is an SIF Application Verification Test

A ver­i­fi­ca­tion test is dif­fer­ent from the SIF cer­ti­fi­ca­tion test in that it tests pairs of appli­ca­tions that are con­nect­ed through a SIF infra­struc­ture.

The end result of the test is to ver­i­fy that the sub­scrib­ing appli­ca­tion con­tin­ues to work prop­er­ly when it receives its input through its SIF agent instead of its nor­mal user inter­face.

The Verification Test Ledger

If the sub­scrib­ing appli­ca­tion suc­cess­ful­ly works and all func­tion­al­i­ty works cor­rect­ly, the results of the tests will be made avail­able on a web site for future ref­er­ence by the two com­pa­nies. This will include (if accept­able to the two com­pa­nies):

• A descrip­tion of the appli­ca­tion and the SIF agent
• A descrip­tion of the tests per­formed
• The ZIS audits and access to the mes­sages (it will be required that deper­son­al­ized test infor­ma­tion only be used in these tests)
• A sec­tion where two agent providers may add ref­er­ence sites where the com­bi­na­tion of appli­ca­tions is in use (assum­ing per­mis­sion to do so has been grant­ed)

The Verification Test Process

The test­ing process will typ­i­cal­ly involve one or more agent providers con­nect­ing to the ZIS test­ing facil­i­ty locat­ed at our office. The con­nec­tions can be HTTP and/or HTTPS – if HTTPS, we have a cer­tifi­cate serv­er and can issue cer­tifi­cates for use in test­ing.

The Test Lab

A test lab has been up that con­tains a four serv­er high-avail­abil­i­ty ZIS­erv­er ZIS clus­ter and two agent machines for test­ing, although the agents would nor­mal­ly be con­nect­ed from user sites.

The Tests

The tests would be deter­mined by the func­tion­al­i­ty of the sub­scrib­ing agent. If the two agents passed infor­ma­tion back and forth, then each agent would con­struct tests to val­i­date the infor­ma­tion it receives from the oth­er. In either case, the receiv­ing par­ty would be the con­troller of the test.

The deter­mi­na­tion of the suc­cess of the test is if the appli­ca­tion behaved cor­rect­ly giv­en the infor­ma­tion passed through the SIF inter­face.

Dispute Resolution

It is bound to occur – both ends of the con­nec­tion believe that their agent is behav­ing prop­er­ly, yet the sub­scriber is not work­ing prop­er­ly. This is one of the rea­sons why this test­ing is done at a site where a team of experts is avail­able to ana­lyze the mes­sage stream, com­pare it with the spec­i­fi­ca­tion and pro­vide advice.

The plan is to not only sched­ule only one or two of these at any giv­en time so that a rea­son­able amount of atten­tion can be paid to the test­ing ses­sion by local staff.

The Test Results

If the test ends up in a suc­cess­ful test, the results are pub­lished on a pub­lic site, includ­ing, the infor­ma­tion about the test, but also the detailed infor­ma­tion about the test, so that oth­ers can look at how the test was per­formed.

Test­ing like this gives those want­i­ng to imple­ment this appli­ca­tion pair con­fi­dence that the two appli­ca­tions work togeth­er suc­cess­ful­ly.