In the ear­ly days of sys­tems devel­op­ment, most appli­ca­tions had their own pro­pri­etary data man­age­ment sys­tems, not because this was a good idea but rather because there weren’t sophis­ti­cat­ed data­base sys­tems that man­aged things such as pre­vent­ing mul­ti­ple users try­ing to update the same data at the same time, sched­ul­ing pre­ven­tive main­te­nance and man­ag­ing secu­ri­ty. But hav­ing all this com­plex­i­ty in many places was not good for many rea­sons, among them were:

• Things like secu­ri­ty and pri­va­cy weren’t imple­ment­ed in a total­ly con­sis­tent man­ner – if each appli­ca­tion com­pa­ny need­ed to devel­op the code itself – even if they were all work­ing from a com­mon inter­na­tion­al stan­dard – it would nev­er be the same as if there was a sin­gle piece of soft­ware enforc­ing all the rules
• The sys­tem as a whole end­ed up being much less reli­able because there were many more places where the "com­plex log­ic" parts were imple­ment­ed
• The indi­vid­ual pieces were much more expen­sive ini­tial­ly to license and to main­tain because they need­ed to do much more than appli­ca­tions that didn’t need to do these activ­i­ties

The same prin­ci­ples could eas­i­ly extend to net­work­ing or any one of sev­er­al oth­er areas we often take for grant­ed when we work with mod­ern servers such as Win­dows Serv­er. What we want to do here is exam­ine a pop­u­lar method for imple­ment­ing SIF in large orga­ni­za­tions, look at the weak spots and see how Man­aged Vir­tu­al Zones can elim­i­nate these weak­ness­es. It is a rad­i­cal depar­ture and will require that the read­er be able to for­get what you’ve heard is “the only way SIF can be imple­ment­ed.” This paper is intend­ed for those who are famil­iar with SIF and will specif­i­cal­ly use exam­ples from the UK ver­sion of the spec­i­fi­ca­tion. The opin­ions expressed here­in are sole­ly those of the author and may not reflect those of any­one out­side of Visu­al Soft­ware, Inc.

Multi-Zone Agents

A mul­ti-zone agent is a SIF agent that par­tic­i­pates in more than one SIF zone. It may be a pub­lish­er or a sub­scriber; it may use push more or pull more and it may run over HTTP or HTTPS. The SIF spec­i­fi­ca­tion itself doesn’t restrict an agent from how many zones it may join – there is only one restric­tion:

• For each zone, for each con­text, for each object there may be at most one provider.

Now, the provider is the agent in the zone that answers requests from oth­er agents when they ask for infor­ma­tion with­out being spe­cif­ic about from which agent they want it. So, an agent may sub­scribe in more than one zone and it may gen­er­ate events in more than one zone. It may also be the provider in more than one zone — the only restric­tion is that there must not more than one provider in the same zone.

Shared Services

Very often, appli­ca­tions are shared between zones either at the Local Author­i­ty (LA) or Region­al Broad­band Con­sor­tium (RBC) lev­el. In a mul­ti-zone envi­ron­ment, the appli­ca­tion needs have log­ic to man­age the mul­ti­ple con­nec­tions and keep the data straight between the dif­fer­ent schools. If it need­ed to gen­er­ate events or if it were to serve as the provider in one or more of the zones, it would need to make sure that it gave the cor­rect infor­ma­tion in a par­tic­u­lar zone using the cor­rect GUIDs that were appro­pri­ate for that zone. It even gets more com­pli­cat­ed when cer­tain SIF object cross-ref­er­ence oth­er SIF objects. Take for exam­ple: what if in one of the school zones a ref­er­ence is made to "Oak Street Nurs­ery School" using a giv­en SIF RefId by a giv­en MIS. It is very doubt­ful that anoth­er MIS from anoth­er sup­pli­er being used at anoth­er school would use the same SIF RefId to rep­re­sent "Oak Street Nurs­ery School", and using the Estab­lish­men­tId won't help much because in the UK, Estab­lish­men­tIds aren't assigned for nurs­ery schools.

The Multi-Zone Agent Approach

The mul­ti-zone agent approach is very sim­ple: cre­ate over­lap­ping zones as need­ed and have the mul­ti-zone agents appear in more than one zone. This approach relies on the agent to be able to man­age the data com­plex­i­ties that occur because an agent is receiv­ing data from many unre­lat­ed sources. By doing this, the agent:

• …is now respon­si­ble for not inap­pro­pri­ate­ly comin­gling data
• …must han­dle the dif­fer­ences between the way dif­fer­ent sup­pli­ers pro­vide infor­ma­tion
• …must make sure all audits are ade­quate­ly main­tained
• …must pro­vide good admin­is­tra­tion facil­i­ties (per­haps a script­ing lan­guage) con­sid­er­ing that any activ­i­ty will need to be applied to many (thou­sands of) con­nec­tions

To illus­trate this and make the dia­gram sim­ple, I'll only show 14 schools where a typ­i­cal RBC might have 2,500 schools. The box labeled "RBC App" rep­re­sents an appli­ca­tion being shared at a Region­al Broad­band Con­sor­tium and the box­es labeled "School" rep­re­sent SIF school zones. Each of these school zones would prob­a­bly have data pro­vid­ed by an MIS (Man­age­ment Infor­ma­tion Sys­tem — the UK equiv­a­lent of a Stu­dent Infor­ma­tion Sys­tem). Now, con­sid­er some of the things you might need to do as an admin­is­tra­tor of this shared appli­ca­tion:

• 2,500 agent con­nec­tions will need to be set up to install this agent.
• If the appli­ca­tion needs to be shut down for main­te­nance, it will need to be put to sleep in all 2,500 zones.
• Any SIF admin­is­tra­tion that is per­formed in one zone will need to be per­formed in all 2,500 zones.
• If the appli­ca­tion is a SIF "pull" mode agent, then it will prob­a­bly have 2,500 pull threads run­ning. If each of these are polling once a minute, 2,500 SIF mes­sages are being sent every minute, sim­ply to ask if there is any­thing to do and 2,500 more to reply back with "noth­ing to do". 5,000 mes­sages per minute per appli­ca­tion to do noth­ing.
• Ele­ment-lev­el fil­ter­ing will need to be set up for all 2,500 con­nec­tions.
• If the con­nec­tions are not local, each of the 2,500 con­nec­tions will require a HTTPS cer­tifi­cate, which will like­ly expire at a dif­fer­ent time of the year

This was with a sin­gle appli­ca­tion. Now, add a few oth­er appli­ca­tions at the RBC lev­el and some at the LA lev­el:

In this exam­ple, there are only four shared appli­ca­tions at the RBC lev­el and two at the LA lev­el and we are only show­ing 14 of the 2,500 schools. In a more real­is­tic exam­ple, we might have 10 appli­ca­tions shared at the RBC lev­el and 5 appli­ca­tions at each of 20 LAs. Doing some quick cal­cu­la­tions:

• Admin­is­tra­tors would need to set up: 10 x 2,500 (RBC appli­ca­tions) + (5 x 20 (LA appli­ca­tions) x 125 schools at each LA) con­nec­tions = 37,600 agents to set up
• If an RBC agent needs main­te­nance it would need to be put to sleep in 2,500 zones (as before)
• If the RBC data cen­ter was need­ing to do main­te­nance, 25,000 sleep requests would need to be sent
• If these were "Pull" mode agents and polled once per minute, 50,200 SIF mes­sages would be sent across the net­work each minute just to see if there were any wait­ing mes­sages
• Ele­ment-lev­el fil­ter­ing would need to be set up on 25,100 con­nec­tions
• If the con­nec­tions are not local, each of the 25,100 con­nec­tions will require a HTTPS cer­tifi­cate, which will like­ly expire at a dif­fer­ent time of the year

Managed Virtual Zones

With Man­aged Vir­tu­al Zones, each appli­ca­tion agent is only respon­si­ble for the data in a sin­gle zone. This archi­tec­ture uses a cen­tral piece of soft­ware (Envoy) that man­ages vir­tu­al zones at dif­fer­ent lev­els (LA, RBC) that con­tain the SIF object infor­ma­tion appro­pri­ate for that lev­el and scope. For exam­ple, a vir­tu­al zone for a giv­en LA will con­tain infor­ma­tion for all learn­ers who spend part of their day in that LA, but not for learn­ers from oth­er LAs. With this archi­tec­ture, the com­plex­i­ties of the large infra­struc­ture are cen­tral­ly man­aged, mak­ing each con­nect­ed agent much sim­pler because it has to do far less. An Envoy-con­nect­ed imple­men­ta­tion might look more like: In this exam­ple, each appli­ca­tion has a sin­gle con­nec­tion to a sin­gle zone. That zone will have the fol­low­ing fea­tures:

• They will con­tain infor­ma­tion for learn­ers who attend any part of the day there. This will prop­er­ly han­dle records for learn­ers who attend more than one school, even if the MIS sys­tems in those schools come from dif­fer­ent sup­pli­ers. This works for 14–19 learn­ers as well as spe­cial edu­ca­tion learn­ers and works at the school lev­el as well as at the LA lev­el. Fur­ther­more, vir­tu­al zones can be cre­at­ed for any group­ing of schools (part­ner­ship) if needs dic­tate.
• Par­ent records from mul­ti­ple MISs may be matched (option), so that mul­ti­ple records for the same per­son map into a sin­gle author­i­ta­tive record
• LAIn­fo and SchoolIn­fo records are mapped into a sin­gle set of author­i­ta­tive records — each school becomes the author­i­ty for its own infor­ma­tion
• Vir­tu­al zones can be assigned sets of ele­ment lev­el fil­ters (just like agents in ZIS­erv­er) — this is called "Blan­ket fil­ter­ing". This is use­ful for set­ting up default fil­ter­ing for large groups of agents
• If a shared appli­ca­tion needs to go to sleep or per­form main­te­nance, it makes the request to its one con­nec­tion
• If a shared appli­ca­tion is a "Pull" mode appli­ca­tion and polls once a minute, it sends one mes­sage and receives one response.

In the same exam­ple as above, where we had 10 appli­ca­tions at the RBC lev­el and 5 appli­ca­tions at each of 20 LAs. Let's do some com­par­isons:

There are many dif­fer­ences between the two mod­els, but the con­cep­tu­al dif­fer­ences between the two are very impor­tant:

• The com­plex parts of "the sys­tem" are being han­dled in one place (Envoy) instead of many (in every agent, writ­ten by every agent devel­op­er)
  • Com­plex deci­sions are han­dled con­sis­tent­ly
  • They are debugged once and are regres­sion test­ed every time any change is made
  • All deci­sions are made in a con­sis­tent man­ner — busi­ness log­ic is made through a "busi­ness rules" engine, leav­ing the deci­sion mak­ing process to those who know the busi­ness best
  • All actions are logged in an audit trail and tools are made avail­able so that the audits are sim­ple to access
  • Data is han­dled a min­i­mum num­ber of times — When an MIS pub­lish­es an object, it is stored in Envoy. Events are for­ward­ed to the appro­pri­ate vir­tu­al zones and then the data stays in Envoy until the data is request­ed. It is not passed from zone to zone or from web ser­vice to web ser­vice as in oth­er archi­tec­tures sim­ply to make it avail­able else­where. In a large envi­ron­ment, data can­not be re-han­dled sim­ply for the sake of mak­ing it avail­able at a sec­ond or third loca­tion.
• Data pri­va­cy is han­dled in a con­sis­tent man­ner and pro­vides a facil­i­ty for fall­back "blan­ket-lev­el" fil­ter­ing.