What's the Problem?

In most larg­er orga­ni­za­tions, sev­er­al com­put­er appli­ca­tions are used to sup­port the process of edu­cat­ing stu­dents, in one way or anoth­er. The major­i­ty of these appli­ca­tions know about who attends the school, who teach­es there, and who the teach­ers are. Some of them know the stu­dents' sched­ules while oth­ers know details such as the grades they received or detailed health-relat­ed infor­ma­tion such as immu­niza­tions or records of injuries. The point is that much of this infor­ma­tion is com­mon to many of them.

So, what is the "best" approach to man­ag­ing the infor­ma­tion that is shared by these appli­ca­tions?

• Some say it is best to have a sin­gle, cen­tral­ized data­base con­tain­ing all the infor­ma­tion and one large appli­ca­tion that does every­thing. (This is referred to as the "inte­grat­ed approach".) This has its effi­cien­cies, but the appli­ca­tion must be:
  • able to do every­thing and do every­thing well
  • able to scale well, because all school users will be using it reg­u­lar­ly
• Oth­ers say that it is best to use indi­vid­ual appli­ca­tions from those com­pa­nies with exper­tise in the appli­ca­tion sub­ject areas (and find a way to resolve the data issues). This is referred to as the "best of breed" approach.

Most of the time, what ends up being adopt­ed is either a "best of breed" or a mix­ture of the two: an orga­ni­za­tion starts off want­i­ng to get an inte­grat­ed sys­tem, but then real­izes that some oth­er sys­tems are need­ed as well.

Regard­less of what's in place, these orga­ni­za­tions will have mul­ti­ple sys­tems that will, by neces­si­ty, con­tain the same data. The big ques­tion is: how does that data get into those sys­tems and who keeps it up to date?

How can this problem be managed?

Before we intro­duce SIF, we'll speak to three oth­er alter­na­tives that have been used in the past and are being used to address this issue; one by default (man­u­al data entry), anoth­er because it was a sim­ple tech­nol­o­gy that was avail­able (exports and imports), and a third that is being exper­i­ment­ed with in oth­er indus­tries (enter­prise ser­vice bus­es). These are only being pre­sent­ed here to give some per­spec­tive.

Manual data entry

The most obvi­ous method (and what gets done by default) is to enter data man­u­al­ly into each sys­tem using the application's own user inter­face. Besides being the most obvi­ous, this is also the most expen­sive and is the most error prone.

We had a cus­tomer dis­trict once fol­low some sam­ple stacks of paper­work around their dis­trict to see how much time was spent enter­ing data from those pieces of paper into var­i­ous sys­tems and how much of that data entry was redun­dant. With a stu­dent pop­u­la­tion of about 7,000 stu­dents, they esti­mat­ed that they spent about two staff posi­tions worth of time in a giv­en year sim­ply doing the redun­dant part of the data entry.

Exports and Imports

This method of syn­chro­niz­ing data works by hav­ing the provider appli­ca­tion of the data cre­ate an "export" file on a reg­u­lar basis, store it in a pre-deter­mined loca­tion for each type of data (stu­dent teacher, par­ent, school, enroll­ment, etc.). Then, each of the "receiv­ing" appli­ca­tions look for files in that loca­tion, open them and read the infor­ma­tion. If they see any changes in the files, they refresh their data­bas­es with the new infor­ma­tion.

A set­up like this can be set up by an IT depart­ment if all the appli­ca­tions have appro­pri­ate inter­faces. This assumes that:

• there is an "author­i­ty" appli­ca­tion for each type of data (the SIS would nor­mal­ly be the author­i­ty for stu­dent data, an HR appli­ca­tion might be the author­i­ty for teacher data, etc.)
• those appli­ca­tions have the abil­i­ty to cre­ate "extract" files (typ­i­cal­ly in CSV (Com­ma-Sep­a­rat­ed-Val­ues) for­mat) — the design (and main­te­nance of that design) of these files is often the respon­si­bil­i­ty of the IT staff mem­ber
• those appli­ca­tions can gen­er­ates these extract files on a reg­u­lar basis
• the oth­er appli­ca­tions that need the data have match­ing inter­faces that can "import" these extract files on a sched­uled basis
• the appli­ca­tions that import the extract files do rea­son­able things when they encounter errors in the source data
• all the clocks on all these servers stay syn­chro­nized so that the "imports" always fol­low the "exports"

Many orga­ni­za­tions use this, but it is cost­ly and error-prone. Cost­ly because whether the IT staff is man­ag­ing the CSV import/export process or the soft­ware sup­pli­ers are, the end user ends up pay­ing the cost. There are no stan­dards for the lay­outs of these files and most appli­ca­tion sup­pli­ers typ­i­cal­ly spend a con­sid­er­able effort main­tain­ing a library of CSV-inter­faces.

These inter­faces are error-prone because the CSV for­mat doesn't have the capa­bil­i­ty to recov­er from errors like oth­er trans­mis­sion meth­ods do. Typ­i­cal­ly, if there is an error in a field, the entire record is reject­ed — or per­haps even the remain­der of the file.

Enterprise Service Bus

There appear to be many def­i­n­i­tions of this term (we will not include the air­port car rental com­pa­ny shut­tles in this dis­cus­sion), but most of them cen­ter around the offer­ings of par­tic­u­lar com­pa­nies, such as Microsoft, Ora­cle or IBM. They allow con­nec­tions from dif­fer­ent oper­at­ing sys­tems types, they sup­port web ser­vices inter­faces and use XML and SOAP to trans­fer data.

Because of the way they are struc­tured, they have a bet­ter error han­dling capac­i­ty than the old­er CSV import and export and because of its design, the trans­fer of infor­ma­tion will be right away, instead of in batch­es. This means when a stu­dent enrolls, his or her infor­ma­tion is imme­di­ate­ly sent to all con­nect­ed sys­tems and is avail­able in a few sec­onds, not in the next day or two.

The prob­lem with ESB is that it is a tech­nol­o­gy — it defines the trans­porta­tion mech­a­nism through which data can pass. It is a very good and well thought-out one, but it doesn't do any­thing for mak­ing life sim­pler or more cost-effec­tive for a school orga­ni­za­tion. Those who devel­op around it will still need to re-do much of the work that all of the com­pa­nies par­tic­i­pat­ing in SIF have done for the last 10 years. This is why these projects tend to be enor­mous­ly expen­sive.

Integration Hub

The biggest dif­fer­ence with SIF and these oth­er ways of solv­ing this prob­lem is that it is an indus­try stan­dard. It wasn't imposed by a gov­ern­ment or invent­ed by a sin­gle com­pa­ny, but was cre­at­ed by indus­try experts who amaz­ing­ly agreed on a num­ber of key con­cepts, includ­ing:

• What things need­ed to be shared between appli­ca­tions (these became SIF objects)
• What char­ac­ter­is­tics those things had (these became the ele­ments and attrib­ut­es in those objects)
• How those things would be passed between appli­ca­tions (this became the def­i­n­i­tion of the Zone Inte­gra­tion Serv­er):
  • Auto­mat­i­cal­ly send­ing data as soon as it changes — the Publish/Subscribe mod­el
  • Allow­ing appli­ca­tions to ask for data when it needs it — the Request/Response mod­el
• How those things would be pro­tect­ed:
  • Encryp­tion, HTTPS, Cer­tifi­cates…
  • Object-lev­el pro­tec­tion — deter­min­ing who has the right to see which objects
  • Ele­ment-lev­el pro­tec­tion — deter­min­ing who has the right to see which data ele­ments in those objects

This didn't hap­pen overnight; this has been an ongo­ing process since the late 1990's and has involved hun­dreds of com­pa­nies and thou­sands of school orga­ni­za­tions (peo­ple who use and sup­port these appli­ca­tions every day). It has grown to sup­port three region­al spec­i­fi­ca­tions in the US, UK and Aus­tralia and is cur­rent­ly being used in tens of thou­sands of schools world­wide.

SIF is stan­dards-based; it is fast and it is secure. It can eco­nom­i­cal­ly meet the needs of a very small instal­la­tion, yet scale to meet the needs of a large state with mil­lions of stu­dents.

How Does ThisWork?

In a SIF envi­ron­ment, appli­ca­tions share data through web con­nec­tions that send and receive XML lan­guage mes­sages in a stan­dard­ized for­mat. All of the appli­ca­tions com­mu­ni­cat­ing con­nect to a com­mon hub (called a Zone Inte­gra­tion Serv­er (ZIS)) that is respon­si­ble for mak­ing sure that all mes­sages are deliv­ered prop­er­ly. So, at the high­est lev­el, the col­lec­tion of sys­tems looks like the pic­ture to the right.

The Zone Inte­gra­tion Serv­er is soft­ware that runs either at the school orga­ni­za­tion or at a host­ed loca­tion.

For all of these appli­ca­tions to be able to com­mu­ni­cate prop­er­ly, they must "use a com­mon lan­guage" as fol­lows:

• The XML mes­sages must have the same for­mat
• If there are encod­ed fields (such as "Tele­phone Num­ber Type" or "Cit­i­zen­ship Sta­tus") in the data, all par­ties must agree to use a com­mon set of val­ues for these codes when the data is being trans­ferred (they may use what­ev­er they want in their own appli­ca­tions, but as the data is being trans­ferred it must be in this stan­dard­ized form).
• All par­ties must agree to the same method for exchang­ing these mes­sages ("the pro­to­col")

The Message Broker

The ZIS is a piece of soft­ware that runs as a web site that routes mes­sages between appli­ca­tions. It does not hold per­ma­nent copies of school infor­ma­tion; the only time it keeps much of any­thing is if one of the receiv­ing appli­ca­tions is offline or slow — it will tem­porar­i­ly hold mes­sages for it until the receiv­ing appli­ca­tion drains its mes­sage queue. (Anoth­er excep­tion is with our ZIS­erv­er prod­uct — we store a com­plete audit trail of all mes­sages that pass through the ZIS — these only main­tained for audit­ing pur­pos­es)

A ZIS can either be host­ed and shared between many orga­ni­za­tions or can be installed local­ly at a school or school dis­trict.

Besides rout­ing mes­sages, it is also respon­si­ble for main­tain­ing the infrastructure's secu­ri­ty.

• Before an appli­ca­tion can con­nect to a SIF infra­struc­ture, the ZIS admin­is­tra­tor must "cre­ate" a con­nec­tion for it and autho­rize the appli­ca­tion for cer­tain types of oper­a­tions. For exam­ple, an admin­is­tra­tor may cre­ate a con­nec­tion for a library appli­ca­tion and autho­rize it to receive stu­dent and teacher demo­graph­ic infor­ma­tion, and stu­dent enroll­ment infor­ma­tion, but not finan­cial infor­ma­tion.
• Before one loca­tion (per­haps a school) con­nects to anoth­er loca­tion (per­haps a host­ing provider), typ­i­cal­ly dig­i­tal cer­tifi­cates are exchanged, so that each par­ty can be assured that the con­nec­tion is safe.

Applications and Connectors

When SIF was first envi­sioned, it was assumed that most appli­ca­tions would be adapt­ed to work with SIF and that they wouldn't be built from the ground-up know­ing how to direct­ly com­mu­ni­cate with a Zone Inte­gra­tion Serv­er. These adapters are called "SIF Agents".

When SIF was in its infan­cy, sev­er­al com­pa­nies set out to address the need to build SIF Agents. We at Visu­al Soft­ware went in one direc­tion; every­one else went in the oth­er. The oth­ers began with the assump­tion that these agents were going to be dif­fer­ent enough that each of them should be built most­ly from the ground-up (with a lit­tle help from an ADK (a programmer's library).

We at Visu­al Soft­ware, how­ev­er, had spent the pre­vi­ous eight years refin­ing tech­niques and build­ing tech­nol­o­gy allow­ing "new ser­vices to attached to exist­ing tech­nolo­gies with­out dis­rupt­ing them." So, instead of start­ing fresh, we used these prin­ci­ples and used this code base to cre­ate the first ver­sion of our con­fig­urable SIF agent ZIA­gent, that pro­vides SIF ser­vices to exist­ing appli­ca­tions with­out requir­ing pro­gram­ming and with­out requir­ing mod­i­fi­ca­tions to the tar­get appli­ca­tion.

In years since, we've con­tin­ued to refine this mod­el, mak­ing it more and more effi­cient, flex­i­ble, reli­able and scal­able. More­over, we've made the agent do "the dif­fi­cult stuff" by default; the things many leave out; the things that sep­a­rate the good from the excel­lent agents. To learn more about our con­fig­urable agent, see ZIA­gent.

How Soft­ware Sup­pli­ers See SIF

Almost uni­ver­sal­ly, soft­ware sup­pli­ers would pre­fer to spend as much effort as pos­si­ble focused on their own appli­ca­tion, but instead find them­selves get­ting side­tracked with mul­ti­ple inter­faces to oth­er appli­ca­tions. They need to stay aware of when the oth­er appli­ca­tions change, then change their inter­faces to match the chang­ing designs of the oth­er appli­ca­tions. This is an expen­sive process that takes away from their abil­i­ty to address issues with­in their own prod­uct.

If it isn't a waste of their time, then they're charg­ing a sig­nif­i­cant price for these inter­faces, because they take sig­nif­i­cant work to main­tain.

What SIF allows them to do is to cre­ate a sin­gle inter­face that can be used to con­nect to many oth­er appli­ca­tions. Unlike these Import/Export inter­faces (which most of the exist­ing ones are), the SIF inter­face can han­dle errors and prop­er­ly recov­er from them, can pro­tect pri­vate infor­ma­tion at lev­els meet or that exceed gov­ern­ment stan­dards and can even sim­pli­fy the entire process for their cus­tomers (if done prop­er­ly).

How School Orga­ni­za­tions See SIF

Schools need to solve a num­ber of data-relat­ed prob­lems:

• Stu­dent, teacher and con­tact infor­ma­tion appears in many dif­fer­ent appli­ca­tions
• If there isn't an auto­mat­ic update feed between them, the data becomes very inac­cu­rate
• If the time-to-update is long, there will be cer­tain oper­a­tional inef­fi­cien­cies (for exam­ple, day-old atten­dance data is not use­ful in the cafeteria/canteen)

SIF is a good way to solve those prob­lems because it has good error han­dling char­ac­ter­is­tics, it scales well and data is passed between appli­ca­tions as soon as the data event hap­pens.

We have found in many instal­la­tions that once the sys­tem has been set up, it doesn't need much care beyond nor­mal back­ups.

Because it is a stan­dard, a school orga­ni­za­tion can look to an appli­ca­tion sup­pli­er to pro­vide such an inter­face with­out that request being con­sid­ered unusu­al, nor seen by the ven­dor as some­thing use­ful for only one cus­tomer.

Is it Per­fect? (time to be hon­est…)

Well, like any­thing human-made, SIF has its issues. From our expe­ri­ence installing and sup­port­ing many SIF agents over the past ten years, we have found that if all par­tic­i­pants use the spec­i­fi­ca­tion the way it was intend­ed to be used, then things work remark­ably well and the users and sup­pli­ers are very pleased with the results.

Issues arise, how­ev­er, when­ev­er one par­ty or anoth­er begins to com­pro­mise the spec­i­fi­ca­tion. The cer­ti­fi­ca­tion process catch­es many of these errors, but it isn't per­fect either.

A good rule for eval­u­a­tion is if you ever hear any­one say­ing that a par­tic­u­lar agent only works when paired with anoth­er agent or when paired with a spe­cif­ic Zone Inte­gra­tion Serv­er, then even if it is SIF cer­ti­fied, don't even con­sid­er it to be a SIF agent. This means that they have com­pro­mised the SIF spec­i­fi­ca­tion and found a way to fool the cer­ti­fi­ca­tion test har­ness.