Over the past many months, we've heard many peo­ple talk about sev­er­al SIF archi­tec­ture choic­es and the rel­a­tive mer­its of each, espe­cial­ly in terms of per­for­mance. So, over the last sev­er­al weeks, we set out to com­pare sev­er­al of these archi­tec­tures in a con­trolled envi­ron­ment with a rel­a­tive­ly large data set with the intent of sim­u­lat­ing a heavy load, such as would be seen in an Aus­tralian State or a UK Region­al Broad­band Consortia.

The data set

The data set we test­ed with uses the Aus­tralia 1.1 spec­i­fi­ca­tion, and four types of objects from it:

• LEAIn­fo
• SchoolIn­fo
• Stu­dent­Per­son­al
• Stu­dentSchoolEn­roll­ment

The main objec­tives were to test for large amounts of data (as you would find with a few very large mes­sages), and more impor­tant­ly to inves­ti­gate the prob­lems you would see with many, many small mes­sages. The data set we used had the fol­low­ing num­bers of records in it:

• LEAIn­fo (16 records) — we used these in an exper­i­ment to split the pop­u­la­tion into 16 sections
• SchoolIn­fo (100 records) — we used these to split the pop­u­la­tion into 100 divisions
• Stu­dent­Per­son­al (300,000 records) — our test data had very large schools
• Stu­dentSchoolEn­roll­ment (330,000 records) — 30,000 of the stu­dents were enrolled in a sec­ond school; no stu­dents were enrolled in more than two schools

The tests

The tests per­formed were the following:

1. Sin­gle Zone, one provider, one sub­scriber, all data is passed from one side to the oth­er by send­ing events
2. Sin­gle Zone, one provider, one sub­scriber, all data is passed from one side to the oth­er by hav­ing the sub­scriber make a request for the data
3. Using 16 Vir­tu­al Zones, each with a provider, hav­ing 16 sub­scribers make requests simultaneously
4. Using 100 Vir­tu­al Zones, each with a provider, hav­ing 100 sub­scribers make requests simultaneously
5. Using a sin­gle zone, but using a sep­a­rate sub­scrib­ing agent for each SIF object, assum­ing that the appli­ca­tion at the sub­scrib­ing end could orches­trate the com­bined inputs of the mul­ti­ple SIF agents

The test lab setup

The fol­low­ing dia­gram depicts the lay­out of the test lab we used for these exper­i­ments. All of the servers have RAID disks and run­ning Win­dows Serv­er 2008 R2, except for the lap­top, which is run­ning Win­dows 7. All data­bas­es are locat­ed on either the HARPO or GROUCHO machines and are cross-mir­rored on the oth­er data­base serv­er. The ZEPPO machine is a NAS device used to reg­u­lar­ly back up data­bas­es dur­ing these tests.

The GUMMO serv­er held the ZIS. In the first test, it was mod­er­ate­ly busy, but nev­er real­ly taxed the serv­er. In the sec­ond and fol­low­ing tests, it act­ed as one of the sub­scriber servers. In tests 3 and 4, the CHICO machine was the Envoy serv­er. In order to be able to "digest" all of the out­put it was pro­duc­ing, we need­ed to enlist the oth­er servers so that we could keep up with all of the mes­sages being pro­duced by Envoy with no queues any­where. If we had put all of the agent con­nec­tions on a sin­gle machine (as in the 4^th test), we would not have been able to process all of them at the same time because of mem­o­ry con­straints. In a pro­duc­tion envi­ron­ment, the sub­scriber appli­ca­tion would like­ly be a "farmed" appli­ca­tion, split across sev­er­al machines as well (sim­i­lar to what we have here).

Test 1 — Single Zone, publisher sends events

In this test, we had the Provider SIF agent reg­is­ter in a sin­gle zone with a sin­gle sub­scriber appli­ca­tion as described in the fol­low­ing diagram:

The Provider agent ran a script that pub­lished events for all LEAIn­fo, SchoolIn­fo, Stu­dent­Per­son­al and Stu­dentSchoolEn­roll­ment objects (all 630,116 of them). As expect­ed, this took a very long time to com­plete. Dur­ing the mid­dle of the process, this snap­shot was tak­en from the ZIServer's mes­sage queue:

If you look at the time­stamps, you can see that the mes­sages were sent through the ZIS at a rate of about 5–6 per sec­ond. Con­sid­er­ing that we were using mod­est com­put­ers for these tests, in a real pro­duc­tion envi­ron­ment, these num­bers may look slight­ly bet­ter. But, at the high side of this aver­age, the mes­sages took about 30 hours to be trans­ferred from the provider to the sub­scriber. Dur­ing the test, no bot­tle­neck occurred in the ZIS­erv­er being used or in either of the two agents. The fol­low­ing screen shows the activ­i­ty from the ZIS­erv­er machine:

The oth­er two machines' resource mon­i­tors looked very sim­i­lar. Much of the time was spent in the "calls" — the mes­sages being deliv­ered, acknowl­edge­ments being sent back, remote pro­ce­dure calls, data­base call over­head, mir­ror­ing costs, etc., the things that are part of the pro­to­col — the "cost of doing business".

Test 2 — Single Zone, subscriber requests data

This test will vary more depend­ing on the actu­al size of the data (espe­cial­ly in the Stu­dent­Per­son­al objects), because the num­ber of objects that will fit in a sin­gle response mes­sage will vary depend­ing on the amount of infor­ma­tion in the aver­age object. DISCLAIMER: In our case, we didn't include too many fields in our Stu­dent­Per­son­al, so we saw a very large improve­ment in speed (the improve­ment we saw will be more typ­i­cal of what nor­mal users would see with a mix of objects and a mix of Add, Change, and Delete events mixed in, so we shouldn't be too far off with this any­way). In this data­base sam­ple, we see the records after they've been insert­ed into the data­base in the sub­scriber SIF Agent. We sam­pled about 40 min­utes' worth of activ­i­ty and took an aver­age of that peri­od to cal­cu­late the num­bers below the screen capture:

Differences in server loads between Tests 1 and 2

Dur­ing these tests, the loads on the servers were char­ac­ter­is­ti­cal­ly very dif­fer­ent from those used dur­ing the first test. Most servers were far less active than dur­ing the first test, with the excep­tion of the sub­scriber com­put­er which was more active, but for a short­er peri­od of time.

The Provider IIS Server

• This machine was more com­pute bound
• Per­formed less net­work I/O (sent many few­er messages),
• Per­formed few­er data­base INSERT oper­a­tions (relat­ed to SIF mes­sages being produced),
• Per­formed few­er SELECT oper­a­tions (it was able to read the con­fig­u­ra­tion and map­ping infor­ma­tion once for many mes­sages generated)

The ZIS IIS Server

• This machine was much less active because it received far few­er messages
• Per­formed far less net­work I/O
• Per­formed few­er data­base operations

The Subscriber IIS Server

• This machine had a dif­fer­ent mix — it was far more com­pute bound — it was more active pro­cess­ing the objects in mes­sages and less active pro­cess­ing the SIF protocol
• Per­formed less net­work I/O (received far few­er messages)
• Per­formed more data­base operations/second because it was not wait­ing for net­work I/O — get­ting these data­base oper­a­tions com­plet­ed more quick­ly allowed it to com­plete its job faster

The Database Server

• The data­base serv­er, in gen­er­al, was far busier for a short­er amount of time

The fol­low­ing resource mon­i­tor was from the busiest of the sys­tems (the Sub­scriber IIS Server):

Even with all the work it had to do, you can see that it was aver­ag­ing only 4% CPU uti­liza­tion, not enough to reg­is­ter for disk and net­work uti­liza­tion, and only a small por­tion of the server's memory.

Test 3 — Zone splitting using 16 zones

In this test and in Test 4, we used the Envoy Vir­tu­al zone man­ag­er, although sim­i­lar results could be derived using mul­ti-zone pub­lish­ing agents. The main dif­fer­ences between using Envoy and the mul­ti-zone agent approach (for the pur­pos­es of these tests) are as follows:

• Envoy comes as com­mer­cial off-the-shelf soft­ware that can add the func­tion­al­i­ty to exist­ing envi­ron­ments with­out requir­ing that any exist­ing agents be modified
• Envoy allows addi­tion­al flex­i­bil­i­ty in that Vir­tu­al Zones may be cre­at­ed that con­tain any com­bi­na­tion of schools (schools may also appear in more than one zone if needed)
• Envoy allows users to set up data qual­i­ty check­ing rules that will pre­vent data from being for­ward­ed if it doesn't meet cer­tain basic data qual­i­ty requirements

As far as the sub­scrib­ing appli­ca­tion is concerned:

• Envoy sees either one zone per region or one zone per school and reg­is­ters as a sub­scriber in each of them
• Envoy makes requests for any of the SIF objects it needs in any of the zones and expects to receive all of the objects that per­tain to that region or school
• Envoy receives data on mul­ti­ple con­nec­tions con­cur­rent­ly, up to the lim­it imposed by the oper­at­ing system

The goal of this test is to see how long it will take to trans­fer the same amount of data between a mul­ti-zone pub­lish­er (in this test, Envoy) and a mul­ti-zone sub­scriber, com­par­ing it to the amount of time it took in the pre­vi­ous two examples.

In this dia­gram, the orange cir­cles rep­re­sent zones con­fig­ured on a sin­gle ZIS­erv­er. The AUCZ zone is the con­sol­i­dat­ed zone where Envoy receives its data from the orig­i­nal test provider (data for all 100 schools). The 100 schools were assigned ran­dom­ly to the 16 zones.

Change in data set size (increase)

Because of the way the stu­dent pop­u­la­tion was set up, some of the records might need to be dupli­cat­ed because of the way the zones were split. Nor­mal­ly, there would be a log­i­cal group­ing (such as group­ing them by LEA) used to split records, so that most of (or all of) the stu­dents would nev­er have their records split between two zones. When a stu­dent is enrolled in mul­ti­ple loca­tions, the Stu­dent­Per­son­al record will need to be pub­lished into both zones, where the Stu­dentSchoolEn­roll­ment record will only be pub­lished into the appro­pri­ate zone (the one that is indi­cat­ed in the SchoolIn­foRe­fId ele­ment inside the Stu­dentSchoolEn­roll­ment object). What this means, for this test and the next, is that Envoy will need to pub­lish an extra 20,000 or so "redun­dant" Stu­dent­Per­son­al objects, so that every zone will be able to retain its inde­pen­dence and so that there will be no Stu­dentSchoolEn­roll­ment records with­out match­ing Stu­dent­Per­son­al records in the same zone.

The Test Activity

This screen cap­ture shows the activ­i­ty lev­el of the Envoy serv­er (CHICO) (the provider of the data to the 16 zones) dur­ing this test. The lev­el of activ­i­ty stayed fair­ly con­stant through­out the test.

Notice that the CPU usage was rel­a­tive­ly small (11%), but that the ZIA­gent process (the engine for Envoy) is run­ning 142 threads at the time the screen shot was tak­en. The num­ber of threads var­ied through­out the test but remained fair­ly large. Envoy served as a proxy provider for the orig­i­nal con­sol­i­dat­ed provider to each of the 16 zones. The fol­low­ing screen shot illus­trates the activ­i­ty on Envoy's SQL Serv­er (you can click on the image to see a large pic­ture). As typ­i­cal with a SQL Server's activ­i­ty, it var­ied wide­ly in terms of the num­ber of batch trans­ac­tions / sec­ond (when this cap­ture was tak­en we saw 1,495 / sec­ond). The four graphs at the top of the screen, how­ev­er, shows an aver­age over a longer peri­od of time.

As this screen illus­trates, the sin­gle data­base (which was being mir­rored by anoth­er SQL Serv­er), has plen­ty of pow­er for this test.

Test 4 — Zone splitting using 100 zones 

In this test, we knew that a sin­gle sub­scrib­ing machine would not have enough pow­er to han­dle all 100 zones, so we split the load between sev­er­al machines that would all oper­ate as sub­scribers and would all be con­nect­ed to the same LAN as the oth­er machines. This test was sim­i­lar to the pre­vi­ous test, except that unbound­ed request mes­sages were sent from each of the 100 school zones to the Envoy provider all at the same time.

The fol­low­ing screen was tak­en to show the ZIS audit log (or part of it) — (if you click on the image you will see a larg­er pic­ture). As you can see in the logs, mes­sages are being dis­trib­uted to sub­scrib­ing agents in the var­i­ous school zones. These mes­sages were Respons­es to "unbound­ed" SIF_Requests that asked for all objects for the giv­en school. Ear­li­er requests has asked for all Stu­dent­Per­son­al objects.

Differences in server loads between Tests 3 and 4

As when com­par­ing the first two tests, we saw some marked dif­fer­ences in serv­er loads between these two tests as well.

The Provider IIS Server (Envoy Server)

• This machine was more com­pute-bound (many more threads exe­cut­ing concurrently)
• Much more mem­o­ry was required by the addi­tion­al zones (new threads are required to han­dle each new request received in each of the new zones)
• More net­work I/O was gen­er­at­ed, but still a rel­a­tive­ly small amount com­pared to gen­er­at­ing events

The ZIServer

• As with the pre­vi­ous two tests, the amount of traf­fic sent was not significant
• No mes­sages were queued at the ZIS at any point dur­ing the test

The Subscriber IIS Server

• These servers need­ed to accept, parse and process the received mes­sages, so they need­ed mul­ti­ple threads and a sig­nif­i­cant amount of mem­o­ry, so that each thread could process an incom­ing mes­sage concurrently
• We solved this prob­lem by using sev­er­al servers, but it could have also been han­dled by one (or few­er) machines with more memory

The Application Database Server

• This serv­er remained very busy through­out the time peri­od, much more than with the oth­er three tests (the same num­ber of SELECTs and INSERTs would need to be done, but in a short­er peri­od of time)

The fol­low­ing screen cap­ture shows the appli­ca­tion data­base server's per­for­mance mon­i­tor. Rel­a­tive to the oth­er mon­i­tor shots, it shows most of its mem­o­ry being used and a much high­er lev­el of activity.

Test 5 — Using a Single Zone with multiple subscribers, one for each object

This archi­tec­ture, although very unusu­al, is legal under all cur­rent SIF specifications.

It uses a sin­gle provider (as in the first test), but has mul­ti­ple sub­scribers that each sub­scribe to a sin­gle SIF object. In our test, we used the same four SIF objects and the same pop­u­la­tion sizes as in the oth­er tests:

• LEAIn­fo
• SchoolIn­fo
• Stu­dent­Per­son­al
• Stu­dentSchoolEn­roll­ment

In this test, we cre­at­ed a script that would begin pub­lish­ing all four objects (rough­ly) at the same time into the same zone. After the first few sec­onds, the LEAIn­fo and SchoolIn­fo records made it to their respec­tive agents. Then, the test con­tin­ued for the next sev­er­al hours for the oth­er two objects. This screen shot of the ZIS­erv­er (Zone Inte­gra­tion Serv­er) audit log screen was tak­en a few min­utes into the test:

The fol­low­ing screen cap­ture was tak­en to illus­trate that, even though a con­sid­er­able amount of traf­fic was mov­ing through the net­work, the bot­tle­neck was still clear­ly not on either the agent machines, the data­base machine or the ZIS­erv­er machine. Each of the servers shared a small queue dur­ing the test — as we looked more close­ly at detailed traces we saw small waits dis­trib­uted in many places through­out the sys­tem in places where we would have expect­ed them to be (sim­i­lar results to test 1 above).