• 532 Durham Rd., Newtown PA USA
  • Info@visualsi.com
  • Office Hours: 8:00 AM – 5:30PM (Eastern US)

What It Does

XML data exchange specifications (such as SIF for the education market) enable applications to exchange data, so that once it is made available in one place, it can be put to good use everywhere else as well. In the process, valuable information from all sorts of applications crosses the interoperability “zone.” We recognize this flow as an asset in and of itself, and we tap into it with Veracity.

The Repository and Data Quality Management

Veracity is our unique repository of exchanged, standardized objects. It leverages the infrastructure’s event and request/response facilities to keep an up-to-date snapshot of the current status of all objects being exchanged. Once in Veracity, it doesn’t matter whether the data came from a transportation system or a food services system, it is available from one place under one data schema.

Can you trust your data? Is it accurate and dependable?

When it comes to making decisions based on your district’s data, one of the biggest challenges to overcome is that of data integrity. Can you trust your data? Is it accurate and dependable?

One of the fundamental goals in data integrity management is to move the correction of data errors back to the source. In an education environment, for example, when census data is being prepared at a school district, the IT staff collects the data from the various systems in the school, matches up the records, cleans them, formats the result, and then submits it. This cycle is repeated several times during the school year. Many times the same error is corrected over and over because it gets fixed in the CSV file instead of the source system.

This is why we have created Veracity. The Veracity approach is the following:

  • Establish a common repository that always reflects the current state of the data in the source applications (Veracity).
  • Monitor the quality of the data by applying a set of rules and looking for problem areas.
  • Notify the owner of the data in question so that it can be corrected in the source application.
  • Use the data from the repository as the source for state reports, cross-application reports or as a feed into a data warehousing system.

By moving the correction of the problem back to the source of the error:dquality 200x187 1 - Veracity - Consolidation Agent and Integrity Manager

  • All applications in the zone benefit from the better-quality data.
  • The data is changed only once and in one place.
  • Your data becomes more reliable, giving you more confidence in decisions you make that results from the data’s analysis.

See if you can rely on your data. The first thing to know is how close the quality of your data is to the standards defined by your organization.

Veracity’s rule set articulates your district’s data quality rules, applies them against data collected from the connected systems and shows you where your district’s data falls short.

These rules can be checked on a scheduled basis or in real time as data is published.

Viewing Errors in Data (Education Example)

HomePage - Veracity - Consolidation Agent and Integrity Manager

See How You Are Progressing

This chart shows what types of errors you have and how they are distributed. If you click on the chart and hover over one of the slices, it tells you how many errors of that type there are. Clicking on the image will cause a full size image to open in a new window.

Veracity requires that errors be corrected at their source, and it does not allow changes to be made in the repository.

Instead, it provides a set of tools that help people track errors and correct them at the source, so that all other systems connected through the SIF interface can also benefit when they are corrected.

The graph to the right gives users a sense of how the district’s data is improving over time. Clicking on the image will cause a full size image to open in a new window.

The different colors represent errors, warnings and other items of interest. If you click on this graph, it scrolls from side to side and, like the other graph, if you click and hover over a bar, it shows you the specific number of errors in a specific category.

In this example, the data was checked several times. As the data was checked, errors were being corrected at the source until the last sample, where some new rules were introduced. These new rules caught additional errors in the source data.

ErrorOverview - Veracity - Consolidation Agent and Integrity Manager

Seeing Error Details

Seeing the details is also important. This screen allows you to view and drill down into the error details. This screen shows a summary of each rule that has been violated by data in your organization as well as how many records violate that rule. Clicking on the “View Records” link allows you to see student-level detail for a specific category.

ErrorPickList - Veracity - Consolidation Agent and Integrity Manager

Seeing Student Details

Clicking on the “View Detail” link allows you to drill down to the student detail level.

ErrorDetail - Veracity - Consolidation Agent and Integrity Manager

 Viewing Student Detail Record

This student’s record violated rules: the birth date/school reasonableness and the school district age limit test rule. At the bottom of the screen, both rules are summarized. The specific wording for each of these rules can be modified by the Veracity administrator through this web interface to word them according to local policy definitions. NOTE: Student information used in these screen captures are for fictional student records randomly created from lists of popular last and first names. Any resemblance to real persons, living or dead, is purely coincidental.

At the bottom of this screen (not shown) would be a summary of other rules that have also been violated by this record (there could be several problems with this record).

Defining Rules

Veracity helps you continuously monitor the quality of your organization’s data by giving you an ability to simply define data integrity rules that will be run against the consolidated data in Veracity and point out to your staff where data needs to be corrected.  It provides a simple configuration form to define rules such as “make sure every elementary school child who lives more than a half mile from school is on a bus route” or “make sure that every child has been assigned to at least one but not more than two schools”.

Sets of rules can be established for different purposes.  For example, you may want to run general student information checks on a daily basis while other, more detailed checks may only be appropriate once the “beginning of the school year crunch” has passed.

Rules are defined through a web interface (assuming that the user has administrative rights) using a process that typically takes only a few minutes per rule.

Data Quality Gateway

Overview

When an application’s input process is automated, and it begins to receive its input from a new source (not manually through its own , the subscribing application may start to exhibit “new behaviors.”

We found this out from our work with the special education community in the US. They began to receive data from an associated student management system, but typically, the data they already had was better quality than the new data they were receiving from the . They referred to the new automated data as “data pollution.” Situations like this occur wherever input is automated, but its effects can be mitigated. The way we addressed this challenge was through something we call a “Data Quality Gateway.”

Clean Zones

Veracity sets up two databases to store data collected from the schools, referred to as the “raw zone” and the “clean zone”. The raw zone stores data in the form received from the schools. The clean zone contains only data that has been tested and judged to be of good enough quality to be shared with other applications.

This is how it works: as the data arrives, Veracity checks it for validity against data quality policies defined by the ministry (implemented as business rules in Veracity). Data that passes all of the quality checks is immediately copied to the “clean zone” and becomes available for connected applications. Data that fails any of the tests is held in the “raw zone,” and the owner of the data is notified. A user interface is provided for that user to see where the errors are so that the errors can be corrected at their source. Once an error is corrected, the change comes through the SIF interface and the record is released to the clean zone, providing connected systems with up-to-date, usable information.

In the following diagram, the SMS applications represent the sources of information (Student Management Systems), and the Subscribing Application represents the place where an extraordinarily high level of quality is required (a much higher level than what is delivered by the SMS systems).

DQGateway - Veracity - Consolidation Agent and Integrity Manager

When a rule is set up at configuration time, there are several options available with respect to the gateway:

  • If the incoming message does not meet the requirements, notify the source of the data and do not pass the data to the subscribers.
  • If the incoming message does not meet the requirements, notify the source of the data and pass the data to the subscribers (warning only).
  • If a tested field causes the data to not meet requirements, notify the source of the data and do not pass the data to the subscribers.
  • If a tested field causes the data to not meet requirements, notify the source of the data and pass the record to the subscribers WITHOUT the tested field’s value.

The tests that are set up can involve any data in the incoming data record. Additionally, they can also test values in previously received data in this table or in other tables.

Veracity User Interface Appearance

Look and Feel

tblHTML - Veracity - Consolidation Agent and Integrity ManagerWe have found that with many customers, the functionality that Veracity provides belongs on the desktop of those who regularly enter data into line of business applications. Many of these users are already likely to be logged into an organization-wide portal that provides other services needed during the day.

For this reason, we gave Veracity the ability to adopt the look and feel of almost any other website with a few simple steps (performed by an administrator) in a matter of a few hours. The Veracity user interface has about 30 different parts (page heading, footer, top of a table, table cell, etc.) – the things you would find on most portal sites. As it is being configured, an administrator views the source of a portal page, copies the style from it into Veracity and the user interface is transformed without coding. This allows the Veracity interface to then be embedded as part of the organization’s portal and look like it was always meant to be there.

Veracity Consolidated Data Access - Reporting

Using the Combined Data

At Visual Software, we understand that a school’s business is not the numbers and strings passed around from application to application, but what those data elements represent. You have data that describes how a child is doing, from discipline to grades to attendance. The problem is that there is just too much data for any human to keep track of and use effectively at the student level.

You may already have systems that give detailed reports to principals, guidance counselors and other staff members. But what happens when you take one line from three different reports, only to find out that the combined lines signal a problem, while none of lines raise any red flags on their own? For example, a particular student’s GPA may be dropping at the same time as their attendance is worsening and their discipline incidents are increasing. Any one of these factors viewed independently might not get much attention, but put together, they might signal a child “dropping off the edge.” If this information was maintained by different systems or if the information was not available in a consolidated report, that important indicator might be lost.

Veracity collects and combines information from whichever applications are publishing in a zone. Records from these different systems for common students, teachers, parents, and classes (groups) are automatically matched, allowing users to access its underlying database to create consolidated reports using tools such as SQL Reporting and Analysis Services.

Veracity Form Designer

In a system like this, especially when we create new industry specifications, the need for many forms arises – forms to display errors, search (and edit) data, and show detailed records for each type of object. This is what consumes an overwhelming amount of time for most companies trying to imitate what we do. We knew that this  development would quickly make implementations like these unaffordable for those who need them most. So, we built a framework into Veracity to quickly add sets of forms with rich functionality for each new object we added to a new interoperability specification.

VeracityForms01 - Veracity - Consolidation Agent and Integrity Manager

The Form Designer

For each object that will be quality-checked in an interoperability specification (such as SIF, HL7, etc.), a set of forms is needed by Veracity to display potential errors and allow users to browse through the consolidated data (if so desired). These include forms to:

  • Display errors
  • Enter search criteria (configured by user as not all fields are searchable)
  • Show a list of search matches
  • Show search results with any errors
  • Edit records (optional)
  • Delete records (optional)
  • Manage attachments (optional)

The optional forms above are for other applications (not the data quality checking described above).

Once the forms have been created here, they are ready to be used in data quality rules.  They can now be added to the navigation, allowing users to browse through objects of this type.

VeracityForms02 - Veracity - Consolidation Agent and Integrity Manager

 The Object Search Screen

For objects where Veracity customers want to provide users search (and optionally update) access to the combined data, a search facility will automatically be added to the interface. This happens only after a form for the object has been defined, and the object has been added into the navigation structure.

Users can be restricted by scope (for example, US education users can be restricted by all statelevel data, school district level data or school level data only) or can be given access to all data. Searches can be specified by using natural language queries. For example, if the user is searching a date field, he or she could search using: “this year”, “ytd”, “1/15/2015-4/2/2016”, “4/2017” or “before May 2013”. We tried to anticipate what our users might type when searching and implemented it, including what hackers might type and prevented it from doing anything.

In this example, the user is searching for “Students” with the letters “jo” anywhere in their LastName. If multiple strings are entered, it will search as if an “OR” were between them. (Logical operators such as AND, OR and NOT can be used in searches.)

VeracityForms03 - Veracity - Consolidation Agent and Integrity Manager

 Search Results

This screen shows the results from the previous search. The fields shown here were specified in the Form Designer’s “Display in Search Results” field. For a given field, if the value was set, the field will show in this form.

At the beginning of each row is a link that allows the user to get to the details for this record.

VeracityForms04 - Veracity - Consolidation Agent and Integrity Manager

 Record Detail

This screen shows the detailed information for the selected student. NOTE: Student information used in these screen captures are for fictional student records randomly created from lists of popular last and first names. Any resemblance to real persons, living or dead, is purely coincidental.

At the bottom of this screen (not shown) would be a summary of rules that have been violated by this record (there could be several problems with this record).