Integrity Schools is Launched

Foremost, I apologize for the lack of new content lately. Many good things will hopefully result from the effort going into my "day job." We have officially launched Integrity Schools as a separate company to focus exclusively on SIF and business intelligence for schools. The full press release follows:

Instructional Services Objects

The SIF Association has been placing a needed emphasis on the technology's use within teaching and learning. As of version 2.2 of the SIF Implementation Specification, there are a total of (8) objects that directly address core needs of teaching and learning processes. The Instructional Services Task Force (ISTF) is the group within SIF that is moving this work forward.

The diagram below of the ISTF objects contains more detail than the general Data Model ERD on which I continue to work. Almost all of the elements and attributes of the Instructional Services objects are present in this diagram, with the exception of a few that I omitted for the sake of clarity. Here is the link to the ISTF objects within the Implementation Specification.

One other item worth noting is that I am hoping to use these objects, specifically the LearningStandardItem object, on a client project to rigorously document their learning standards for use within multiple future projects. This is a good example where the SIF Data Model has significant intrinsic value, even without a specific need to "move" data using the Infrastructure.

Lastly, the SIF Association is holding a developer's camp for teaching and learning in Portland on Thursday, September 25 and Friday, September 26. For more information, see this page on the SIF website.

New Reports Show SIF Object Usage Across Certified Applications

The Certification Register at http://certification.sifinfo.org is the authoritative source of information about the capabilities of SIF Certified Applications and Agents. The site has recently been enhanced in two ways.
It is now possible to view a Data Model report that details and summarizes the use of SIF Data Objects by Certified Agents. This enables you to get a sense of which objects are most widely supported, and in what ways.
Located underneath this information (on the same page) is a matrix that represents the intersection of two applications. Hovering over a cell reveals the ways that the two systems may interoperate, at a Data Object level.
In combination, these reports provide end-users and developers with much information about how SIF is being used, and how it may be used. This is the direct link to the Data Model report.

STATS-DC 2008

Next week I will be attending and presenting at the STATS-DC 2008 data conference in Bethesda, MD. Both presentations that I am a part of are on the SIF track of the conference. The National Education Data Model will be the focus of one of the presentations. My role will be to discuss how we have used it in one of our projects at work. The second presentation will cover what the SIF Association has done (and is doing) about collecting and reporting race and ethnicity data. If you are coming to the conference, please attend these sessions!

SIF Message Objects

SIF's Infrastructure defines a set of Message Objects (synonyms in other vocabularies include stickers and envelopes) that underpin its asynchronous data exchange capabilities. SIF Agents wrap commands and Data Objects in appropriate Message Objects to accomplish interoperability functions. Zone Integration Servers lightly parse the Message Objects to control the routing of data from Agent to Agent. SIF Data Objects are the subject of the Entity Relationship Diagram previously discussed here. To reprise: Data Objects are the implementation of SIF's data model for interoperability (students, staff, enrollment, etc.). Data Objects also happen to be the essential unit of data exchange in SIF. The following diagram illustrates an overview of SIF Message Objects (click the image to view it standalone).

SIF and Open Source

Outside observers of SIF often make the comment that the SIF Association (SIFA) should implement an Open Source Zone Integration Server, or an Open Source Agent development tool set. I generally respond to these comments by analogizing SIF with other standards bodies, like the W3C. Similarly to the W3C, the primary output of SIFA is its standards.

Take as an example the W3C's XHTML specification. Under the auspices of the W3C, member organizations collaborate to draft, test, refine, and release the detailed technical definition of the XHTML technology. Acting as individual organizations, these members implement software that leverage, for commercial and non-commercial purposes, the specification developed by the standards body. It would be, however, very unlikely for the W3C to actually develop a web browser or web server that implemented XHTML in software.

Many standards bodies, like SIFA and the W3C, publish their works openly. Any organization or individual, including Open Source projects, may benefit by deciphering, adopting, and implementing the standards, regardless of membership status, and without paying licensing fees to the standards body. Other standards bodies, in contrast, choose to sell their work using fee-based licenses. Although SIFA is not an Open Source initiative, the open publication of its standards is certainly an enabler for industrious Open Source developers.

SIFA, its members, and the wider world of educational technology would greatly benefit from additional documentation to fill the void between the SIF specifications and their implementation in software. To that end, one of the purposes of this blog is to share some of my practical experience with SIF. I would like to take this opportunity to encourage SIFA to work towards providing more developer-centric documentation, including code samples, as part of its work. Since SIF's core technologies (XML and HTTP) are ubiquitous, basic code samples could be developed for every modern programming language and platform. The results of such work would be clarity for a wider audience of developers in the school market, the demystification of SIF, and, ultimately, wider adoption.

I would also encourage individuals and groups developing Open Source SIF software to become involved in the Association. Involvement will greatly accelerate your efforts in building good software that solves schools’ interoperability and data management needs.

The "Moveable Feast"

The "Moveable Feast" is an Illinois technology conference that gathers school personnel from across the state to share information about a diverse set of education technologies. It is the longest running education technology training program in Illinois. The "Feast" started over 10 years ago with the objective of providing relevant training to teachers, technologists, and administrators. Approximately 10,000 people have benefited from the program during its run. "Feast" collaborators include the University of Illinois College of Education, District 87 and several other Illinois school districts, and corporate sponsors like Microsoft and Discovery Education.

The "Geek Edition" of the "Feast" was held this week in Bloomington. Sessions covered topics ranging from content management, to software development, to SIF (of course!), and business intelligence. Another "Geek Edition" of the "Feast" will be held in Belleville in July. Following is a sampling of the sessions: Schools Interoperability Framework*, Moodle I & II, Reporting and Business Intelligence*, Web Design & Scripting with PHP/AJAX, Beginning and Advanced Virtualization, Data Warehousing*, Joomla, and SharePoint

* Indicates one of my sessions

My thanks to Jim Peterson and Brian Adams for inviting Integrity to participate in the "Feast!"

Gradebook Objects Added to SIF ERD

The SIF Data Model Entity Relationship Diagram has been updated to include the Grade Book Data Objects (thanks to Andy Elmhorst for the suggestion).

When Good XML Goes Bad

XML.com recently published an article titled "Bad XML." It is a good read for those of you that are interested in markup design. In SIF I think we do a fairly good job of not creating "Bad XML," but there is definitely room for improvement. Here is the link to the article.

SIF ERD Updated with Assessments

The SIF Data Model Entity Relationship Diagram has been updated to include the Assessment Data Objects. Again, please feel free to nominate what you would like to prioritize for inclusion in the diagram as I continue to update it.

SIF Data Model ERD - Updated

The SIF Data Model Entity Relationship Diagram that I recently published as a Google Presentation has been updated to include two new areas: Special Programs and HR Finance. Assessments are likely to be next. Please feel free to nominate what you would like to prioritize. Enjoy!

SIF, SOA, and WOA

The following article talks about the successes of Web Oriented Architecture (WOA) and the challenges of Service Oriented Architecture (SOA). It is particulary interesting as SIFA embarks upon enriching its current messaging platform with new approaches to services.
http://blogs.zdnet.com/Hinchcliffe/?p=168

SIF Data Model Entity Relationship Diagram

I've been developing a high level Entity Relationship Diagram for the SIF Data Model. The main goal is to help people more easily see and understand the relationships among SIF objects. Here is the link to a Google Presentation that contains the diagram for the Student Information Systems (SIS) objects. My long term goal is to make this more interactive, but hopefully it is useful to some people in its current state. Please note two things: 1) this is a small subset of the overall data model (I will be adding more areas soon) and 2) the level of detail in the ERD only reflects objects and their primary and foreign keys. Please see the SIF Implementation Specification for full details.

Using the SIF Data Model in Software for Schools

Introduction

The SIF Data Model, which consists of the XML data “objects” that make up the bulk of the specification, has grown and evolved from the expertise that application vendors contribute to the SIF Association. Traditionally, Working Groups within the SIF Association have aligned on particular "vertical" areas of focus within the K-12 software market. Examples of Working Groups within the SIF Association include Student Information Systems, Human Resources and Financials, Transportation, and Food Services. Using a community-based process, members of these working groups collaborate to build their respective areas of the SIF Data Model. In summary, the SIF Data Model has evolved from the bottom-up versus being specified from the top-down.

It is probably useful to first understand what the SIF Data Model is, and what it is not. Foremost, the SIF Data Model was developed around interoperability business and use cases. That is to say, it was never intended to be an optimized, normalized application-level schema. It is intended to facilitate the exchange of data between two or more applications connected to a common message bus. As a result of this, and the fact that a community-based process, rather than a single mind or focused team, produced the data model, you will notice varying consistency within the SIF Data Model. The purpose of this writing is to provide some thoughts on how best to use the SIF Data Model when building new, or enhancing existing, K-12 software products.

Logical Structure of the SIF Data Model

Reasons for Using the SIF Data Model in Your Application

If you are building or enhancing a software application for K-12 schools you may want to take a close look at the SIF Data Model for a few reasons, even if you do not immediately plan to build a SIF Agent.

1. Ensuring Completeness The first, and probably most obvious comparison point between your application's data model and SIF, should be for completeness. When considering the full SIF Data Model, including all of its objects, elements and attributes you will notice that it has become, over time, fairly rich. This is especially true for the core objects that are most often used in the field (e.g. StudentPersonal, StaffPersonal, SchoolInfo, StudentSchoolEnrollment, StudentContact).

2. Aligning to SIF for Interoperability Market demand from schools (SEA's and LEA's) is driving SIF adoption. However, even if your application implements a data model that is an exact copy of the SIF Data Model, you do not have an interoperable application. Interoperability requires the ability to interact with other applications over the SIF Infrastructure, according to the rules defined in the SIF Implementation Specification. Implementing an application data model that resembles the SIF Data Model, or that is easily mapped to the SIF Data Model, definitely puts you a step closer to interoperability via SIF.

3. Relating to Other Software Systems By using the information that is available in the SIF Certification Register, you can analyze the capabilities of other applications from which you wish to obtain data today, and to which you may wish to publish data. Depending on your target market, you may wish to analyze specific applications or broad categories of applications to determine how your system may interoperate with other systems. I also highly recommend talking to your customers, or potential customers, about their specific needs when it comes to interoperability. Most technically savvy people within schools have at least heard of SIF, and many will have excellent ideas to share about how they wish their systems to work together.

4. Decreasing the Level of Effort to Support Data Exchange The vision behind SIF is to make data exchange easier for both the end-users in schools and software vendors. While it is true there can be a relatively high learning curve in the beginning to implement SIF, over time the benefits of automated processes become evident. Depending on how you support your customers' data import/export needs, there is great potential for you as a software vendor to reduce the overall level of effort in managing data exchange processes.

Example Intra-Zone and Extra-Zone Integration Architectures

One of the books I often recommend to people who are new to message based integration solutions is "Enterprise Integration Patterns" by Hohpe and Woolf. Visio stencils that contain shapes for the various patterns and pattern components can be downloaded from the book's website. The following diagram (click it to view a larger version) tries to do a few things...

First, it states SIF architecture in terms of Enterprise Integration Patterns (EIP). The second thing it does is illustrate one possible way to implement extra-zone services using EIP concepts. The third thing it does is to begin calling out the business and use cases for intra-zone versus extra-zone services, which are duplicated in the text below.

Intra-Zone Business and Use cases
1. Applications wish to exchange data on a regular basis
2. Applications are contained within a single enterprise, or closely related enterprises
3. Guaranteed delivery is desired or required
4. Central administration of message flow is desired or required
5. A collection of applications wish to share the same data

Extra-Zone Business and Use cases
1. Applications wish to exchange data on an infrequent basis
2. Applications have minimal or no organizational ties
3. Guaranteed delivery is not required
4. Central administration is not required
5. Two applications needs to share specific data

Pennsylvania Schools Interested in SIF

I attended the PETE&C conference in Hershey, PA on Sunday and Monday. It was encouraging to see a good turnout from both small and large districts at the pre-conference sessions on SIF. We were invited to attend by the Pennsylvania Department of Education and graciously hosted by our friends at the Central Susquehanna Intermediate Unit (CSIU). The SIF Association and several other SIFA members were also present. Thanks to everyone who attended! Here is the link to the slides that I presented. They contain information about my involvement in SIFA, information about Integrity, and examples of integrations that we have done.

A High Level Overview of SIF Services

The SIFA Infrastructure group is currently working on adding service-oriented messaging to SIF. This work is currently being labeled "SIF Services." It should not be confused with web services and the attendant set of WS-* standards. The Technical Board has had some recent discussions around what services ARE and ARE NOT. Following is a summary that I have put together that will hopefully bring additional clarity for all.

SIF Services will:

• Be built on top of the SIF Infrastructure
• Require a Zone Integration Server
• Be based on message exchange integration patterns

SIF Services will not:

• Be dependant on SOAP
• Directly interoperate with SOAP-based web services
• Be based on RPC integration patterns

SIF Infrastructure will:

• Add support for SOAP as a transport
  
• Be required for SIF Services

Building services on the existing SIF Infrastructure has many advantages. However, there are many business and use cases that have been put forward by the community that are not necessarily well addressed only by the current SIF Services proposal. Following are the main areas of functionality that need further discussion:

• Teaching and learning
• Exchange of large data payloads (e.g. multimedia, SCORM SCO's)
  

• General interoperability
• Point-to-point interoperability between applications
• Expanding interoperability to new applications/vendors, especially those that are not "native" to K12
  

More discussion will take place at the SIFA Annual meeting in Washington, D.C. in January and in planned discussions about "services" within SIF.

SIFA and ADL Partner

Interoperable Learning Content Vision
Consider all of the educational content that society has developed. Courses on world religions, lesson plans for high school geometry, corporate training materials on risk management, military training programs on surface to air missile operations. An immense body of educational content is "out there." Yet what untold and countless number of hours are spent every day recreating educational content that already exists?
The vision of interopable learning content is to provide both visibility and accessibility to the widest possible body of digital instructional materials. Interopable learning content technologies allow teachers to systematically find, obtain,license (if necessary), customize, and incorporate relevant materials into instruction. Fulfilling this vision is one of the missions of the SIFA-ADL partnership.
SIFA Partners with ADL on Interoperable Learning Content
The SIF Association is holding regional meetings in the United States to discuss some of the details of its partnership with ADL (Advanced Distributed Learning; http://www.adlnet.gov). The focus of the partnership between the two organizations is on interoperable learning content. I attended the meeting hosted by Chicago Public Schools on November 8, 2007. Organizations that had a presence at the meeting included Chicago Public Schools, Pearson, Follett, Integrity Technology Solutions, Educational Systemics and Plato Learning. Jill Abbott, SIFA's Learning Strategist, and Paul Jesukiewicz, ADL's Deputy Director, lead the meeting. This article is a brief summary of what was presented at the meeting.
About ADL
ADL is an initiative sponsored by the US Department of Defense. The original concept behind ADL was to create a standard to facilitate the sharing of learning content for the US government. The result of this work has since moved into the mainstream and been adopted by higher education, industry, and international concerns.
ADL's Work Product: SCORM
SCORM is the standard, or reference model, defined by ADL that specifies how learning objects may be packaged and exchanged. The scope of an individual learning object may range from all of the content needed for a semester long course, down to a single course exercise. A SCORM shareable content object consists of two parts: a manifest and instructional content payload.
The manifest contains metadata about the instructional content payload that tags the content and describes its organization. The manifest can also indicate sequence and branching among members of the content payload. For example, based on the results of a student's formative assessment, a learning management system could potentially select a different sequence through the content. The sequencing component of the manifest facilitates dynamic adaptation to an individual student's needs.
Learning management systems are the primary consumers of shareable content objects. SCORM compliant content can come from a variety of sources ranging from content publishers to individual teachers. Various tools exist to enable the development of shareable content objects. The authoring subsystems of learning management systems can generally export shareable content objects. Standalone tools exist that allow authors to package content developed in other formats (e.g. office products, HTML) into shareable content objects.
Direction of ADL/SCORM
One of ADL's stated objectives is to divest itself of the stewardship and future development of SCORM. A new body called LETSI (Learning-Education-Training Systems Interoperability) will take this over. LETSI will be an international, lightweight organization that will govern SCORM development going forward. LETSI will build on the current version of SCORM to develop a common framework, known as Core SCORM, that will be extensible to meet the needs of diverse industries and groups. Core SCORM will define the fundamentals of shareable content objects, most likely including packaging, basic metadata, and sequencing.
Status of the Partnership
The SIFA-ADL partnership has been established, but the work of integrating SIF and SCORM is in its formative stages. The two organizations are soliciting use cases from vendors and end users, with an emphasis on how schools want to use the technology. This communities of practice pattern will be intrinsic to the LETSI/Core SCORM development process. Like SIFA and its contingent of school-affiliated organizations, other organizations, industries and groups will build on Core SCORM to meet community-specific use cases.
SIF and Educational Content
SIF is traditionally known as a way to model and move administrative and demographic data among operational information systems in schools. Those involved closely with SIF (including myself) are convinced of its overall, positive impact on school operations and data, including data quality and availability. With a widely implemented transport infrastructure and data model, SIFA is building capacity that will more directly impact teaching and learning within schools. The alliance with ADL will result in the eventual integration of Core SCORM and SIF, which will advance SIFA's teaching and learning initiatives and further establish SCORM as a global platform for interoperable learning content.
Impact on Online Learning and Personalized Learning
Although compelling from the standpoint of sharing and efficiency, the vision stated at the outset of this article does not give full justice to the potential of interoperable learning content. Online learning and personalized learning are two major beneficiaries of interoperable learning content.
Online learning continues to proliferate. Students are clearly expressing the preference to learn via the technology that is already intrinsic to their lives. In addition to becoming the learning method of choice for students, virtual schools fill critical gaps in traditional educational systems. Virtual schools can help students maximize their potential by serving as an alternative form of instruction. Interoperable learning content has the potential to remove the burden of content development from virtual schools by providing access to a world-wide network of instructional material.
Personalized learning takes the benefits of online learning to a new level. By implementing a continuous feedback loop between teaching and learning, custom "virtual" curricula can be developed to dynamically address the needs every student as an individual. Each student can learn to her or his maximum potential.

The Data Warehousing Applications of SIF

In SIF, the concept of interoperability has traditionally been applied to applications sharing data with each other. Each application in a Zone has the ability to publish data for which it is authoritative. Other applications in the Zone subscribe to that data as needed. For example, some student information systems subscribe to food services and transportation data. Almost all SIF-enabled applications have the ability to subscribe to the student demographic data supplied by the SIS.

As data warehouse solutions become more prevalent in schools it is worth discussing how they can interoperate with SIF. One of the most labor intensive components involved in implementing and supporting a data warehouse are the extract/transform/load (ETL) processes. ETL processes are responsible for mining data from operational systems (the “E”), modifying and scrubbing the incoming relational data to fit a dimensional model (the “T”), and then placing that data into the dimensional model or a staging area (the “L”). Whether the data warehouse has been built from scratch using industry standard tools or it is a commercial product, significant challenges lie in getting intimate with the native data structures of the operational systems. Over time, operational systems schemas change. This can add a potentially significant support burden to the data warehouse, just for the ETL component!

SIF has the potential to ease the ETL burden for data warehouse initiatives. First, SIF provides a standard data model that most leading vendors in the K12 space have adopted. In other words, the SIF Agents for those systems conform to a standard definition for data groups like student, staff, assessment, special programs, and finance. The SIF data model can thus be leveraged for ETL staging. Just be aware that SIF does not (yet?) provide any standards for dimensional data models.

Second, SIF provides a standard transport method for mining data out of the operational systems using XML over HTTP/S. Traditional approaches to ETL involve using ODBC, JDBC, file dumps, and other one-off approaches for accessing operational systems’ data. SIF’s Request/Response and Event models provide a much simpler way to load and maintain synchronization of operational data in a data warehousing environment. A “universal subscriber” Agent that stores incoming SIF data in its native XML representation or that “shreds” the XML into a relational staging area can be implemented for this purpose. The major advantage of the “universal subscriber” approach is that it essentially implements one interface to a Zone, versus implementing an interface per operational system as would be necessary with traditional ETL tools.

Another caveat that must be addressed if you consider using SIF for ETL into a data warehouse is data quality. SIF provides a powerful, open mechanism for moving data. However, one must never forget the GIGO principal (garbage in, garbage out). As applied to SIF, GIGO says that if bad data is entered into an operational system, then bad data will be in ALL subscribing operational systems AND the data warehouse. How can this challenge be addressed?

I believe it starts with business processes. Organizational standards for data entry must first be defined. The people responsible for data entry must then be trained on those processes. An even better approach is to have them participate in development of the standards in the first place. There must also be systematic checks of data before it is loaded into its final destination, the dimensional model. The data quality checks can be implemented at various points in the process. The previously mentioned “universal subscriber” could be made responsible for some of the necessary data validation. The process that loads the staged data into the dimensional model would also likely share in the responsibility of checking data quality.

SIF can be a great tool for inter-application data sharing. It can also play an important role in data warehousing initiatives by streamlining and simplifying ETL via a standard data model and data transport.

What's New in SIF 2?

The SIF 2 Implementation Specification has been available for about a year. Quite honestly, general adoption of SIF 2 has lagged in the market due to various reasons that I won't go into at this time. SIF 1.5r1 is still the most widely implemented version of the specification. It's worth noting that SIF 2 offers some compelling new capabilities. Following is a summary of some of the new capabilities in SIF 2 (this information comes from the specification and from my own experience):

• Increased use of other standards, including XML schema and namespaces
  
• Addition of SIF_Extended query to support more dynamic query and reporting
  
• Many new data objects across the specification
  
• Student Record Exchange objects provide new capabilities for data exchange between K12 and higher education, and for many other applications
• Significant enhancements to assessment objects
• Various infrastructure improvements to provide for "smarter" agents
• Implementation of contexts as a way to sub-divide zones
• Implementation of metadata; this will become increasingly more important as SIF gains momentum in teaching and learning technologies
  
• For developers with existing "responder" agents it is also important to note that SIF 2 now requires that you support querying by all mandatory elements, not just the root elements
  

Should you wait for SIF 2?
Emphatically: NO! You should not wait for wider adoption of SIF 2 prior to implementing.
If you are a school looking to implement then it's likely that most of your applications support 1.5r1 today, and that the vendor companies should have a roadmap to SIF 2. Going forward today will allow you to get the benefits that SIF offers, and it will allow you to gain experience that you can apply to SIF 2 implementations when the time is right. Just make sure that your ZIS will support SIF 2!
If you are a software vendor looking to implement a SIF agent the answer on waiting is also "no, don't wait." However, you should plan on implementing support for both 1.5r1 and 2 within your agent. Depending on how you implement the Agent, the level of effort to do "double duty" is definitely not double; however, it's also not necessarily trivial.
Another item of note is that SIFA is striving to release specifications more rapidly to better accommodate end user needs. As of the time of this post, SIF 2.1 is the current release, 2.2 is in review, and work is being done on SIF 2.3+. Rest assured that the development process that SIFA follows ensures backward compatibility within the 2.x series.
Please make sure to visit the specification site if you are interested in all of the details.