WinFS

Architecture of the WinFS Stack

WinFS uses a relational engine, which derives from SQL Server 2005,[25] to provide the data-relations mechanism. WinFS stores are simply SQL Server database (.MDF) files with the FILESTREAM attribute set.[26] These files are stored in the access-restricted folder named "System Volume Information" (placed in the volume root),[27] in folders under the folder "WinFS" with names of GUIDs of these stores.[26]

At the bottom of the WinFS stack lies WinFS Core, which interacts with the filesystem and provides file-access and -addressing capabilities.[8] The relational engine leverages the WinFS core services to present a structured store and other services such as locking, which the WinFS runtime uses to implement the functionality. The WinFS runtime exposes services such as Synchronization and Rules that can be used to synchronize WinFS stores or perform certain actions on the occurrence of certain events.[8]

WinFS runs as a service that runs three processes:[28]

1. WinFS.exe, which hosts relational datastore
2. WinFSSearch.exe, which hosts the indexing and querying engine
3. WinFPM.exe (WinFS File Promotion Manager), which interfaces with the underlying file-system

It allows programmatic access to its features via a set of .NET Framework APIs. These enable applications to define custom-made data types, define relationships among data, store and retrieve information, and allow advanced searches.[6][8] The applications can then aggregate the data and present the aggregated data to the user.

WinFS stores data in relational stores, which are exposed as virtual locations called stores.[12] A WinFS store is a common repository where any application can store data along with its metadata, relationships and schema. WinFS runtime can apply certain relationships itself; for example, if the values of the subject property of a picture and the name property of a contact are same, then WinFS can relate the contact with the picture.[29] Relations can also be specified by other applications or the user.[30]

WinFS provides unified storage, but stops short of defining the format that is to be stored in the data stores. Instead it supports writing data in application-specific formats. But applications must provide a schema that defines how the file format should be interpreted.[6] For example, a schema could be added to allow WinFS to understand how to read and thus be able to search and analyze, (say) a PDF file. By using the schema, any application can read data from any other application, and this also allows different applications to write in each other's format by sharing the schema.[30]

Multiple WinFS stores can be created on a single machine.[30] This allows different classes of data to be kept segregated; for example, official documents and personal documents can be kept in different stores. WinFS, by default, provides only one store, named "DefaultStore".[12] WinFS stores are exposed as shell objects, akin to Virtual folders, which dynamically generate a list of all items present in the store and present them in a folder view. The shell object also allows searching information in the datastore.[12]

A data unit that has to be stored in a WinFS store is called a WinFS Item.[6][30] A WinFS item, along with the core data item, also contains information on how the data item is related to other data. This Relationship is stored in terms of logical links. Links specify which other data items the current item is related with. Put in other words, links specify the relationship of the data with other data items. Links are physically stored using a link identifier, which specifies the name and intent of the relationship, such as type of or consists of.[6] The link identifier is stored as an attribute of the data item. All the objects that have the same link ID are considered to be related.[6] An XML schema, defining the structure of the data items that will be stored in WinFS, must be supplied to the WinFS runtime beforehand.[6] In Beta 1 of WinFS, the schema assembly had to be added to the GAC before it could be used.

WinFS models data using the data items, along with their relationships, extensions and rules governing its usage.[8] WinFS needs to understand the type and structure of the data items, so that the information stored in the data item can be made available to any application that requests it. This is done by the use of schemas. For every type of data item that is to be stored in WinFS, a corresponding schema needs to be provided to define the type, structure and associations of the data. These schemas are defined using XML.[6]

Predefined WinFS schemas include schemas for documents, e-mail, appointments, tasks, media, audio, video, and also includes system schemas that include configuration, programs, and other system-related data.[8] Custom schemas can be defined on a per-application basis, in situations where an application wants to store its data in WinFS, but not share the structure of that data with other applications, or they can be made available across the system.[8]

WinFS Type Hierarchy

The most important difference between a file system and WinFS is that WinFS knows the type of each data item that it stores. And the type specifies the properties of the data item. The WinFS type system is closely associated with the .NET framework's concept of classes and inheritance. A new type can be created by extending and nesting any predefined types.[6]

WinFS provides four predefined base types – Items, Relationships, ScalarTypes and NestedTypes.[6] An Item is the fundamental data object which can be stored, and a Relationship is the relation or link between two data items. Since all WinFS items must have a type, the type of item stored defines its properties. The properties of an Item may be a ScalarType, which defines the smallest unit of information a property can have, or a NestedType, which is a collection of more than one ScalarTypes and/or NestedTypes. All WinFS types are made available as .NET CLR classes.[30]

Any object represented as a data unit, such as contact, image, video, document etc., can be stored in a WinFS store as a specialization of the Item type.[30] By default, WinFS provides Item types for Files, Contact, Documents, Pictures, Audio, Video, Calendar, and Messages. The File Item can store any generic data, which is stored in file systems as files. But unless an advanced schema is provided for the file, by defining it to be a specialized Item, WinFS will not be able to access its data. Such a file Item can only support being related to other Items.[6]

Defining a new Type

A developer can extend any of these types, or the base type Item, to provide a type for their custom data. The data contained in an Item is defined in terms of properties, or fields that hold the actual data. For example, an Item Contact may have a field Name that is a ScalarType, and one field Address, a NestedType, which is further composed of two ScalarTypes. To define this type, the base class Item is extended and the necessary fields are added to the class.[6] A NestedType field can be defined as another class that contains the two ScalarType fields. Once the type is defined, a schema has to be defined, which denotes the primitive type of each field, for example, the Name field is a String, the Address field is a custom defined Address class, both the fields of which are Strings. Other primitive types that WinFS supports are Integer, Byte, Decimal, Float, Double, Boolean and DateTime, among others.[6] The schema will also define which fields are mandatory and which are optional.[31] The Contact Item defined in this way will be used to store information regarding the Contact, by populating the properties field and storing it. Only those fields marked as mandatory needs to be filled up during initial save.[30] Other fields may be populated later by the user, or not populated at all. If more properties fields, such as last conversed date, need to be added, this type can be extended to accommodate them. Item types for other data can be defined similarly.

Relationships

WinFS creates tables for all defined Items.[31] All the fields defined for the Item form the columns of the table and all instances of the Item are stored as rows in the table for the respective Items. Whenever some field in the table refers to data in some other table, it is considered a relationship. The schema of the relationship specifies which tables are involved and what the kind and name of the relationship is. The WinFS runtime manages the relationship schemas.[30] All Items are exposed as .NET CLR objects, with a uniform interface providing access to the data stored in the fields. Thus any application can retrieve object of any Item type and can use the data in the object, without being aware of the physical structure the data was stored in.[6]

WinFS types are exposed as .NET classes, which can be instantiated as .NET objects. Data are stored in these type instances by setting their properties. Once done, they are persisted into the WinFS store. A WinFS store is accessed using an ItemContext class (see Data retrieval section for details). ItemContext allows transactional access to the WinFS store; i.e. all the operations since binding an ItemContext object to a store till it is closed either all succeed or are all rolled back. As changes are made to the data, they are not written to the disc; rather they are written to an in-memory log. Only when the connection is closed are the changes written to the disc in a batch. This helps to optimize disc I/O.[10] The following code snippet, written in C#, creates a contact and stores it in a WinFS store.

 //Connect to the default WinFS store
 using(ItemContext ic = ItemContext.Open())
 {
     //Create the contact and set the data in appropriate properties
     ContactEAddress contact = new ContactEAddress() {
         Name = new PersonName() {                      // Name is a ComplexType
             Displayname = "Doe, John",
             FirstName = "John",
             LastName = "Doe"
         },
         TelephoneNumber = new TelephoneNumber() {     // Telephone number is a ComplexType
             Country = CountryCode.Antarctica,
             Areacode = 4567,
             Number = 9876543210
         },
         Age = 111                                     // Age is a SimpleType
    };

    //Add the object to the user's personal folder.
    //This relates the item with the Folder pseudo-type, for backward
    //compatibility, as this lets the item to be accessed in a folder 
    //hierarchy for apps which are not WinFS native.
    Folder containingFolder = UserDataFolder.FindMyPersonalFolder();
    containingFolder.OutFolderMemberRelationship.AddItem(ic, contact);

    //Find a document and relate with the document. Searching begins by creating an
    //ItemSearcher object. Each WinFS type object contains a GetSearcher() method
    //that generates an ItemSearcher object which searches documents of that type.
    using (ItemSearcher searcher = Document.GetSearcher(ic))
    {
         Document d = searcher.Find(@"Title = 'Some Particular Document'");
         d.OutAuthoringRelationship.AddItem(ic, contact);
    }
    //Since only one document is to be found, the ItemContext.FindOne() method
    //could be used as well.

    //Find a picture and relate with it
    using (ItemSearcher searcher = Picture.GetSearcher(ic))
    {
         Picture p = searcher.Find(@"Occasion = 'Graduation' and Sequence = '3'");
         p.OutSubjectRelationship.AddItem(ic, contact);
    }

    //Persist to the store and close the reference to the store
    ic.Update();
 }

A datum can be related to one more item, giving rise to a one-to-one relationship, or with more than one items, resulting in a one-to-many relationship.[6] The related items, in turn, may be related to other data items as well, resulting in a network of relationships, which is called a many-to-many relationship. Creating a relationship between two Items creates another field in the data of the Items concerned which refer the row in the other Item's table where the related object is stored.[30]

WinFS Relationships

In WinFS, a Relationship is an instance of the base type Relationship, which is extended to signify a specialization of a relation. A Relationship is a mapping between two items, a Source and a Target. The source has an Outgoing Relationship, whereas the target gets an Incoming Relationship.[31] WinFS provides three types of primitive relationships – Holding Relationship, Reference Relationship and Embedding Relationship.[6] Any custom relationship between two data types are instances of these relationship types.

• Holding Relationships specify ownership and lifetime (which defines how long the relationship is valid) of the Target Item. For example, the Relationship between a folder and a file, and between an Employee and his Salary record, is a Holding Relationship – the latter is to be removed when the former is removed. A Target Item can be a part of more than one Holding Relationships. In such a case, it is to be removed when all the Source Items are removed.
• Reference Relationships provide linkage between two Items, but do not have any lifetime associated, i.e., each Item will continue to be stored even without the other.
• Embedding Relationships give order to the two Items that are linked by the Relationship, such as the Relationship between a Parent Item and a Child Item.

Relationships between two Items can either be set programmatically by the application creating the data, or the user can use the WinFS Item Browser to manually relate the Items.[31] A WinFS item browser can also graphically display the items and how they are related, to enable the user to know how their data are organized.[30]

WinFS includes Rules,[22] which are executed when a certain condition is met. WinFS rules work on data and data relationships. For example, a rule can be created that states that whenever an Item is created which contains field "Name" and if the value of that field is some particular name, a relationship should be created that relates the Item with some other Item. WinFS rules can also access any external application. For example, a rule can be built which launches a Notify application whenever a mail is received from a particular contact.[22] WinFS rules can also be used to add new properties fields to existing data Items.[22]

WinFS rules are also exposed as .NET CLR objects. As such any rule can be used for any purpose. A rule can even be extended by inheriting from it to form a new rule that consists of the condition and action of the parent rule plus something more.[22]

WinFS supports creating Rich Application Views (RAV) by aggregating different data in a virtual table format. Unlike database view, where each individual element can only be a scalar value, RAVs can have complex Items or even collections of Items. The actual data can be across multiple data types or instances and can even be retrieved by traversing relationships.[10] RAVs are intrinsically paged (dividing the entire set of data into smaller pages containing disconnected subsets of the data) by the WinFS runtime. The page size is defined during creation of the view and the WinFS API exposes methods to iterate over the pages. RAVs also supports modification of the view according to different grouping parameters. Views can also be queried against.

Even though all data are shared, everything is not equally accessible. WinFS uses the Windows authentication system to provide two data protection mechanisms.[30] First, there is share-level security that controls access to your WinFS share. Second, there is item level security that supports NT compatible security descriptors. The process accessing the item must have enough privileges to access it. Also in Vista there is the concept of "integrity level" for an application. Higher integrity data cannot be accessed by a lower integrity process.

The WinFS API provides a class called the ItemContext class, which is bound to a WinFS store. The ItemContext object can be used to scope the search to the entire store or a subset of it. It also provides transactional access to the store.[15] An object of this class can then spawn an ItemSearcher object which then takes the type (an object representing the type) of the item to be retrieved or the relationship and the OPath query string representing the criteria for the search.[31][33] A set of all matches is returned, which can then be bound to a UI widget for displaying en masse or enumerating individually.[30] The properties items can also be modified and then stored back to the data store to update the data. The ItemContext object is closed (which marks the end of association of the object with the store) when the queries are made or changes merged into the store.

Related items can also be accessed through the items. The IncomingRelationships and OutgoingRelationships properties give access to all the set of relationship instances, typed to the name of the relationship. These relationship objects expose the other item via a property. So, for example, if a picture is related to a picture, it can be accessed by traversing the relationship as:

ContactsCollection contacts = picture.OutgoingRelationships.Cast(typeof(Contact)).Value;
// This retrieves the collection of all outgoing relationships from a picture object
// and filters down the contacts reachable from them and retrieves its value.

// Or the relationship can be statically specified as
ContactsCollection contacts = picture.OutgoingRelationships.OutContactRelationship.Contact;

An OPath query string allows to express the parameters that will be queried for to be specified using Item properties, embedded Items as well as Relationships. It can specify a single search condition, such as "title = Something'", or a compound condition such as "title = 'Title 1' || title = 'Title 2' && author = 'Someone'". These boolean and relational operations can be specified using C# like &&, ||, =, != operators as well as their English-like equivalent like EQUAL, NOT EQUAL. SQL like operators such as LIKE, GROUP BY and ORDER BY are also supported, as are wildcard conditions. So, "title LIKE 'any*'" is a valid query string. These operators can be used to execute complex searches such as

using (ItemContext ic = ItemContext.Open())
{
    // Searching begins by creating a ItemSearcher object. The searcher is created from a
    // relationship instance because the contacts being searched for are in relation. The
    // first parameter defines the scope of the search. An ItemContext as the scope means
    // the entire store is to be searched. Scope can be limited to a set of Items which may
    // be in a holding relationship with the contacts. In that case, the set is passed as
    // the scope of the search.
    ItemSearcher searcher = OutContactRelationship.GetTargetSearcher(ic, typeof(Contact));
    ContactCollection contacts = searcher.FindAll("OutContactRelationship.Contact.Name LIKE 'A*'");
}

The above code snippet creates an ItemSearcher object that searches on the OutContactRelationship instance that relates pictures and contacts, in effect searching all pictures related with a contact. It then runs the query Name LIKE 'A*'" on all contacts reachable through OutContactRelationship, returning the list of "contacts whose names start with A and whose pictures I have". Similarly, more relationships could be taken into account to further narrow down the results.[10][31] Further, a natural language query processor, which parses query in natural language and creates a well-formed OPath query string to search via proper relationships, can allow users to make searches such as "find the name of the wine I had with person X last month", provided financial management applications are using WinFS to store bills.

Different relations specify a different set of data. So when a search is made that encompasses multiple relations, the different sets of data are retrieved individually and a union of the different sets is computed. The resulting set contains only those data items that correspond to all the relations.[31]

WinFS includes better support for handling data that changes frequently. Using WinFS Notifications, applications choose to be notified of changes to selected data Items. WinFS will raise an ItemChangedEvent, using the .NET Event model, when a subscribed-to Item changes, and the event will be published to the applications.[31]

WinFS includes an Information Agent feature for the management, retrieval, and storage of end-user notification rules and preferences for changes to items in the data store. Using Information Agent, it is possible to automatically define relations to new items based on events such as appointments, with an example being that appointments can be related to photos based on the dates the photos were taken, enabling queries for birthdays or holidays without needing to know the actual dates of such events ("find all photos taken on this birthday"). Other examples include automatically moving new items to specific folders based on a rule as determined by appointment times and dates the photos were taken ("when I import a photo taken during a business event, move it to the Business Events folder") or more complex possibilities. Information Agent can also forward notifications to other devices ("if I receive a high priority email from my boss, send a notification to my phone") and is similar to Rules and Alerts functionality of Microsoft Outlook.

WinFS Beta 1 includes a shell namespace extension, which surfaces WinFS stores as top level objects in My Computer view.[12] Files can be copied into and out of the stores, as well as applications can be directly used to save there. Even folders such as My Documents can be redirected to the stores.[12] WinFS uses Importer plug-ins to analyze the files as they were being imported to the store and create proper WinFS schemas and objects, and when taking the objects out, re-pack them into files.[34] If importers for certain files are not installed, they are stored as generic File types.

Microsoft Rave is an application that shipped with WinFS Beta 1. It allows synchronization of two or more WinFS stores, and supports synchronization in full mesh mode as well as the central hub topology. While synchronizing, Microsoft Rave will determine the changes made to each store since the last sync, and update accordingly. When applying the changes, it also detects if there is any conflict, i.e., the same data has been changed on both stores since the last synchronization. It will either log the conflicting data for later resolution or have it resolved immediately. Microsoft Rave uses peer-to-peer technology to communicate and transfer data.

WinFS included StoreSpy, a WinFS Item browser that could be used to browse structured data instances with their properties and relationships.

With WinFS Beta 1, Microsoft included an unsupported application called StoreSpy,[35] which allowed one to browse WinFS stores by presenting a hierarchical view of WinFS Items. It automatically generated virtual folders based on access permissions, date and other metadata, and presented them in a hierarchical tree view, akin to what traditional folders are presented in. The application generated tabs for different Item types. StoreSpy allowed viewing Items, Relationships, MultiSet, Nested Elements, Extensions[36] and other types in the store along with its full metadata. It also presented a search interface to perform manual searches, and save them as virtual folders. The application also presented a graphical view of WinFS Rules. However, it did not allow editing of Items or their properties, though it was slated for inclusion in a future release.[37] But the WinFS project was cut back before it could materialize.

WinFS also includes another application, named WinFS Type Browser, which can be used to browse the WinFS types, as well as visualize the hierarchical relationship between WinFS types.[38] A WinFS type, both built-in types as well as custom schemas, can be visualized along with all the properties and methods that it supports. It also shows the types that it derives from as well as other types that extend the type schema. However, while it was included with WinFS, it was released as an unsupported tool.[38]

WinFS Beta 1 also includes an unsupported application, named OPather.[39] It presents a graphical interface for writing Opath queries. It can be used by selecting target object type and specifying the parameters of the query. It also includes Intellisense-like parameter completion feature. It can then be used to perform visualization tasks like binding results of a query to a DataGrid control, create views of the data in WinFS itself, or just extract the query string.

Microsoft launched a project to build a data visualization application for WinFS. It was codenamed "Project Orange" and was supposedly built using Windows Presentation Foundation.[40] It was supposed to provide exploration of Items stored in WinFS stores, and data relationships were supposed to be a prominent part of the navigation model. It was supposed to let people allow organization of the WinFS stores graphically as well – productizing many of the concepts shown in the IWish Concept Video WMV File. However, since the WinFS project went dark, the status of this project is unknown.