About the Data Transfer ProjectThe Data Transfer Project was launched in 2018 to create an open-source, service-to-service data portability platform so that all individuals across the web could easily move their data between online service providers whenever they want.Why do we need the DTP -- Companies have been adding the ability to download you data (Takeout, Microsoft Export). This is great and is a huge step forward in terms of empowering users to be in control of their data. However just being able to download your data has a number of shortcomings: * Bandwidth If you have limited bandwidth downloading and re-uploading data can either take a LONG time and/or be expensive, oftentimes to the point of making it impractical. * Incompatible Formats Companies have (for some reason) all started offering their data in structured, commonly used and machine-readable formats, however in most cases those formats are not compatible with one another making it hard for users to re-import data they have exported. * Discoverability When exporting data it’s often hard to know where else you could take the data. ^ DTP aims to solve those problems by making it easy for users to directly transfer data directly from one provider to another. This puts both the burden for bandwidth and figuring out compatibility with the service providers and not with the user. An alternative approach to DTP is to have each company build direct transfer capabilities with each alternative service provider directly.Data Transfer ProjectDTP_548a28da-5e00-11eb-bc62-15561e83ea00Data Transfer Project (DTP) is a collaboration of organizations committed to building a common framework with open-source code that can connect any two online service providers, enabling a seamless, direct, user initiated portability of data between the two platforms.Data Transfer Project ContributorsThe contributors to the Data Transfer Project believe portability and interoperability are central to innovation. Making it easier for individuals to choose among services facilitates competition, empowers individuals to try new services and enables them to choose the offering that best suits their needs. Current contributors include:AppleFacebookGoogleMicrosoftTwitterIndividualsIndividuals should be able to easily transfer their files and data directly between online service providers.DTP ParticipantsHow to participate -- We welcome everyone to participate, the more expertise and viewpoints we have contributing to the project the more successful it will be. There are four ways to join the Data Transfer Project: * Partner - To become a partner in the Data Transfer Project, an organization must agree to follow the principles and best practices described in the DTP white paper, contribute to DTP efforts, and participate in the DTP by committing to add and maintain adapters. These organizations have their logos on the DTP website and represent the project publicly in public conversations. * Provider - A provider is an organization integrated into the project, either because it developed its own adapter or had a public API another developer integrated with the project. Data can move into and out of these organizations subject to the terms of the provider APIs. * Contributor - As an open source project, anyone can contribute to the code repository comprising the DTP codebase. * Thought leadership - The Data Transfer Project meets regularly with thought leaders, other interested members of the public, and the research community. ^ Anyone can join the mailing list at (DTP Discuss) and (Slack Channel) to stay informed of developments and discussions.Data ProvidersWhat providers may participate in the Data Transfer Project? The DTP facilitates the transfer of an individual’s data between providers. For a provider to be included in the Project, they have to provide export (and ideally import) functionality for some or all of their users’ data. *(If the organization is a Partner, they should plan to build import functionality due to the principle of reciprocity.) Functionality may be executed through a public-facing API or other reasonable substitute. Code for some providers may have been included in the repository via contributions to the developer community, who can utilize public-facing APIs to build an adapter.Seamless, direct, user initiated portability of data between platforms_548a2b78-5e00-11eb-bc62-15561e83ea00To building a common framework with open-source code that can connect any two online service providers_548a2e34-5e00-11eb-bc62-15561e83ea00DataPortabilityInteroperabilityInnovationChoiceCompetitionEmpowermentData TransferEnable consumers to transfer data in and out of any participating provider_548a2f4c-5e00-11eb-bc62-15561e83ea00ConsumersData Transfer Project (DTP) extends data portability beyond downloading a copy of your data from your service provider, to providing consumers the ability to directly transfer data in and out of any participating provider.Data ProvidersData Transfer Project is an open source initiative to encourage participation of as many Providers as possible.DTP will enhance the data portability ecosystem by reducing the infrastructure burden on both service providers and users which should in turn increase the number of services offering portability. The protocols and methodology of DTP enable direct, service-to-service data transfer with streamlined engineering work... The Data Transfer Project uses services’ existing APIs and authorization mechanisms to access data. It then uses service specific adapters to transfer that data into a common format, and then back into the new service’s API.Data ModelsEstablish a common understanding of how to transfer data_548a303c-5e00-11eb-bc62-15561e83ea001Data Models -- Data Models are the canonical formats that establish a common understanding of how to transfer data. Adapters provide a method for converting each Provider’s proprietary data and authentication formats into a form that is usable by the system. Task Management Library provides the plumbing to power the system. Data Models Data Models represent the data when being transferred between two different companies. Ideally each company would use interoperable APIs (e.g. ActivityPub) to allow data to flow between them. However in many cases that is not the case. In those cases there needs to be a way to transfer the data from one companies representation to another companies representation. Data Models are clustered together, typically by industry grouping, to form Verticals. A Provider could have data in one or more Verticals. Verticals could be photos, email, contacts, or music. Each Vertical has its own set of Data Models that enable seamless transfer of the relevant file types. For example, the Music vertical could have Data Models for music, playlists and videos. Ideally, a Vertical will have a small number of well-defined and widely-adopted Data Models. In such a situation, the generally accepted standard will be used as the Data Model for that Vertical across companies. This is not currently the case for most Verticals because Data Models have emerged organically in a largely disconnected ecosystem. One goal of DTP is to encourage organizations to use common Data Models in their systems, which will happen if organizations take importing and exporting data into consideration when initially designing their systems or providing updates. Using a common Data Model will significantly reduce the need for companies to maintain and update proprietary APIs.Adapters_548a3118-5e00-11eb-bc62-15561e83ea002Company Specific Adapters -- There are two main kinds of Adapters: Data Adapters and Authentication Adapters. These Adapters exist outside of a Provider’s core infrastructure and can be written either by the Provider itself, or by third parties that would like to enable data transfer to and from a Provider. Data Adapters Data Adapters are pieces of code that translate a given Provider’s APIs into Data Models used by DTP. Data Adapters come in pairs: an exporter that translates from the Provider’s API into the Data Model, and an importer that translates from the Data Model into the Provider’s API. Authentication Adapters Authentication Adapters are pieces of code that allow consumers to authenticate their accounts before transferring data out of or into another Provider. OAuth is likely to be the choice for most Providers, however DTP is agnostic to the type of authentication.TasksHandle background tasks_548a3208-5e00-11eb-bc62-15561e83ea003Task Management -- The rest is just plumbing. The Task Management Libraries handle background tasks, such as calls between the two relevant Adapters, secure data storage, retry logic, rate limiting, pagination management, failure handling, and individual notifications. DTP has developed a collection of Task Management Libraries as a reference implementation for how to utilize the Adapters to transfer data between two Providers. If preferred, Providers can choose to write their own implementation of the Task Management Libraries that utilize the Data Models and Adapters of DTP.2018-07-202021-01-24https://datatransferproject.dev/OwenAmburOwen.Ambur@verizon.net