On this article, I will try to explain the data model that is being used in Adobe Experience Platform and how you can better plan for data integration into Adobe Experience Platform.
This topic is quite complex and is at the base of how Adobe Experience Platform will be used by your organization, so it is worth spending some time there. By that I mean spend some time now reading this article and some time when you are doing AEP implementation.
Adobe has great consultants to help you on your strategy for Experience Platform but they will never have your knowledge and definition of use-cases for your company. They are not all knowing and because this solution is quite new, things that may not be possible now, might be available if a good use case is found.
AEP is based on Schema
Example of Schema
First of all I felt that I needed to give you a picture of what a schema is in AEP and what are you seeing when you create one.
I already wrote an extensive article on AEP Schema creation, these schemas are based on the Experience Data Model and it is important to understand that you are going to need to transform your data to fit a particular schema.
There is a possibility to map some csv files to a particular schema so you can still limit your data preparation but you will still need to create your schema to match your data.
In my article about Experience Data Model, I went deep into details, in reality you will not have to deal with such details, you will “just” need to create your Schema with a defined class and some mixins.
Below you can find an example based on the XDM Individual Profile class.
As you can see there are some elements that are linked together (Class, Mixins) and we will try to explain each of them. You also have to notice that the elements on the right are fields of that schema. Those fields are going to contain the value you are inserted.
Each field is expecting a specific type of data to be inserted.
Per example, on the schema path “_emeaconsulting.identities.crmId” a string is expected during data insertion. Trying to ingest an object and it will return an error on your data insertion.
Schema are based on classes & mixins
There are some important facts that you need to know when you create your Schemas and I will try to summarize them at the top, while you are still reading:
- Default schemas are XDM Individual Profile (attribute of a user) and XDM ExperienceEvent (timestamp data).
You want to try to use them as much as possible for defining your data. We will see why later. - One Schema can be used to create multiple datasets.
Datasets are where your data are going to be stored. You will have some that are automatically created for you (Adobe Analytics ReportSuites are going to be a dataset) and some that you will need to create when you ingest data from other sources (not Adobe ones).
You can see them as tables in a SQL data base because you will be able to request them through Query Service at some point. - The fields (that host the data) that you are creating in your schema are going to create path in your global schema. You can only have one unique path exiting in your schemas. This may be quite unclear at the moment, but we will see with example later on. Just remember the uniqueness of the schema path.
- You don’t have to populate all fields of your schema when you import data against that schema. It is working the other way around. You must have a field defined in your schema for each data that you are ingesting.
What are classes ?
The first element we are going to defined on our data model is the different data type that are going to be ingested on Experience Platform. The type of data that are going to be ingested will define what type of schema you will use.
The rule is quite simple here :
- Does your data has a timestamp associated ?
Yes –> Highly consider “XDM ExperienceEvent” class - Does your data gives your user an attribute ?
Yes –> Highly consider “XDM Individual Profile”
This is not automatic to assign XDM ExperienceEvent to timestamp based data but almost. There are some inner logic that you will need to know.
So let me know digress for now to explain some of the end result you will have when ingesting data into Platform.
Data views in Platform
One important concept that you need to understand is that the different data that you are ingesting into Platform will create different views on your data.
There are 3 different views that you will be able to build:
1. Dataset view
The dataset view is the most expected view that you are going to have.
Once you upload the data into Platform, you can query it like a table through Query Service.
This will give you the possibility to retrieve the information you have ingested in an easy way. Also you will be able to load them directly into your Data Science Workspace for any Model you want to do.
2. Combined datasets
As you can imagine, you will have the possibility to create a query to combine multiple datasets together and derive insights very easily.
Adobe Experience Platform also provide the possibility to export that data into any system that you would wish for.
You can do that by directly exporting the output of your query through Query Service or you can create a new Data Set from your query and download that dataset directly.
Because you can create new insights by using the Data Science Workspace, you can also generate new dataset directly from there and download these new datasets if needed.
3. Union Profile View
Hopefully you will not be surprised by that one if you already have Adobe Experience Platform, but if you don’t have Adobe Experience Platform this may be a bit new to you.
In the background of all of these data insertions, AEP is building an unified profile for your users. A unified profile is a graph data for your users that have attributes and events attached to their profile. Adobe Experience Platform is based on creating enriched profile for your actions and every data ingested should add new data points to the user profile graph that you are building.
This a very abstract construct because you cannot really request this dataset as a whole, you can uniquely request a Unified Profile for a specific user.
You can have more detail if you look on this page , on the entity element. https://www.adobe.io/apis/experienceplatform/home/api-reference.html#!acpdr/swagger-specs/real-time-customer-profile.yaml
We will see this in more detail in another post but you have to understand that this profile is being built based on the different paths that you have created in your schema.
Therefore, if you schema is using a XDM Profile class, the path “_emeaconsulting.personData.gender” exists, it will create an attribute to this user unified profile based on the data being ingested. All of the ingestion will be added on his profile and enrich the user unified profile.
I don’t think this is the best way of representing the Graph data but every dataset nurrish it. The user is at the center of this and get more and more data points affected to him.
Now that we have done this interlude, what are classes ?
Classes defines the type of data that is being ingested and this will create 2 different way to process the data.
If the data are XDM ExperienceEvent, it will be attached to the user in a array-like data points.
All of the events realized by your user will be attached to the user and you will be able to request the whole list of element that has been done by the user. Nothing is being deleted or merged, you are just building a full history of interactions, cross-device if you have, at some point, 2 identity fields being ingested at the same time (like idSync for AAM, but we will see that later).
If the data are XDM Individual Profile, you are now ingesting attributes to this Unified Profile. This is where the uniqueness of the path is important, if you are ingesting 2 datasets using the same schema, then data are being merged. Your users will sometimes have the same attributes (like always “Female”) but it may be that 2 datasets give 2 different attributes to the same user (per example address). In that case, what are you going to do when this conflict arise?
In that case there is something called “Merge Policy” that comes into play. These merge policies will define how you are handling your conflicts. At the moment, you can either take timestamp as referent (younger data are overwriting older data) or you have you source of truth so you set a specific dataset to never be overwritten. More precisely the attributes defined by this dataset cannot be overwritten by other data ingestion.
It is important to highlight that this merging policy only apply to the XDM Individual Profile data. As explained previously, ExperienceEvent are not merging anything when you request a unified profile.
What is a mixin ?
We now know what the schema are being defined but what about mixin ?
At the end, they don’t defined anything useful for our data being ingested or for the fields themselves.
Mixins are used to group different fields in a common thema. The most common mixin is the “identity mixin” where you gather all of the different identity possibility that you own.
Creating a mixin for these fields is very practical because all of your Schema will have an identity field. By just importing your “identity mixin”, you will directly have it in your schema.
Actually you could use pre-existing mixins for your schema but some of them are not supposed to be used outside of specific context. Therefore it can be quite risky because you may miss some dependencies and then make your schema unusable. Also the pre-existing mixins are not possible to be transformed. Consequently if you have a field, let say “age”, it is not possible to extend the person pre-existing mixin to add this field. You will need to create an additional mixin.
How do you know which mixin to use and which one to not use ? You don’t (so far).
Adobe Experience Platform team is working on a way to better identify which mixin are reusable for custom purposes and which one are supposed to live only within a specific context. It is not there yet but as soon as it is available, I will try to update this post.
About Data Type
We do know about the mixin and why we create them, they are cluster of information that we want to have.
Therefore, one of the most important step when you start on Adobe Experience Platform is to identify all of the different information you have about your customer, that you want to import into AEP and group them into cluster or groups.
Once you have that, you can start to create your own mixins and add the fields that you want with the appropriate data type that are expected (string, double, integer, boolean, etc…).
Adobe Experience Platform also provide other predefined data type that you can use and they serve as collection bulk of fields that you may want to have for your categories.
In my example, you can see that I used the “Person” data type at some point and because it is not a mixin (so not ) but a data type, I am able to add any other fields (in that case “myAge”) in the associated mixin I created (in that case “myPersonMixin)”.
Conclusion on Data Model for AEP
On the conlusion for the data model on AEP, I would like to report the important information you need to consider when creating your schemas.
- Gather all the different data that you are going to import into AEP
- From these data, defined the attributes and event attached to each of your users.
It will help defining the different class you will use in your Schema. - Remember about the Unified Profile and the uniqueness of the paths you are creating in your schema.
- You can use the same schema to create multiple datasets and each of the fields in your schema don’t have to be filled. But each of the value you are sending you should be defined in your schema.
- Data Type can help you defining the different information you want to retrieve from your users.
- Remember of the different views on your customers that you are going to create in AEP.
I hope this article helped you understanding how to better structure your data with AEP and what AEP can create for you.