Data Imports

Follow

Applies to: Pro, Plus, & Enterprise Plans

Introduction

This article details the overall process and best practices for importing various types of data into Absorb. Please note that these imports are an additional service that usually involve an additional fee and technical resources on both the Absorb and client sides. If you are just looking to import users in bulk, we have a standard User Import tool available on the admin side portal for all clients.

In the following sections we will heavily reference an Absorb Data Mapping Spreadsheet. As such, for all mentions of fields and/or files, readers should be referencing the Absorb Data Mapping Spreadsheet as well unless otherwise indicated. Download the Absorb Data Mapping Spreadsheet and make sure to review the Instructions tab.

Contents:

Historical Data Import

Alongside its many other features, Absorb has been designed to allow the importation of data from other different Learning Management Systems, so that any new LMS customer can continue making use of their information (for both reporting and user experience purposes) as soon as they start using our product. These types of data imports are done one-time, usually during the onboarding of new Absorb clients to allow organizations a seamless transition from one system to another – for both LMS administrators and learners alike. 

CSV Sync Integration

CSV sync integrations are essentially recurring historical data imports (except the data isn't really "historical" after the first import). The main difference is that after the initial import (usually a full extract from the other system), we schedule automatic processing of subsequent files dropped on our FTPS site at some decided frequency. For example, we may automatically process a file containing new or updated user records each night at 3am, which the client's system automatically exports and drops on our FTPS each night at 2:30am.

Tip: We will provide the FTPS server and account details as part of the implementation. If you require that we pick the files up from your own secure FTP server, please discuss and confirm with us. You can find more information in our FTP Support for Integrations article.

External ID

Of the many important concepts that will be discussed in this article, the use of the ExternalId is definitely key. These fields exist as a means for uniquely identifying and linking related LMS records. The value of these special fields can either be ‘real’ as taken from some external system or they can be purpose-generated for the sole use of your Absorb data import. An example would be when importing users and their course enrollments into the system, in which case the Course Enrollment table's User_ExternalId would refer to the User table's ExternalId, thereby relating the two elements during the import.

 

Import Process

We will walk you through the steps below during an initial scoping call where requirements and timelines are discussed. The first major deliverable that we typically work towards is obtaining sample file set and completed Client Data Import Requirements document. Our development team then reviews your sample file set and attempts to import data to a sandbox environment. For that reason, it is very important that the sample file set is provided in the exact final format that your final production data will be provided in. Any change to data format after import work begins can result in delays and/or added cost.

Tip: It's usually best to provide real data in your sample files instead of mock data. This allows us to identify any data quality issues early on in the process.

A slightly difference process is followed for "Standard" vs. "Custom" imports. Most imports are considered "Standard" unless specifically scoped, quoted and discussed otherwise with us.

Standard Import

A Standard Import requires the client to provide their import files in a standard format, defined in the Absorb Data Mapping Spreadsheet. Any modifications needed beyond these stock formats will require a Custom Import (as outlined in the next section). 

Standard Import Steps: 

  • Scoping call and confirmation of requirements (complete the Data Import Requirements Template)
  • Client sample files provided (must match Absorb Data Mapping document exactly
  • Import to sandbox environment
  • Data revision and re-import (if required > maximum of 2 re-imports* allowed
  • Client review and approval
  • Final data files provided (generally just prior to / or after launch) 
  • Final import to sandbox
  • Final approval from client
  • Final import to production
  • Client review and confirmation

Custom Import

A Custom Import allows for greater flexibility in the file formats that an Absorb client can provide for the importation process of their historical data. The companion Absorb Data Mapping Spreadsheet is still used to identify available data fields, however with this type of import the order of a client’s columns, their field delimiters and custom logic (subject to scoping and approval) can all be modified.
 
Custom Import Steps: 
  • Scoping call and confirmation of requirements (complete the Data Import Requirements Template)
  • Client sample files provided
  • Custom data mapping completed (fields mapped as per the Absorb Data Mapping document) 
  • Import to sandbox environment
  • Data revision and re-import (if required > maximum of 4 re-imports* allowed
  • Client review and approval
  • Final data files provided (generally just prior to or after launch) 
  • Final import to sandbox
  • Final approval from client
  • Final import to production
  • Client review and confirmation
*A re-import involves the client providing updated data files, generally in order to resolve issues with data accuracy and/or consistency identified after a previous importation event. Files should be provided in the same format as the previously imported versions. Slight modifications such as adding or dropping a single column may be allowed provided they are discussed with Absorb first. However, be advised that Absorb reserves the right to consider any modification made to a file format after the initial import development has begun as out of scope, additional work.
 

File Format

Files should be provided in CSV (Comma Separated Values) format. The Instructions tab of the Data Mapping Spreadsheet provides further directions for naming and creating your CSV files. 
 

Requirements

The Data Import Requirements Template document should be reviewed and completed with our assistance. Please review this document carefully as it contains all of the options and configuration that we will use for this data import. 

Common Elements

This section will detail some of the more commonly imported elements, their dependencies, and special considerations. Keep these in mind when formulating your file set.

Users:

All Users

In order to import users (which includes learners, administrators and instructors) Absorb requires the client’s Users file at a minimum. Generally a Departments file is also required, though the client may wish to set up their departments manually through the Absorb Admin Interface first (note: the ExternalId field must be filled in during setup). Be advised that all admin users (IsAdmin=1) will be assigned only a single role (or perhaps no role at all – either or which will be decided by the client prior to carrying out the data import) if no other files are provided.

Administrators

Advanced user imports can also configure the assignment of different admin access roles per each user (with the inclusion of a Role Assignments file), restrict admin access to users in certain departments (by incorporating a Department Admins file), and/or limit admin access to users in certain groups (by adding a Group Admins file).

Special Considerations

Any user referenced in the Supervisor_ExternalId field must also be set as an admin user (i.e. IsAdmin=1). Clients should note that all admin users will be assigned whatever default admin role is chosen for the import unless a Role Assignments file is also provided.

Custom Field columns should be named exactly as they are named in the Admin Interface, without spaces (e.g. "My Custom Field" in the admin site becomes the "MyCustomField" column in your file).

Courses, Curriculums and Course Bundles:

Metadata vs Content
It is important to note that Absorb only imports course metadata such as name, description, some course settings, etc. and not the actual course content (SCORM files, videos, etc.) itself. This means that the uploading of course content still needs to be done on a course-by-course basis through the Admin Interface once the initial importation is completed. Some Absorb clients prefer to complete their course setup entirely through the Admin Interface for this reason. If this situation applies to your organization, then please ensure that you fill out each course’s External ID field if Absorb is required to import any other entities that must be linked to courses (e.g. Course Enrollments).
 
Minimum Requirements
In order to import courses (i.e. Online and Instructor Led) Absorb requires a Courses file – at a minimum. Individual lessons underneath each Online Course can also be populated using the Lessons file, though keep in mind that clients must still upload and link content for each of those lessons to the Course through the Admin Interface. Scheduled instances of your Instructor Led Courses can be populated using the Sessions and Classes files, but note that the Classes file must also be accompanied by a Venues file.
 
Multiple Courses
Curriculums (learning paths) and Course Bundles contain courses within them and can also be imported using their respective files. 
 
Special Considerations
The AdminVisibilityDepartmentId field restricts admin visibility to specific courses in the Admin Interface only. This setting is generally not required unless you need to segregate your course visibility to different sets of admins. Should your import require this then it is recommended that you set up and test examples of this through the admin portal before applying the settings in a bulk import.
 
Certificate_IsEnabled must be enabled before setting any of the other Certificate fields (e.g. Certificate_ExpireType). The same sequence also applies to EvaluationEnabled, ToggleEmails, SendNudgeEmails, AllowReEnroll and AllowFailure in that you must enable these fields first in order to set other fields related to them. (Be advised that not all of these fields are applicable to Curriculums and Course Bundles however).
 

Course Progress and Completion Data:

General
In order to import learner course progress and completion data (i.e. scores, progress, start / completion dates, etc.) Absorb requires the Course Enrollments file – at a minimum. This file will set learner activity at the Course level only, meaning that each learner will have a score, status, etc. for the Course itself but not for each individual lesson under each Course. This level of detail is usually sufficient. Please note that the Course Enrollment file refers to both users and courses, so those files must also be provided or Users and/or Courses must be set up through the Admin Interface with the External ID fields populated before the import commences.
 
Additional Detail
If an Absorb client deems that more granular learner activity is needed, then they may provide Absorb with both Course Enrollments (course level) and Enrollment Activities (lesson level) files as part of their import inputs. It’s very important to note that Absorb will also require Courses and Lessons files as well, if asked to incorporate this level of detail.
 
Learning Paths
If a client has courses that are organized into learning paths with learner activity and enrollment that they would like to preserve, then a Curriculum Enrollments file should also be provided. Curriculums themselves must also be provided either in a file or created manually through the Admin Interface, as described above for the Course Enrollments file.
 
Certificates / Badges
Absorb clients may also want to preserve or issue their existing learner certificates or competencies (badges) in the new LMS. In this case they would need to provide Absorb with the Certificates and/or Competencies files. Please note that Certificates must be set up, either within the Courses and/or Curriculums files, or manually through the Admin Interface, before importing the Certificates file. Competencies must be set up through the Admin Interface before importing the Competencies file.
 
Special Considerations - Course and Curriculum Enrollments:
Absorb supports the concept of multiple enrollments in a single course or curriculum. The IsActive field is used to control this. Generally, if a client is planning to import only a single enrollment per user and course pair then this value should be set to 1 (Active). 
 
However, if a client’s user had taken the course twice, then the client would likely set the oldest enrollment record to 0 (Historical) and the most recent enrollment record to 1 (Active). It is important to point out that historical enrollments are available for reporting only and are not displayed on the learner interface. For this reason it is very important that those enrollments that should be deemed as ‘active’ are correctly set to 1 when and where appropriate.
 

Absorb Assessments:

Question and Answers 
Questions and answers for use in either Absorb assessments (e.g. Exams and Quizzes) and/or surveys can also be imported. To accomplish this, both the Questions and Answers files are required. Absorb clients also have the option to provide the Question Banks file to group questions for use in multiple assessment scenarios.
 

Data Validation

Before providing your files to Absorb please take a moment to review and ensure they pass these checks:
  • All External IDs must be unique. 

  • All referenced entities do actually exist where one file refers to an entity in another file (e.g. User file referring to a Department_ExternalId from the Department file). References should either occur in another file or have been previously been set up through the Admin Interface.

  • All special characters (e.g. © ● ™ etc.) are represented correctly in the files. (Some software and/or export processes can sometimes automatically replace these with incorrect characters).

  • All dependencies have been checked and verified. (For example if enabling certificate expiry for a course (Certificate_ExpireType) also ensure the issuance of a certificate for that course (Certificate_IsEnabled) has also been enabled).

  • All active / inactive flags (i.e. fields like ActiveStatus, IsActive, etc.) are set correctly. (Double check documentation to ensure fields are set to 1 or 0 as appropriate.).

  • All column headers are correct. Confirm that they have no spaces and match the data mapping document (e.g. “FirstName”).

  • All field values are absolutely accurate. Double check data in a text editor like Notepad to confirm; as programs like Excel may remove leading 0’s, truncate very long numbers, and may generally change your original data.
 

File Set Examples

The following examples are meant to illustrate which combinations of files you might provide us with. Please note that clients may elect to choose a different combination of files which is acceptable, as long as they observe the dependencies noted in the previous section.
 
(See the next section for samples of actual file content)
 
Example 1 – Basic: 
  • Departments
  • Users
  • Courses / Curriculums / Course Bundles
  • Course Enrollments / Curriculum Enrollments
Example 2 – Advanced User: 
  • Departments
  • Users
  • Role Assignments
  • Department Admins / Group Admins
  • Courses / Curriculums / Course Bundles
  • Course Enrollments / Curriculum Enrollments
Example 3 – Instructor Led Courses: 
  • Departments
  • Users
  • Courses / Curriculums / Course Bundles
  • Sessions
  • Classes
  • Venues
  • Course Enrollments / Curriculum Enrollments
Example 4 – Advanced Learner Activity: 
  • Departments
  • Users
  • Courses / Curriculums / Course Bundles
  • Lessons
  • Course Enrollments / Curriculum Enrollments 
  • Enrollment Activities
  • Certificates / Competencies


Appendix: Data Examples

Examples of the pre-importation ‘working’ data for some of the more popular import files are shown below.

Departments:

Spreadsheet

 
Text
Name,ExternalId,Parent_ExternalId
Demo Organization,00001,
Sales,00002,00001
Support,00003,00001
Human Resources,00004,00001
North America Sales,00005,00002
European Sales,00006,00002
  

Users:

Spreadsheet

 
Text
Username,Password,FirstName,LastName,ActiveStatus,Department_ExternalId,ExternalId,IsAdmin,IsLearner,EmailAddress,CustomField,Supervisor_ExternalId
john.doe@demo.com,Password5467,John,Doe,0,00002,100012,1,1,john.doe@demo.com,Custom Value,
jane.doe@demo.com,Password5812,Jane,Doe,0,00002,100013,0,1,jane.doe@demo.com,Custom Value 2,100012

 

Courses:

Spreadsheet

 
Text 
Name,ExternalId,TypeName,ActiveStatus,Description,Certificate_IsEnabled,ToggleEmails,SendNewEnrollmentEmail,SendEnrollmentCompletionEmail,AllowReEnroll,AllowFailure
Marketing 101,901,OnlineCourse,0,Course description 1,0,1,1,1,1,1
Introduction to Calculus,902,InstructorLedCourse,0,Course description 2,1,1,1,0,,1

 

Course Enrollments:

Spreadsheet

 
Text
User_ExternalId,Course_ExternalId,IsActive,Status,DateStarted,DateCompleted,Score,Credits,AcceptedTermsAndConditions,DateExpires,Session_ExternalId
100012,901,1,1,2015/06/05,,,,1,2016-06-05,
100012,901,0,3,2005/11/15,2006/01/06,70,3,1,,
100013,902,1,3,2010/04/10,2010/07/30,87,,1,,647

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

0 Comments

Article is closed for comments.