Data Imports - Configuration



This article covers the configuration of a one-time or scheduled data import to Absorb. The main Data Imports - An Overview article should also be reviewed for information on importing data to Absorb in general. Please note that these imports are an additional service that usually involve an additional fee and technical resources on both the Absorb and client sides. Interested? Please get in touch with your Client Success Manager. 

An Absorb representative will gather the necessary requirements in order to configure the import. The sections below describe the various options available and their defaults. If there is a specific feature you are interested in using, make sure to let your Absorb representative know.


Quick Start

There are quite a few options available, but here are the minimum pieces of input that we'll need from you in order to set things up:

Import Structure

Setup & configuration is handled by Absorb, but it's still useful to understand how each import is structured, especially for a scheduled import.

Each import consists of one or more "modules", each of which processes a different type of file when the import runs. For example you may have a single scheduled HRIS import consisting of one Users module and one Departments module. The import is scheduled to run nightly at 3am, at which point each of its modules (each with its own configuration), run one after the other. All department files would be processed (oldest to newest) followed by all users files (oldest to newest). Fig 1 below illustrates this:

Fig 1: Example of how an import is structured

The import itself holds minimal configuration, with each module holding the majority of it. This means you can have a single import with many modules, processing many files, with many sets of configuration. For example you may have a single import that processes two different Users files, each one handling things like default roles, user management, etc. in a different way.


Import Configuration

The import itself is configured with the following details, most notably the schedule (mandatory fields marked with *):

Import Name*
This is what our support staff will refer to the import as. E.g. "HRIS Import", "Users Import", etc.

This description is available to our support staff for future troubleshooting or updates. Any information you think is pertinent can be provided here. At a minimum, we'll usually request something like: "The purpose of this import is to keep user profiles in Absorb up to date based on changes made in a HRIS system on a nightly basis."

External Application*
This is the application that is providing the data (e.g. Workday, Salesforce, etc.). This is useful for our support staff to know.

Usually something like "nightly at 3am EST", but daily, weekly, monthly, quarterly, yearly, etc. are supported. Depending on the type and volume of data, the time of day may be restricted to very early in the morning North American Mountain Standard Time (e.g. 1-5am MST). Schedules are subject to approval by Absorb.

Module Configuration (Generic)

Each module has its own configuration. Some configuration is specific to the type of module (see Module Configuration (Specific)), but other configuration is generally available across all module types. For example every module has a mandatory 'Filename Starts With' option, every module has the same set of email notification options available, and every module must be configured to map columns in the import file to fields in Absorb. Below are the configuration options available across all modules (mandatory fields marked with *):


Module Name*
This is used to identify this specific module, and is included in any error/success emails. We will use "[[Module Name]] Import" by default.

Filename Starts With*
The import will process any file in the processing directory that matches this naming convention, processing in order from oldest to newest. A filename starting with something like CompanyName_Departments (in the case of a Departments file) is generally recommended; a full filename example would then be Company_Departments_20170606.csv. It's strongly recommended that filenames include a date and/or time to prevent duplicate filenames overwriting old files.

File Extension
Usually .csv or .txt

File Delimiter
How are records delimited in the file? The default is comma (') separated, but sometimes other characters like pipes (|) are used.

Date Format
If dates are included in your data (e.g. a Date Terminated value for Users), what format should we expect the values in? An example would be yyyy-MM-dd (e.g. 2017-08-24). Note that all date values in the file need to be in the same format, but different formats can be used across different files (assuming a different module is configured for each).

Delta File?
Will each file contain all records (full extract) or only new & updated records? By default we will treat each file as a full extract, automatically comparing it to the previous file and submitting only new & updated records. This means that if a change is made in Absorb it won't necessarily be overwritten when the next file is processed, unless that record in the file is different from the previous run's file.

Records to Process
The import can detect whether a record is new (no match in Absorb) or existing (match found in Absorb). The default is to process both new and existing records (All records), but you can also choose to process only new or only existing. This option does not apply to the following modules:

  • Enrollments
  • Activity Enrollments
  • Role Assignments
  • Group Assignments
  • Group Admins
  • Department Admins

Column Mapping

Each module will take in a different file type; for example the Users module will take in a file containing information about user profiles. When you provide sample files of each type, you will also need to let us know which Absorb fields should be mapped to each column in the files. For example you might provide something like the following mapping along with a sample Users file:

Column Header Absorb Field
Email Username
Email Email
Fname First Name
Lname Last Name
EmployeeNum External ID
EmployeeNum Employee Number
Title Job Title
SalesRegion Sales Region (custom field)

There are a few advanced options that can be specified along with the column mapping:

  • Default Values: Imagine you have a Job Title column in your file, but some records are missing a value. You might want to default the value in this case to something like "Associate". Taking it a step further, what if you don't have a Job Title column in your file, but you want to add "Associate" as a default value for all records? Default values let you do this - they will be inserted as the value for this field assuming we don't receive an actual value from the file.

  • Field Value Mapping: This allows you to effectively transform a certain value in your file into a different value before writing it to Absorb. A common example is changing a "Yes" in the "IsAdmin" column of your Users file into a "0", which is what Absorb accepts for that field. More creative uses would include mapping many different department values from your file into a single department in Absorb, replacing "bad" values with something more desirable, etc. 

Field Combining

Field combining is an optional feature that allows fields & strings to be combined and the output mapped to another field. An example is combining the FirstName and LastName columns to create a Username:

FirstName With String . With Field LastName Map Output To Username

(The above would result in Usernames like John.Doe)

There are a few rules for creating these combinations:

  • The combination must start with a Field (e.g. FirstName), not a string.
  • The first field can then be combined with either another field or a string.
    e.g. FirstName With Field EmployeeNumber       OR      FirstName With String -001
  • The result then can be further combined with other fields/strings, or the result mapped to a final field (e.g. FirstName With String . With Field LastName Map Output To Username)


Send Error Email
Send an email if the import encounters any errors. Individual record errors are provided as a text file attachment. Errors that prevented the import from running, such as failing to connect to a remote FTP or an inconsistent columns in the file, are also included here. Enabled by default and recommended to be left on.

Send Warning Email
Send an email if the import encounters any warnings. Warnings are generally related to records that were partially processed (e.g. the user was created, but their supervisor was not valid). Individual warnings are provided as a text file attachment. Enabled by default.

Send Success Email
Send an email if the import is successful (no errors or warnings). Can be used to create an audit trail in your email of each run. Disabled by default.

Send No Files Found Email
Send an email if their is no file found in the processing directory at the scheduled run time. Enabled by default.

Delivery Email
The emails above are sent to this address. Multiple emails can be specified, though we usually recommend providing a single email alias for a distribution list controlled by your team. This allows you to update recipients without having to update the import itself.

Module Configuration (Specific)

Each module (e.g. Departments, Users, etc.) will have additional configuration options, usually including at least a few mandatory ones, that will need to be reviewed and decided on. Check out our documentation for each object to review these options (Hint: Each one will have a Quick Start section noting the minimum items that need to be considered). 

Published on
Have more questions? Submit a request


Article is closed for comments.