View Categories

Bulk API Delta Replication All Objects

6 min read

ss_DeltaBAll allows you to run ss_DeltaB in bulk across multiple Salesforce objects.

It follows the same inclusion/exclusion model as ss_DeltaAll, but each object is processed using ss_DeltaB and Salesforce Bulk API 2.0.

The optional @BAPIMethod parameter allows the Bulk API execution method to be applied consistently across all objects in scope, while @AutoReplica controls whether individual Delta operations are allowed to fall back to a full replication where required.

Simple Example

exec ss_DeltaBAll 'DEMO'

By default, ss_DeltaBAll uses:

@AutoReplica = 'No'
@BAPIMethod = 'WAIT'

Each object is therefore Delta replicated sequentially, with SQL-Sales waiting for the current object to complete before moving to the next.

With @ObjectNameFrom

exec ss_DeltaBAll 'DEMO', 'Opportunity'

This will commence the bulk ss_DeltaB replication from the Object immediately following Opportunity, which could be OpportunityCompetitor if there is no other custom or standard object with a name closer to Opportunity when sorting alphabetically.

This is useful if a prior run of ss_DeltaBAll has failed or been stopped and you wish to recommence from the point of failure.

With AutoReplica = Full

exec ss_DeltaBAll 'DEMO', null, 'Full'

@AutoReplica is passed through to each attempted ss_DeltaB.

When set to Full, an individual Delta operation is permitted to fall back to a full replication when an AutoReplica condition is encountered.

Examples include an object which cannot technically be Delta replicated, missing prior replication information, metadata changes which require a rebuild, or another condition where SQL-Sales determines that a full replication is required.

Parameters

ParameterPurpose
@EnvSQL Sales Environment Name
@ObjectNameFromCommences the bulk ss_DeltaB replication from the Object immediately following the one passed in. This is useful if a prior ss_DeltaBAll run has failed or been stopped and you wish to recommence from that point.
@AutoReplicaOptional. Passed through to each ss_DeltaB call. No is the default. Full permits an individual Delta to fall back to full replication where an AutoReplica condition is encountered.
@BAPIMethodOptional Bulk API execution method passed through to each ss_DeltaB. Supported methods are WAIT, WAIT(n), BACK and CHECK(n). Default is WAIT.

Bulk API Execution Methods

ss_DeltaBAll passes the selected Bulk API execution method through to each individual ss_DeltaB call.

MethodPurpose
WAITProcesses each object sequentially. SQL-Sales waits for the current Delta Bulk API Query job to complete before moving to the next object.
WAIT(n)Sequential resilient wait mode. The JobId for each object is recorded immediately and SQL-Sales checks the job every n seconds until complete before moving to the next object. Valid values are 1–3600 seconds.
BACKSubmits Delta Bulk API Query jobs for all eligible objects in scope without waiting for each job to complete.
CHECK(n)Processes previously submitted outstanding Delta Bulk API Query jobs, checking every n minutes until each completes. Valid values are 1–59 minutes.

GET is intentionally not supported at the ss_DeltaBAll level. A one-shot GET is more appropriate when working with an individual object through ss_DeltaB.

WAIT

WAIT is the default method.

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'WAIT'

Each object is fully processed before SQL-Sales proceeds to the next object.

For each object, SQL-Sales determines the required Delta scope, submits the Bulk API Query job, waits for Salesforce to complete it, then applies the resulting Inserts, Updates and Deletes to the existing SQL replica.

WAIT(n)

WAIT(n) provides the resilient synchronous execution model across all objects.

For example:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'WAIT(10)'

For each object, SQL-Sales submits the Delta Bulk API Query job, records the JobId immediately and checks the job until Salesforce reports that it is complete.

The value of n is the number of seconds between subsequent checks and can be between 1 and 3600.

Once the current object has completed, ss_DeltaBAll proceeds to the next object.

For example, to check each running job every five minutes:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'WAIT(300)'

BACK

BACK allows Delta Bulk API Query jobs to be submitted across all eligible objects without waiting for each object to complete before the next job is created.

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'BACK'

SQL-Sales iterates through the objects in scope and submits a Delta Bulk API Query job for each eligible object.

The corresponding JobIds are recorded in ss_BulkAPILog.

Because each submission returns without waiting for Salesforce processing to complete, multiple Bulk API jobs can be outstanding at the same time.

A common pattern is therefore:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'BACK'

followed later by:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'CHECK(5)'

CHECK(n)

CHECK(n) is the companion method to a prior BACK execution.

For example:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'CHECK(5)'

For each object in scope, SQL-Sales identifies the applicable outstanding Delta Bulk API Query job.

If the job is not yet complete, SQL-Sales waits the specified number of minutes and checks again.

Once the job completes, the results are retrieved and the Delta is applied before SQL-Sales moves to the next object.

The value of n can be between 1 and 59 minutes.

Although a prior BACK may have submitted multiple jobs to Salesforce, CHECK(n) processes the outstanding jobs serially as ss_DeltaBAll iterates through the objects.

Recommended detached way of working

For larger Delta runs across many Salesforce objects, the following pattern allows the Salesforce Bulk API jobs to be submitted first and completed afterward.

First submit the jobs:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'BACK'

Then, either immediately or later, process the outstanding jobs:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'CHECK(5)'

This allows Salesforce to begin processing multiple submitted jobs without requiring the first object to complete before the next job is created.

Combining AutoReplica with Bulk API methods

@AutoReplica and @BAPIMethod can be used together.

For example:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'WAIT(10)'

In this example each object is processed sequentially using resilient Bulk API polling, while an individual object is permitted to fall back to full replication if required.

A detached run can similarly be started with:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'BACK'

and subsequently completed with:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'CHECK(5)'

The same @AutoReplica setting should be used consistently between the submission and subsequent processing of the run.

ss_BulkAPILog

ss_DeltaBAll uses the same ss_BulkAPILog lifecycle as individual ss_DeltaB operations.

Each submitted Bulk API Query job is recorded against the relevant Salesforce Object and target SQL replica table.

This allows a later CHECK(n) execution to identify the correct outstanding Delta job for each object.

For normal operation there is no need to manually supply Bulk API JobIds when using ss_DeltaBAll.

ss_ObjectInclusion inclusion table

FieldPurpose
ObjectNameDefines Objects you wish to include in each ss_DeltaBAll run. Only Objects contained in this table will be considered in scope. Accompanying table ss_ObjectExclusion is additionally referenced, therefore Objects contained there will be ignored.

The inclusion model is the same as conventional ss_DeltaAll.

Generally, where only a defined set of Salesforce objects needs to be maintained, ss_ObjectInclusion provides the simplest and most controlled approach.

ss_ObjectExclusion exclusion table

FieldPurpose
ObjectNameAdd Objects to this table to have them excluded from each ss_DeltaBAll run. On deployment, ss_EnableDatabase populates a number of known problem Objects together with their Salesforce API response details for reference.
ErrorMsgSalesforce API Response Error Msg
ErrorCodeSalesforce API Response Error code

It is a matter of choice whether you define an inclusion set using ss_ObjectInclusion or instead exclude unwanted objects through ss_ObjectExclusion.

Generally, only a relatively small number of Salesforce objects are required to maintain a working replication database, in which case ss_ObjectInclusion is usually the more appropriate approach.

Alternatively, where you wish to maintain most or all available Salesforce objects, exclusions can instead be maintained through ss_ObjectExclusion.

Choosing an execution method

For a straightforward sequential Delta run (default WAIT):

exec ss_DeltaBAll 'DEMO'

For resilient sequential execution:

exec ss_DeltaBAll 'DEMO',
    @BAPIMethod = 'WAIT(10)'

For a sequential run which may automatically fall back to full replication where required:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'WAIT(10)'

For larger detached Delta runs:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'BACK'

followed later by:

exec ss_DeltaBAll 'DEMO',
    @AutoReplica = 'Full',
    @BAPIMethod = 'CHECK(5)'

For most smaller or routine runs, WAIT or WAIT(n) will be the simplest approach. For larger multi-object Delta runs, BACK followed by CHECK(n) allows the Salesforce job-processing stage to be detached from the original submission process.