Crm Minds

Hey everyone

In this post I want to talk about something that comes up regularly on integration and migration projects: sending requests to Dataverse in parallel. It sounds simple, but there are a few things you need to get right to avoid broken connections, corrupted data, and hitting service protection limits. I will walk through the concept, show you working code for both modern .NET and .NET Framework, cover the connection settings most people forget, and end with the one rule you absolutely cannot break.

The Problem with Doing Everything One at a Time

Picture this scenario. You have just been handed a data migration project. A customer is moving off a legacy CRM system and the target is Dataverse. The migration tool exports about 200,000 contact records. You write a loop that reads each record and calls service.Create() one at a time.

You run it. The first few hundred records come through fine. Then you check the ETA and realize it is going to take fourteen hours.

This is a pattern I have seen on almost every migration project I have worked on. The sequential approach is safe and easy to reason about, but it wastes the capacity that Dataverse is designed to support.

What Parallelism Means

Parallelism means doing multiple things at the same time using multiple threads, rather than doing them one after another.

Think of it like a checkout queue at a supermarket. If there is only one till open, every customer waits for the person in front of them to finish. If five tills are open at the same time, five customers are served simultaneously and the queue clears much faster. The work per customer does not change, but the total time drops significantly.

In code, instead of one thread sending a request and waiting for the response before sending the next, you send ten requests at the same time across ten threads. While one response is in flight over the network, the others are too.

Why Dataverse Specifically Benefits

Dataverse runs on Azure across multiple servers. It is architected to handle many concurrent users at the same time. A production environment with many licensed users has a lot of allocated server capacity sitting behind a load balancer.

When you send requests sequentially from a single thread, you are using a tiny fraction of that capacity. Most of your time is spent waiting for network round-trips. A single Dataverse request might take 100-200 milliseconds end-to-end. If you have 100,000 records to process, that is three to five hours of waiting for the network.

Send those same requests across ten threads and the elapsed time drops by close to that same factor. The Dataverse service is not the bottleneck. Your own sequential client is.

The Connection Problem: Why You Cannot Share One Client

Before you reach for Parallel.ForEach and wrap it around your existing code, there is something you need to understand about the Dataverse SDK clients.

ServiceClient and CrmServiceClient are not thread-safe when shared across threads. If you create one instance and use it from multiple threads at the same time, you will get corrupted state, dropped connections, and unpredictable errors.

The SDK is designed this way because each client instance maintains its own internal connection state, authentication context, and request pipeline. Sharing that state across threads without synchronization causes race conditions.

The correct approach differs depending on which SDK client and .NET version you are using.

Approach 1: ServiceClient with .NET 6 or Higher

If you are on .NET 6 or higher, ServiceClient exposes async methods like CreateAsync, UpdateAsync, and ExecuteAsync. Combined with Parallel.ForEachAsync, you can fan out across multiple threads cleanly.

ServiceClient is designed to be safe for concurrent async operations. You use a single authenticated instance and call the async methods from multiple concurrent tasks. The client handles thread safety internally for the async pattern.

Before running the parallel loop, set EnableAffinityCookie = false. By default, Dataverse uses a sticky session cookie that routes all your requests to the same backend server. When you are sending in parallel, you want your requests distributed across all eligible servers. Disabling affinity spreads the load and reduces the chance of hitting per-server service protection limits.

Use RecommendedDegreesOfParallelism to cap concurrency. This property reads the x-ms-dop-hint response header, which Dataverse uses to tell you how many concurrent requests the environment can comfortably absorb. The value changes depending on how the environment is provisioned. Let Dataverse guide you rather than hardcoding a number.

Use ConcurrentBag<T> to collect results. Because multiple threads are writing results at the same time, you need a thread-safe collection. A regular List<T> will corrupt under concurrent writes.

Here is an example of a bulk create job migrating records from a legacy system:

public static async Task<Guid[]> MigrateContactsAsync(
    ServiceClient serviceClient,
    List<Entity> contactsToMigrate)
{
    // Disable affinity so requests are spread across all backend servers
    serviceClient.EnableAffinityCookie = false;
    var createdIds = new ConcurrentBag<Guid>();
    var parallelOptions = new ParallelOptions
    {
        // Use the environment's recommended concurrency level
        MaxDegreeOfParallelism = serviceClient.RecommendedDegreesOfParallelism
    };
    await Parallel.ForEachAsync(
        source: contactsToMigrate,
        parallelOptions: parallelOptions,
        async (contact, cancellationToken) =>
        {
            Guid id = await serviceClient.CreateAsync(contact, cancellationToken);
            createdIds.Add(id);
        });
    return createdIds.ToArray();
}

Notice that createdIds is a ConcurrentBag<Guid>, not a List<Guid>. The order of IDs returned will not match the order of the input list. If you need to map results back to source records, include a unique external identifier in each entity before creating it, or track the mapping inside the loop.

For operations that are not bulk creates but require custom logic per record, you can call ExecuteAsync the same way:

await Parallel.ForEachAsync(
    source: accountsToUpdate,
    parallelOptions: parallelOptions,
    async (account, cancellationToken) =>
    {
        var request = new UpdateRequest { Target = account };
        await serviceClient.ExecuteAsync(request, cancellationToken);
    });

Approach 2: CrmServiceClient with .NET Framework

If you are on .NET Framework and using CrmServiceClient, the async approach above is not available. Instead, you use Parallel.ForEach with the Clone() method.

Clone() creates a new authenticated CrmServiceClient instance that shares the same underlying credential context but has its own independent connection. Each thread gets its own clone and uses it exclusively. This is the correct way to get a dedicated, thread-safe connection per thread.

The clone must be disposed when the thread’s work is done. The overload of Parallel.ForEach that supports local state handles this correctly: initialize the clone in the thread-local initializer, use it in the body, and dispose it in the thread-local finalizer.

public static Guid[] MigrateContacts(
    CrmServiceClient crmServiceClient,
    List<Entity> contactsToMigrate)
{
    // Disable affinity cookie before starting
    crmServiceClient.EnableAffinityCookie = false;
    var createdIds = new ConcurrentBag<Guid>();
    Parallel.ForEach(
        source: contactsToMigrate,
        parallelOptions: new ParallelOptions
        {
            MaxDegreeOfParallelism = crmServiceClient.RecommendedDegreesOfParallelism
        },
        localInit: () =>
        {
            // Each thread gets its own cloned connection
            return crmServiceClient.Clone();
        },
        body: (contact, loopState, threadLocalClient) =>
        {
            Guid id = threadLocalClient.Create(contact);
            createdIds.Add(id);
            return threadLocalClient;
        },
        localFinally: (threadLocalClient) =>
        {
            // Always dispose cloned instances when the thread finishes
            threadLocalClient?.Dispose();
        });
    return createdIds.ToArray();
}

Do not skip the localFinally dispose. Every cloned client holds a live connection. If you do not dispose it, you leak connections until garbage collection eventually cleans them up, which can cause connection pool exhaustion during long-running jobs.

Connection Settings You Need to Configure

Most developers write the parallel loop correctly and then wonder why throughput is still poor. The answer is usually connection-level settings that throttle the number of concurrent outbound connections at the .NET or OS level.

Set these before you start sending requests:

// Allow the thread pool to ramp up quickly instead of waiting for its default algorithm
ThreadPool.SetMinThreads(100, 100);
// .NET Framework: raise the default connection limit from 2 to something that matches your parallelism
System.Net.ServicePointManager.DefaultConnectionLimit = 65000;
// Do not wait for a round-trip confirmation before sending the request body
System.Net.ServicePointManager.Expect100Continue = false;
// Disable Nagle algorithm: sends packets immediately instead of buffering small packets together
System.Net.ServicePointManager.UseNagleAlgorithm = false;

The most impactful one on .NET Framework is DefaultConnectionLimit. Its default value is 2. That means even if you have 10 parallel threads, only 2 of them can have an open outbound TCP connection at the same time. The other 8 queue behind them. This effectively serializes your parallel work at the network layer. Raise it to a large number so connections are never the bottleneck.

On .NET Core and .NET 6+, ServicePointManager is mostly deprecated. The default connection limit is effectively unlimited (int.MaxValue). The ThreadPool.SetMinThreads call still applies and helps the thread pool respond faster under burst load.

Degree of Parallelism: Let Dataverse Guide You

Dataverse enforces service protection limits at the API level. One of those limits is the number of concurrent requests per user, per server. The default threshold is 52 concurrent requests. If you exceed it, you get this error:

Error Code: -2147015898
Message: Number of concurrent requests exceeded the limit of 52.

Do not hardcode a thread count. Use RecommendedDegreesOfParallelism on either ServiceClient or CrmServiceClient. This property reflects the x-ms-dop-hint value returned by Dataverse in response headers, which is calculated based on the actual resource allocation of the environment at that moment. Production environments with many licensed users typically return a higher value than development sandboxes.

If you do hit the concurrent request limit, reduce the MaxDegreeOfParallelism value. Disabling the affinity cookie also helps because the limit applies per server. When requests are distributed across all backend servers, each server sees fewer concurrent requests from your client.

Real-World Use Cases

Here are the scenarios where I have used parallel requests most often in projects.

Bulk data migration. Migrating tens of thousands of records from a legacy system into Dataverse is the most common case. The data is already prepared and validated; you just need to push it in. Parallelism typically cuts a multi-hour job down to under thirty minutes.

Nightly enrichment sync. A customer had a scheduled Azure Function that read about 50,000 account records from Dataverse, called an external service to fetch enriched company data, and wrote the results back. Each account needed one outbound API call and one Dataverse update. Running both the external API calls and the Dataverse writes in parallel dropped the nightly run from four hours to under forty minutes.

Mass recalculation jobs. When business logic changes require reprocessing existing records, for example recalculating scores or resetting statuses, parallel execution of custom actions or updates makes it practical to run these as a one-time operation rather than a scheduled job spread over days.

Parallel read-heavy reporting queries. If you are building a report that requires aggregating data from multiple unrelated tables, you can fire those queries in parallel and combine the results in memory, rather than running each query sequentially.

The One Hard Rule: Never Inside a Plugin

This is not a guideline. It is a constraint enforced by the platform.

Plugins and custom workflow activities run inside a database transaction managed by the Dataverse sandbox. The transaction is a single, ordered unit of work. The sandbox is designed for sequential execution within that transaction boundary.

If you introduce multiple threads inside a plugin using Task.Run, Parallel.ForEach, Thread, or any other mechanism, you corrupt the transaction. Multiple threads compete to update shared pipeline objects. The result is errors like:

Generic SQL error.
The transaction active in this session has been committed or aborted by another session.

Non-thread-safe collections like those in System.Collections (not System.Collections.Concurrent) also become corrupted when accessed from multiple threads.

The rule is simple: never use parallel execution inside a plugin or custom workflow activity. No Parallel.ForEach. No Task.WhenAll. No background threads. If you need to do work in parallel, that work belongs in a client application, an Azure Function, or a console app that calls Dataverse from outside the plugin context.

Pros and Cons

Advantages:

• Dramatically reduces elapsed time for bulk operations. Throughput improvements of 5x to 10x are realistic.
• Makes large migration and sync jobs practical without overnight windows.
• Uses the server capacity that Dataverse already has allocated for your environment.
• Works with standard SDK patterns and does not require custom infrastructure.

Disadvantages:

• Harder to reason about errors. If one thread fails, others are still running. You need explicit error handling per item, not a single try/catch around the loop.
• Order of operations is not guaranteed. If the sequence in which records are created matters, parallel execution is not appropriate.
• Concurrency limit applies. Exceeding 52 concurrent requests returns a service protection error. You need to handle this or stay within the recommended degree of parallelism.
• Not usable inside plugins. This restriction eliminates the pattern from a large portion of Dataverse customization code.
• More complex code. The clone pattern for .NET Framework and the ConcurrentBag requirement for result collection add cognitive overhead.

Wrapping Up

Parallel requests are one of the most impactful performance tools available when you are building integration tools, migration jobs, or any client-side automation that touches large amounts of Dataverse data. The key points to remember are: use ServiceClient with Parallel.ForEachAsync on modern .NET, use Clone() per thread on .NET Framework, disable the affinity cookie, respect the recommended degree of parallelism, configure your connection settings, and never bring any of this into a plugin.