Retrieving move than 500 objects with LinqToLdap

By default, LinqToLdap sets the page size to be 500 objects, meaning standard queries will only return the first 500 objects.

This article builds on the article OrganizationalUnit CRUD and uses the same OrganizationalUnitObject and OrganizationalUnitObjectMap classes.

Before we start, it’s probably a good idea to make sure you’ve got a large number of OUs to play with. If you haven’t, you’ll have to adapt the code to work with objects you have got. Or wait until I write about users and groups. ADU&C will show all that there are: I suggest you go to the top of your domain and use the Find dialog to get a count. I had about 8,000.

So, what do we get if we don’t do anything special? Here’s the Program.cs to start with:

using System;
using System.Linq;
using LinqToLdap;

namespace BlogOuTest1
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new LdapConfiguration()
                .AddMapping(new OrganizationalUnitObjectMap());
            config.ConfigureFactory("dc1");

            var counter = 0;
            var context = new DirectoryContext();

            // ----- Start of section to replace -----
            var allOUsQuery = context.Query();

            var ous = from o in allOUsQuery
                      select o;

            foreach (var ou in ous)
            {
                counter++;
                Console.WriteLine("Ou: {0}",
                    ou.organizationalUnitName);
            }
            // ----- End of section to replace -----

            Console.WriteLine("Counter = {0}", counter);
            Console.WriteLine("\r\nPress a key to continue...");
            Console.ReadKey(true);
        }
    }
}

With this, I get a list of OUs and then

Counter = 500

Which was as expected because the default page size in LinqToLdap is 500. So, how do we get more?

Since the Active Directory default maximum page size, on the server, is 1000 we could increase the page size used by LinqToLdap. If you replace the config definition with the following, you’ll get 1000 objects returned:

    var config = new LdapConfiguration()
        .AddMapping(new OrganizationalUnitObjectMap())
        .MaxPageSizeIs(1000);

It’s better, but still not what we’re after. To get the rest, we’re going to have to get the server to page through the data. So, how do we do that in LinqToLdap? First, I’m going to revert to the original (LinqToLdap) default page size of 500 by removing the MaxPageSizeIs(1000) line.

Now, you have two choices. If you just want to retrieve all objects, you can use InPagesOf(n); if you want to process the objects as you retrieve them, you can use ToPage().

InPagesOf() – retrieving the objects in one pass

You can use a query like this to retrieve all of the objects in one pass:

using System;
using System.Collections.Generic;
using System.DirectoryServices.Protocols;
using System.Linq;
using LinqToLdap;
using LinqToLdap.Collections;

namespace BlogOuTest3
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new LdapConfiguration()
                .AddMapping(new OrganizationalUnitObjectMap());
            config.ConfigureFactory("dc1");

            var context = new DirectoryContext();

            // ----- Start of section to replace -----

            var ous = context.Query<OrganizationalUnitObject>()
                .Select(u => u.organizationalUnitName)
                .InPagesOf(10);

            var counter = 0;
            foreach (var ou in ous)
            {
                counter++;
                Console.WriteLine("Ou: {0}",ou);
            }

            // ----- End of section to replace -----

            Console.WriteLine("Counter = {0}", counter);

            Console.WriteLine("\r\nPress a key to continue...");
            Console.ReadKey(true);
        }
    }
}

I’ve chosen a silly page size for this one, so it’ll page even if you’ve only a handful of OUs. There is a scenario where this won’t work and it’s one that had me scratching my head for quite a while. So far, I’ve been chosing the root container (the domainDNS container, otherwise known as the defaultNamingContext) of my domain as the NamingContext on the ClassMaps, for example:

NamingContext("DC=big,DC=wooden,DC=badger");

But for searches to work against the root container, you need another search control:

SearchOptionsControl(SearchOption.DomainScope)

So the query statement should look like this:

var ous = context.Query<OrganizationalUnitObject>()
    .WithControls(new[] { new SearchOptionsControl(SearchOption.DomainScope) })
    .Select(u => u.organizationalUnitName)
    .InPagesOf(10);

I eventually found this here. It doesn’t say why and I still haven’t worked that out. I’ve run individual searches against every OU and container in the root container and they all work without the control, so it’s something to do with searching the root container itself. I’ll update if I ever sort it out.

ToPage() – processing objects in pages

You can use ToPage() in a query to process the objects in pages as you retrieve them. Here’s an example. First, I’ve added a method to the Program class:

   private static int ProcessPage(int counter, ILdapPage<string> page)
    {
        if (page.Count > 0)
        {
            foreach (var ou in page.ToList())
            {
                counter++;
                Console.WriteLine("Ou: {0}", ou);
            }
            System.Threading.Thread.Sleep(500);
        }

        return counter;
    }

All it does is display the results and increment a counter, then sleep for half a sec so you can see that it’s happening page by page.

Then, I’ve replaced the section between the comments, in the code sample above, with the following:

    // ----- Start of section to replace -----

    var counter = 0;

    ILdapPage<string> page = context.Query<OrganizationalUnitObject>()
        .WithControls(new[] { new SearchOptionsControl(SearchOption.DomainScope) })
        .Select(o => o.distinguishedName)
        .ToPage(10);

    counter = ProcessPage(counter, page);

    while (page.HasNextPage)
    {
        var pageSize = page.PageSize;
        var nextPage = page.NextPage;

        page = context.Query<OrganizationalUnitObject>()
            .WithControls(new[] { new SearchOptionsControl(SearchOption.DomainScope) })
            .Select(o => o.distinguishedName)
            .ToPage(pageSize, nextPage);

        counter = ProcessPage(counter, page);
    }

    // ----- End of section to replace -----

When you run the code you should see the results appear a page at a time.