Category: development

Myoddweb.DirectoryWatcher 0.2.0: six years on, still watching

development Myoddweb Directorywatcher No Comments

It has been six years since the last release of Myoddweb.DirectoryWatcher, my high-performance, asynchronous file and directory watcher for .NET and C++. Version 0.1.9 went out in August 2020, and then, as these things go, life happened. The library kept quietly doing its job in a couple of my own projects, but it didn’t get the attention it deserved.

This week I finally sat down and gave it the update it was owed. Version 0.2.0 is out now, both on GitHub and on NuGet.

Why this library exists

If you’ve ever used .NET’s built-in FileSystemWatcher, you’ll know it’s… fine, until it isn’t. It has a fixed internal buffer that silently overflows under heavy load, it happily throws exceptions that can bring your whole app down, and it really doesn’t enjoy being pointed at a large volume or a UNC path.

Myoddweb.DirectoryWatcher was built to fix exactly that:

  • Non-blocking, asynchronous events — a slow consumer never stalls the watcher.
  • No buffer overflow crashes, even under bursts of file activity.
  • Duplicate suppression, so three rapid writes to the same file become one clean notification instead of three.
  • Volume-wide monitoring of creates, deletes, touches and renames, across multiple paths at once.
  • Built-in statistics and logging, so you can see throughput and event rates without instrumenting it yourself.

Under the hood it’s a native C++ core built on Win32’s ReadDirectoryChangesW, wrapped in a clean, platform-agnostic managed interface. That split is what let this update focus so heavily on the native side without touching the public API.

What’s new in 0.2.0

This release is best described as “getting the house back in order” — it’s mostly stability, correctness and modernisation, not new features. But there’s a lot of it. A few highlights from the full changelog:

Toolchain caught up to 2026:

  • Solution and projects now build with Visual Studio 2022 (toolset v143).
  • The managed libraries target .NET Framework 4.6.2, .NET Standard 2.0 and .NET 8.0 — the old .NET Framework 4.5.2 and .NET Core 3.0 targets, both long out of support, are gone.
  • Tests moved to NUnit 4.6.1 and Google Test 1.18.0.
  • A GitHub Actions workflow replaces the now-retired Travis CI build.

A genuine bug fix, not just busywork:

  • Issue #20 — files could be silently missed when a whole populated folder was copy-pasted into a watched tree. That’s now fixed and covered by a new test.

A pile of native-code hardening:

  • An intermittent WorkerPool deadlock that could hang indefinitely.
  • A race condition in MonitorsManager::ready() for recursive monitors, where child monitors could report readiness before they’d actually started.
  • A use-after-free and a null-pointer dereference in the Windows-specific win::Data handling.
  • win::Data::process_error() no longer leaves a watched directory’s read loop silently stalled after a Win32 error.
  • A data race between a monitor’s in-flight update and its own cleanup task.
  • Several smaller ones: a leaking allocation in win::Data::clone(), a reserve() that should have been resize() in the logger, a null Monitor* dereference in MonitorsManager::start().

None of that changes how you use the library day to day, but it means the thing underneath is considerably more solid than it was in 2020 — particularly around startup/shutdown races and Windows error handling, which is exactly the kind of thing that only turns up under real-world load.

Using it

The public API hasn’t changed, so if you used an earlier version, this is a drop-in upgrade. If you’re new to it, here’s the gist.

Install it:

dotnet add package MyOddWeb.DirectoryWatcher

Watch a few paths and react when files show up:

using (var watch = new Watcher())
{
  watch.Add(new Request("c:\\", true));
  watch.Add(new Request("d:\\foo\\bar\\", true));

  watch.OnAddedAsync += async (f, t) =>
  {
    Console.WriteLine($"Added: {f.FullName}");
  };

  watch.Start();
}

Renames give you both the old and new name:

watch.OnRenamedAsync += async (f, t) =>
{
  Console.WriteLine($"{f.PreviousFullName} -> {f.FullName}");
};

And if you want to keep an eye on throughput, statistics are one line away:

watch.Add(new Request("c:\\", true, new Rates(50, 10000)));

watch.OnStatisticsAsync += async (s, t) =>
{
  Console.WriteLine($"{s.NumberOfEvents} events in the last {s.ElapsedTime}ms");
};

There’s a lot more in the README — logging levels, checking whether all your watch requests are actually ready, disposing cleanly, and so on.

What’s next

This is still Windows-only for now — the public interfaces are deliberately platform-agnostic, so a Linux or macOS backend is possible in principle, but it’s not on the immediate roadmap. For the moment, the goal was simply to bring the project back up to date and squash the bugs that had been sitting there since 2020.

If you use it and hit something, issues and PRs are very welcome — I’d rather it not take another six years for the next release.

Grab it from NuGet or GitHub.

Parallel.ForEach Async in C#

development No Comments

As mentioned in my previous post, to get a ‘proper’ parallel foreach that is async is a bit of a pain

So the solution is to write a true async function

public static async Task ForEach<T>(ICollection<T> source, Func<T, Task> body, CancellationToken token )
{
  // create the list of tasks we will be running
  var tasks = new List<Task>(source.Count);
  try
  {
    // and add them all at once.
    tasks.AddRange(source.Select(s => Task.Run(() => body(s), token)));

    // execute it all with a delay to throw.
    for (; ; )
    {
      // very short delay
      var delay = Task.Delay(1, token );

      // and all our tasks
      await Task.WhenAny( Task.WhenAll(tasks), delay).ConfigureAwait(false);
      if (tasks.All(t => t.IsCompleted))
      {
        break;
      }
      
      //
      // ... use a spinner or something
    }
    await Task.WhenAll(tasks.ToArray()).ConfigureAwait(false);

    // throw if we are done here.
    token.ThrowIfCancellationRequested();
  }
  finally
  {
    // find the error(s) that might have happened.
    var errors = tasks.Where(tt => tt.IsFaulted).Select(tu => tu.Exception).ToList();

    // we are back in our own thread
    if (errors.Count > 0)
    {
      throw new AggregateException(errors);
    }
  }
}

And you can call it …

await ParallelAsync.ForEach(number, async (numbers) =>
{
  // blah ... 
  
  // blah ....
  await DoSomethingAmazing( number ).ConfigureAwait(false);
}, CancellationToken.None).ConfigureAwait( false );

Of course, you can refine it by adding check for tokens that cannot be cancelled as well as empty sources

First prize you must make sure that the body of the ForEach takes in the token and cancels cleanly otherwise this will jump out with thread left up in the air… but at least it will get out.

Edit: As someone pointed out to me on StackOverflow there are a couple of subtle ways I can improve my implementation … so I added them here

What happens when you throw an exception in Parallel Loops

development 1 Comment

Bad things, bad things happen … explosions, tears … and maybe more

        var numbers = new[] {1, 2, 3, 4, 5};
        Parallel.ForEach( numbers, (number) =>
          {
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception( "Boom!");
            }
          }

In the example above, the code will explode because of the exception.
So obviously we will add a try catch block…

try
      {
        var numbers = new[] {1, 2, 3, 4, 5};
        Parallel.ForEach( numbers, (number) =>
          {
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception( "Boom!");
            }
          }
        );
      }
      catch (Exception e)
      {
        Console.WriteLine( $"Caught exception! {e.Message}");
      }

And that will work, (of course)

But what if we want to run an async function in parallel … then what do we do?

      try
      {
        var numbers = new[] { 1, 2, 3, 4, 5 };
        Parallel.ForEach(numbers, async (number) =>
          {
            
Console.WriteLine($"Working on number: {number}");
            
            if (number == 3)
            {
              throw new Exception("Boom!");
            }
          }
        );
      }
      catch (Exception e)
      {
        Console.WriteLine($"Caught exception! {e.Message}");
      }

In the case above you fire tasks, but don’t really wait for them to complete.
The async and await might give you the feeling that you are doing something truly in parallel … but you are not
Because Parallel.ForEach has no overload accepting a Func<Task>, it accepts only Action delegates.

The easy way out is to use Task as they were intended

try
      {
        // use a concurent queue so all the thread can add
        // while it is an overhead we do not expect to have that many exceptions in production code.
        var exceptions = new ConcurrentQueue<Exception>();
        var numbers = new[] { 1, 2, 3, 4, 5 };
        var tasks = new List<Task>();

        async Task t(int number)
        {
          // protect everything with a try catch
          try
          {
            await Task.Delay(100).ConfigureAwait(false);
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception("Boom!");
            }
          }
          catch (Exception e)
          {
            // save it for later.
            exceptions.Enqueue(e);
          }
        }

        foreach (var number in numbers)
        {
          tasks.Add( t(number) );
        }

        Task.WaitAll(tasks.ToArray());
         
        // we are back in our own thread
        if (exceptions.Count > 0)
        {
          throw new AggregateException( exceptions );
        }
      }
      catch (Exception e)
      {
        Console.WriteLine($"Caught exception! {e.Message}");
      }

Now obviously this is ugly code, but you get the idea … run all the tasks in parallel and wait for them all to finish.

Once they are all done, you can throw an aggregation of errors…. if there are any

Have a look at the code on my github page for more information/sample

Git authentification failed with no password prompt

development No Comments

Sometimes when trying to pull/push to a git repo, (on my own server or github), I get something like

fatal: Authentication failed for ‘https://….'”

Authentication failed

Now the problem is that I don’t even get asked for a username and password … sounds a bit stupid if you ask me.

On Windows 10:

  • Press Start, (the windows key)
  • Start typing “Credential Manager” or look for it in the control panel
  • Then select the Windows Credentials
  • Look for whatever website is the one that has your credentials
    Usually something like git:http://...
  • Remove the entry or edit it.
    If you remove it you will be asked for the credentials again.

Proudly powered by WordPress & Skyrocket Themes

MENU

Navigation