Bad things, bad things happen … explosions, tears … and maybe more

        var numbers = new[] {1, 2, 3, 4, 5};
        Parallel.ForEach( numbers, (number) =>
          {
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception( "Boom!");
            }
          }

In the example above, the code will explode because of the exception.
So obviously we will add a try catch block…

try
      {
        var numbers = new[] {1, 2, 3, 4, 5};
        Parallel.ForEach( numbers, (number) =>
          {
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception( "Boom!");
            }
          }
        );
      }
      catch (Exception e)
      {
        Console.WriteLine( $"Caught exception! {e.Message}");
      }

And that will work, (of course)

But what if we want to run an async function in parallel … then what do we do?

      try
      {
        var numbers = new[] { 1, 2, 3, 4, 5 };
        Parallel.ForEach(numbers, async (number) =>
          {
            
Console.WriteLine($"Working on number: {number}");
            
            if (number == 3)
            {
              throw new Exception("Boom!");
            }
          }
        );
      }
      catch (Exception e)
      {
        Console.WriteLine($"Caught exception! {e.Message}");
      }

In the case above you fire tasks, but don’t really wait for them to complete.
The async and await might give you the feeling that you are doing something truly in parallel … but you are not
Because Parallel.ForEach has no overload accepting a Func<Task>, it accepts only Action delegates.

The easy way out is to use Task as they were intended

try
      {
        // use a concurent queue so all the thread can add
        // while it is an overhead we do not expect to have that many exceptions in production code.
        var exceptions = new ConcurrentQueue<Exception>();
        var numbers = new[] { 1, 2, 3, 4, 5 };
        var tasks = new List<Task>();

        async Task t(int number)
        {
          // protect everything with a try catch
          try
          {
            await Task.Delay(100).ConfigureAwait(false);
            Console.WriteLine($"Working on number: {number}");
            if (number == 3)
            {
              throw new Exception("Boom!");
            }
          }
          catch (Exception e)
          {
            // save it for later.
            exceptions.Enqueue(e);
          }
        }

        foreach (var number in numbers)
        {
          tasks.Add( t(number) );
        }

        Task.WaitAll(tasks.ToArray());
         
        // we are back in our own thread
        if (exceptions.Count > 0)
        {
          throw new AggregateException( exceptions );
        }
      }
      catch (Exception e)
      {
        Console.WriteLine($"Caught exception! {e.Message}");
      }

Now obviously this is ugly code, but you get the idea … run all the tasks in parallel and wait for them all to finish.

Once they are all done, you can throw an aggregation of errors…. if there are any

Have a look at the code on my github page for more information/sample