Change mutex to counting semaphore

The mutex in zuul is great but is limited to run one job at the same
time. Some use cases like using a limited number floating licenses in
jobs cannot be handled with this. Thus this changes the mutex
functionality to a counting semaphore (which defaults to 1).

This is a port of Ida589e49bc6694f4ccc4c586e0d43b391b8c3ae4 to zuulv3
branch.

Change-Id: Icf4013a6215e2b10ca8e6309928b9e5881dda02c
diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst
index e4ce737..d41ed89 100644
--- a/doc/source/zuul.rst
+++ b/doc/source/zuul.rst
@@ -644,10 +644,12 @@
   would largely defeat the parallelization of dependent change testing
   that is the main feature of Zuul.  Default: ``false``.
 
-**mutex (optional)**
-  This is a string that names a mutex that should be observed by this
-  job.  Only one build of any job that references the same named mutex
-  will be enqueued at a time.  This applies across all pipelines.
+**semaphore (optional)**
+  This is a string that names a semaphore that should be observed by this
+  job.  The semaphore defines how many jobs which reference that semaphore
+  can be enqueued at a time.  This applies across all pipelines in the same
+  tenant.  The max value of the semaphore can be specified in the config
+  repositories and defaults to 1.
 
 **branch (optional)**
   This job should only be run on matching branches.  This field is
@@ -850,6 +852,21 @@
 or specified in the project itself, the configuration defined by
 either the last template or the project itself will take priority.
 
+
+Semaphores
+""""""""""
+
+When using semaphores the maximum value of each one can be specified in their
+respective config repositories.  Unspecified semaphores default to 1::
+
+  - semaphore:
+      name: semaphore-foo
+      max: 5
+  - semaphore:
+      name: semaphore-bar
+      max: 3
+
+
 logging.conf
 ~~~~~~~~~~~~
 This file is optional.  If provided, it should be a standard