Update Python guidelines: remove Python 2, modernize typing for 3.9+, promote keyword-only args over **kwargs

docs/python/design.md

@@ -64,7 +64,7 @@ Please contact the [Architecture board] for more guidance on non HTTP/REST based
 
 ### Supported python versions
 
-{% include requirement/MUST id="python-general-version-support" %} support Python 3.8+.
+{% include requirement/MUST id="python-general-version-support" %} support Python 3.9+.
 
 ## Azure SDK API Design
 
@@ -613,11 +613,11 @@ Here are some examples of namespaces that meet these guidelines:
 
 ### Async support
 
-The `asyncio` library has been available since Python 3.4, and the `async`/`await` keywords were introduced in Python 3.5. Despite such availability, most Python developers aren't familiar with or comfortable using libraries that only provide asynchronous methods.
+Despite the availability of the `asyncio` library and the `async`/`await` keywords, most Python developers aren't familiar with or comfortable using libraries that only provide asynchronous methods.
 
 {% include requirement/MUST id="python-client-sync-async" %} provide both sync and async versions of your APIs
 
-{% include requirement/MUST id="python-client-async-keywords" %} use the `async`/`await` keywords (requires Python 3.5+). Do not use the [yield from coroutine or asyncio.coroutine](https://docs.python.org/3.4/library/asyncio-task.html) syntax.
+{% include requirement/MUST id="python-client-async-keywords" %} use the `async`/`await` keywords. Do not use the [yield from coroutine or asyncio.coroutine](https://docs.python.org/3.4/library/asyncio-task.html) syntax.
 
 {% include requirement/MUST id="python-client-separate-sync-async" %} provide two separate client classes for synchronous and asynchronous operations.  Do not combine async and sync operations in the same class.
 
@@ -685,18 +685,14 @@ from azure.storage.blob.aio import BlobServiceClient # Async client
 
 {% include requirement/MUST id="python-packaging-follow-repo-rules" %} follow the specific package guidance from the [azure-sdk-packaging wiki](https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging)
 
-{% include requirement/MUST id="python-packaging-follow-python-rules" %} follow the [namespace package recommendations for Python 3.x](https://docs.python.org/3/reference/import.html#namespace-packages) for packages that only need to target 3.x.
+{% include requirement/MUST id="python-packaging-follow-python-rules" %} follow the [namespace package recommendations for Python 3.x](https://docs.python.org/3/reference/import.html#namespace-packages).
 
 {% include requirement/MUST id="python-general-supply-sdist" %} provide both source distributions (`sdist`) and wheels.
 
 {% include requirement/MUST id="python-general-pypi" %} publish both source distributions (`sdist`) and wheels to PyPI.
 
 {% include requirement/MUST id="python-general-wheel-behavior" %} test correct behavior for both CPython and PyPy for [pure](https://packaging.python.org/guides/distributing-packages-using-setuptools/#id75) and [universal](https://packaging.python.org/guides/distributing-packages-using-setuptools/#universal-wheels) Python wheels.
 
-{% include requirement/MUST id="python-packaging-nspkg" %} depend on `azure-nspkg` for Python 2.x.
-
-{% include requirement/MUST id="python-packaging-group-nspkg" %} depend on `azure-<group>-nspkg` for Python 2.x if you are using namespace grouping.
-
 {% include requirement/MUST id="python-packaging-init" %} include `__init__.py` for the namespace(s) in sdists
 
 #### Service-specific common library code
@@ -772,14 +768,17 @@ Only applications are expected to pin exact dependencies. Libraries are not. A l
 
 {% include requirement/MUST id="python-docstrings-all" %} provide docstrings for all public modules, types, constants and functions.
 
-{% include requirement/MUST id="python-docstrings-kwargs" %} document any `**kwargs` directly consumed by a method. You may refer to the signature of a called method if the `**kwargs` are passed through.
+{% include requirement/MUST id="python-docstrings-kwargs" %} document any `**kwargs` directly consumed by a method. If `**kwargs` are passed through to another API, you **must** document which API(s) will be called with the forwarded `**kwargs`.
 
 Example:
 ```python
 def request(method, url, headers, **kwargs): ...
 
 def get(*args, **kwargs):
-    "Calls `request` with the method "GET" and forwards all other arguments."
+    """Calls `request` with the method "GET" and forwards all other arguments.
+
+    Keyword arguments are passed to :func:`request`.
+    """
     return request("GET", *args, **kwargs)
 ```
 
@@ -832,7 +831,7 @@ Code samples are small applications that demonstrate a certain feature that is r
 
 {% include requirement/MUST id="python-samples-runnable" %} ensure that each sample file is runnable.
 
-{% include requirement/MUST id="python-samples-coding-style" %} avoid using features newer than the Python 3 baseline support. The current supported Python version is 3.8.
+{% include requirement/MUST id="python-samples-coding-style" %} avoid using features newer than the Python 3 baseline support. The current minimum supported Python version is 3.9.
 
 {% include requirement/MUST id="python-samples-grafting" %} ensure that code samples can be easily grafted from the documentation into a users own application. For example, don't rely on variable declarations in other samples.
 

docs/python/documentation.md

@@ -33,7 +33,7 @@ As you write your code, *doc it so you never hear about it again.* The less ques
 
 {% include requirement/MUST id="python-docstrings-all" %} provide docstrings for all public modules, types, and methods.
 
-{% include requirement/MUST id="python-docstrings-kwargs" %} document any `**kwargs` directly consumed by a method and add a ref link to [core options](https://aka.ms/azsdk/python/options) to provide introduction for shared options. You may refer to the signature of a called method if the `**kwargs` are passed through.
+{% include requirement/MUST id="python-docstrings-kwargs" %} document any `**kwargs` directly consumed by a method and add a ref link to [core options](https://aka.ms/azsdk/python/options) to provide introduction for shared options. If `**kwargs` are passed through to another API, you **must** document which API(s) will be called with the forwarded `**kwargs`.
 
 Example:
 ```python
@@ -45,6 +45,7 @@ def get(*args, **kwargs):
     :param str method-param: The method-param parameter
     :keyword int method-kwarg: The optional method-kwarg parameter
 
+    Keyword arguments are passed to :func:`request`.
     For additional request configuration options, please see https://aka.ms/azsdk/python/options.
     """
     return request("GET", *args, **kwargs)

docs/python/implementation.md

@@ -42,7 +42,7 @@ from azure.core.pipeline.policies import (
     UserAgentPolicy,
 )
 
-class ExampleClient(object):
+class ExampleClient:
 
     ...
 
@@ -375,15 +375,15 @@ def DoSomething():
 
 ```python
 # Yes:
-class ThisIsCorrect(object):
+class ThisIsCorrect:
     pass
 
 # No:
-class this_is_not_correct(object):
+class this_is_not_correct:
     pass
 
 # No:
-class camelCasedTypeName(object):
+class camelCasedTypeName:
     pass
 ```
 
@@ -412,26 +412,24 @@ Static methods are rare and usually forced by other libraries.
 
 ```python
 # Yes
-class GoodThing(object):
+class GoodThing:
 
     @property
     def something(self):
         """ Example of a good read-only property."""
         return self._something
 
 # No
-class BadThing(object):
+class BadThing:
 
     def get_something(self):
         """ Example of a bad 'getter' style method."""
         return self._something
 ```
 
-{% include requirement/SHOULDNOT id="python-codestyle-long-args" %} have methods that require more than five positional parameters. Optional/flag parameters can be accepted using keyword-only arguments, or `**kwargs`.
+{% include requirement/SHOULDNOT id="python-codestyle-long-args" %} have methods that require more than five positional parameters. Optional/flag parameters can be accepted using keyword-only arguments.
 
-See TODO: insert link for general guidance on positional vs. optional parameters here.
-
-{% include requirement/MUST id="python-codestyle-optional-args" %} use keyword-only arguments for optional or less-often-used arguments for modules that only need to support Python 3.
+{% include requirement/MUST id="python-codestyle-optional-args" %} use keyword-only arguments for optional or less-often-used arguments.
 
 ```python
 # Yes
@@ -451,6 +449,38 @@ def copy(source, dest, *, recurse=False, overwrite=False) ...
 def copy(source, dest, recurse=False, overwrite=False) ...
 ```
 
+{% include requirement/MUSTNOT id="python-codestyle-no-kwargs-for-arguments" %} use `**kwargs` to accept arguments that are consumed directly by the method. Use keyword-only arguments instead.
+
+```python
+# Yes - keyword-only arguments are explicit and discoverable:
+def create_thing(name: str, *, size: int = 0, color: str = "blue") -> None: ...
+
+# No - using **kwargs to accept arguments consumed by the method:
+def create_thing(name: str, **kwargs) -> None:
+    size = kwargs.pop("size", 0)
+    color = kwargs.pop("color", "blue")
+    ...
+```
+
+{% include requirement/MAY id="python-codestyle-kwargs-passthrough" %} use `**kwargs` when the method needs to pass parameters through to other methods (e.g. pipeline policies or an underlying API). When doing so, you **must** document which API(s) will be called with the forwarded `**kwargs`.
+
+```python
+# Yes - kwargs are passed through to the pipeline, and this is documented:
+def get_thing(self, name: str, **kwargs) -> "Thing":
+    """Get the thing with the given name.
+
+    :param name: The name of the thing.
+    :type name: str
+    :return: The thing.
+    :rtype: ~Thing
+
+    For additional request configuration keyword arguments, please see
+    https://aka.ms/azsdk/python/options.
+    """
+    request = self._build_get_thing_request(name)
+    return self._pipeline.send(request, **kwargs)
+```
+
 {% include requirement/MUST id="python-codestyle-positional-params" %} specify the parameter name when calling methods with more than two required positional parameters.
 
 ```python
@@ -510,11 +540,37 @@ azure.exampleservice.some_internal_module
 
 {% include requirement/MUST id="python-codestyle-structural-subtyping" %} prefer structural subtyping and protocols over explicit type checks.
 
-{% include requirement/MUST id="python-codestyle-abstract-collections" %} derive from the abstract collections base classes `collections.abc` (or `collections` for Python 2.7) to provide custom mapping types.
+{% include requirement/MUST id="python-codestyle-abstract-collections" %} derive from the abstract collections base classes `collections.abc` to provide custom mapping types.
 
 {% include requirement/MUST id="python-codestyle-pep484" %} provide type hints [PEP484](https://www.python.org/dev/peps/pep-0484/) for publicly documented classes and functions.
 
-- See the [suggested syntax for Python 2.7 and 2.7-3.x straddling code](https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code) for guidance for Python 2.7 compatible code. Do not do this for code that is Python 3 specific (e.g. `async` clients.)
+{% include requirement/MUST id="python-codestyle-use-builtin-generics" %} use built-in generic types (`list`, `dict`, `tuple`, `set`) in type annotations instead of their `typing` module counterparts (`typing.List`, `typing.Dict`, `typing.Tuple`, `typing.Set`). This is supported as of Python 3.9 ([PEP 585](https://www.python.org/dev/peps/pep-0585/)).
+
+```python
+# Yes (Python 3.9+):
+def get_things() -> list[str]: ...
+def get_mapping() -> dict[str, int]: ...
+
+# No:
+from typing import List, Dict
+def get_things() -> List[str]: ...
+def get_mapping() -> Dict[str, int]: ...
+```
+
+{% include requirement/MUST id="python-codestyle-union-optional" %} use `typing.Union` and `typing.Optional` for union types.
+
+<!-- NOTE: If the minimum supported Python version is raised to 3.10+, use the `X | Y` union syntax
+(PEP 604) instead of `typing.Union[X, Y]` and `X | None` instead of `typing.Optional[X]`.
+For example: `def foo(x: int | str) -> str | None: ...` -->
+
+```python
+# Yes (Python 3.9):
+from typing import Optional, Union
+def foo(x: Union[int, str]) -> Optional[str]: ...
+
+# No (requires Python 3.10+):
+def foo(x: int | str) -> str | None: ...
+```
 
 ### Threading