diff --git a/doc/admin-guide/files/index.en.rst b/doc/admin-guide/files/index.en.rst index 1ce15d62421..867d945f6b3 100644 --- a/doc/admin-guide/files/index.en.rst +++ b/doc/admin-guide/files/index.en.rst @@ -33,6 +33,7 @@ Configuration Files plugin.config.en records.yaml.en remap.config.en + remap.yaml.en splitdns.config.en ssl_multicert.config.en sni.yaml.en @@ -69,6 +70,9 @@ Configuration Files :doc:`remap.config.en` Defines mapping rules used by |TS| to properly route all incoming requests. +:doc:`remap.yaml.en` + Defines mapping rules used by |TS| to properly route all incoming requests in YAML format + :doc:`splitdns.config.en` Configures DNS servers to use under specific conditions. diff --git a/doc/admin-guide/files/remap.yaml.en.rst b/doc/admin-guide/files/remap.yaml.en.rst new file mode 100644 index 00000000000..398837aa1c1 --- /dev/null +++ b/doc/admin-guide/files/remap.yaml.en.rst @@ -0,0 +1,1212 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +=========== +remap.yaml +=========== + +.. configfile:: remap.yaml + +.. include:: ../../common.defs + +.. toctree:: + :maxdepth: 2 + + +The :file:`remap.yaml` file (by default, located in +``/usr/local/etc/trafficserver/``) provides a YAML-based alternative to +:file:`remap.config` for configuring URL remapping rules. Traffic Server +uses these mapping rules to perform the following actions: + +- Map URL requests for a specific origin server to the appropriate + location on Traffic Server when Traffic Server acts as a reverse + proxy for that particular origin server +- Reverse-map server location headers so that when origin servers + respond to a request with a location header that redirects the client + to another location, the clients do not bypass Traffic Server +- Redirect HTTP requests permanently or temporarily without Traffic + Server having to contact any origin servers + +Refer to :ref:`reverse-proxy-and-http-redirects`, for information about +redirecting HTTP requests and using reverse proxy. + +After you modify the :file:`remap.yaml` run the +:option:`traffic_ctl config reload` to apply the changes. The current configuration is replaced +with the new configuration only if there are no errors in the file. Any syntax error will prevent +an update. Even if syntactically correct the file is considered valid only if it has at least :ts:cv:`proxy.config.url_remap.min_rules_required` +rules in it. This defaults to 0, but can be set higher if it is desirable to prevent loading an +empty or missing file. + +Configuration File Fallback +============================ + +Traffic Server will attempt to load :file:`remap.yaml` first. If this file is not found, +it will fall back to loading :file:`remap.config`. If both files exist, only +:file:`remap.yaml` will be used. This allows for a gradual migration from the legacy +configuration format to YAML. + +Format +====== + +The :file:`remap.yaml` file uses YAML syntax with two main sections: + +- ``acl_filters`` (optional): Global named filter definitions +- ``remap`` (required): Sequence of remapping rules + +Basic Structure +--------------- + +.. code-block:: yaml + + # Optional: Global named filter definitions + acl_filters: + filter_name: + # filter definition + + # Required: Remapping rules + remap: + - type: map + from: + url: http://example.com/foo + to: + url: http://backend.com/bar + - type: redirect + from: + url: old.example.com + to: + url: new.example.com + +Comments and Empty Lines +------------------------- + +Lines starting with ``#`` are comments and are ignored. Empty lines are also ignored. +Unlike :file:`remap.config`, YAML does not support line continuation with ``\``. + +Rule Structure +============== + +Each remap rule is a YAML mapping with the following fields: + +``type`` +-------- + +Specifies the type of remapping rule. Required field. One of: + +- ``map`` -- translates an incoming request URL to the appropriate + origin server URL. + +- ``map_with_recv_port`` -- exactly like 'map' except that it uses the port at + which the request was received to perform the mapping instead of the port present + in the request. The ``regex_`` prefix can also be used for this type. When present, + 'map_with_recv_port' mappings are checked first. + +- ``map_with_referer`` -- extended version of 'map', which can be used to activate + "deep linking protection", where target URLs are only accessible when the Referer + header is set to a URL that is allowed to link to the target. + +- ``reverse_map`` -- translates the URL in origin server redirect + responses to point to the Traffic Server. + +- ``redirect`` -- redirects HTTP requests permanently without having + to contact the origin server. Permanent redirects notify the + browser of the URL change (by returning an HTTP status code 301) + so that the browser can update bookmarks. + +- ``redirect_temporary`` -- redirects HTTP requests temporarily + without having to contact the origin server. Temporary redirects + notify the browser of the URL change for the current request only + (by returning an HTTP status code 307). + +.. note:: use the ``regex_`` prefix for the type to indicate that the rule uses regular expressions. + +``from`` +-------- + +Specifies the request ("from") URL as a YAML mapping. URL can be defined as a single URL or in components. +Single URL takes precedence over components: + +- ``url``: scheme://host:port/path + +URL components: + +- ``scheme`` (optional): ``http``, ``https``, ``ws``, ``wss``, or ``tunnel``. +- ``host`` (optional for forward maps with path, required otherwise): Hostname or IP address +- ``port`` (optional): Port number +- ``path`` (optional): Path prefix + +Example: + +.. code-block:: yaml + + from: + url: http://example.com:80/foo + from: + scheme: http + host: example.com + port: 80 + path: /foo + +.. note:: A remap rule for requests that upgrade from HTTP to WebSocket still require a remap rule with the ``ws`` or ``wss`` scheme. + +``to`` +------ + +Specifies the origin ("to") URL as a YAML mapping. Required field. + +URL components are the same as ``from``. + +Example: + +.. code-block:: yaml + + to: + url: http://backend.example.com:8080/bar + to: + scheme: http + host: backend.example.com + port: 8080 + path: /bar + +Optional Fields +--------------- + +``acl_filter`` +~~~~~~~~~~~~~~ + +Inline ACL filter definition for a single remap rule. See `ACL Filters`_ for details. + +.. code-block:: yaml + + acl_filter: + src_ip: + - 10.0.0.0/8 + method: + - GET + - POST + action: allow + +``plugins`` +~~~~~~~~~~~ + +Sequence of plugin configurations. Each plugin has: + +- ``name`` (required): Plugin filename +- ``params`` (optional): List of plugin parameters + +.. code-block:: yaml + + plugins: + - name: header_rewrite.so + params: + - config.txt + - param2 + - name: another_plugin.so + params: + - param1 + +``strategy`` +~~~~~~~~~~~~ + +NextHop selection strategy name. See :doc:`../configuration/hierarchical-caching.en` and :doc:`strategies.yaml.en`. + +.. code-block:: yaml + + strategy: my_strategy + +``redirect`` +~~~~~~~~~~~~ + +Used with ``map_with_referer`` type. Specifies redirect URL and allowed referer patterns. + +.. code-block:: yaml + + redirect: + url: http://example.com/denied # or "default" for default redirect URL + regex: + - "~https://trusted\\.com/.*" # regex pattern + - "~" # allow any referer (makes referer optional) + +Precedence +========== + +Remap rules in :file:`remap.yaml` follow the same precedence order as :file:`remap.config`: + +1. ``map_with_recv_port`` and ``regex_map_with_recv_port`` +#. ``map`` and ``regex_map`` and ``reverse_map`` +#. ``redirect`` and ``redirect_temporary`` +#. ``regex_redirect`` and ``regex_redirect_temporary`` + +For each precedence group the rules are checked in two phases. If the first phase fails to find a +match then the second phase is performed against the same group of rules. In the first phase the +rules are checked using the host name of the request. Only rules that specify a host name can match. +If there is no match in that phase, then the rules are checked again with no host name and only +rules without a host will match. The result is that rules with an explicit host take precedence over +rules without. + +Match-All +========= + +A map rule with only a path of ``/`` acts as a wildcard, it will match any +request. This should be use with care, and certainly only once at the +end of the remap section. E.g. + +.. code-block:: yaml + + remap: + - type: map + from: + url: / + to: + url: http://all.example.com + +:file:`remap.config` equivalent :: + + map / http://all.example.com + +Examples +======== + +The following sections show example mapping rules in the :file:`remap.yaml` file. + +Reverse Proxy Mapping Rules +---------------------------- + +The following example shows a map rule that does not specify a path +prefix in the target or replacement: + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://www.x.com/ + to: + url: http://server.hoster.com/ + - type: reverse_map + from: + url: http://server.hoster.com/ + to: + url: http://www.x.com/ + +:file:`remap.config` equivalent :: + + map http://www.x.com/ http://server.hoster.com/ + reverse_map http://server.hoster.com/ http://www.x.com/ + +This rule results in the following translations: + +================================================ ======================================================== +Client Request Translated Request +================================================ ======================================================== +``http://www.x.com/Widgets/index.html`` ``http://server.hoster.com/Widgets/index.html`` +``http://www.x.com/cgi/form/submit.sh?arg=true`` ``http://server.hoster.com/cgi/form/submit.sh?arg=true`` +================================================ ======================================================== + +The following example shows a map rule with path prefixes specified in +the target: + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://www.y.com/marketing/ + to: + url: http://marketing.y.com/ + - type: reverse_map + from: + url: http://marketing.y.com/ + to: + url: http://www.y.com/marketing/ + - type: map + from: + url: http://www.y.com/sales/ + to: + url: http://sales.y.com/ + - type: reverse_map + from: + url: http://sales.y.com/ + to: + url: http://www.y.com/sales/ + - type: map + from: + url: http://www.y.com/engineering/ + to: + url: http://engineering.y.com/ + - type: reverse_map + from: + url: http://engineering.y.com/ + to: + url: http://www.y.com/engineering/ + - type: map + from: + url: http://www.y.com/stuff/ + to: + url: http://info.y.com/ + - type: reverse_map + from: + url: http://info.y.com/ + to: + url: http://www.y.com/stuff/ + +:file:`remap.config` equivalent :: + + map http://www.y.com/marketing/ http://marketing.y.com/ + reverse_map http://marketing.y.com/ http://www.y.com/marketing/ + map http://www.y.com/sales/ http://sales.y.com/ + reverse_map http://sales.y.com/ http://www.y.com/sales/ + map http://www.y.com/engineering/ http://engineering.y.com/ + reverse_map http://engineering.y.com/ http://www.y.com/engineering/ + map http://www.y.com/stuff/ http://info.y.com/ + reverse_map http://info.y.com/ http://www.y.com/stuff/ + +These rules result in the following translations: + +=============================================================== ========================================================== +Client Request Translated Request +=============================================================== ========================================================== +``http://www.y.com/marketing/projects/manhattan/specs.html`` ``http://marketing.y.com/projects/manhattan/specs.html`` +``http://www.y.com/stuff/marketing/projects/boston/specs.html`` ``http://info.y.com/marketing/projects/boston/specs.html`` +=============================================================== ========================================================== + +The following example shows that the order of the rules matters: + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://www.g.com/ + to: + url: http://external.g.com/ + - type: reverse_map + from: + url: http://external.g.com/ + to: + url: http://www.g.com/ + - type: map + from: + url: http://www.g.com/stuff/ + to: + url: http://stuff.g.com/ + - type: reverse_map + from: + url: http://stuff.g.com/ + to: + url: http://www.g.com/stuff/ + +:file:`remap.config` equivalent :: + + map http://www.g.com/ http://external.g.com/ + reverse_map http://external.g.com/ http://www.g.com/ + map http://www.g.com/stuff/ http://stuff.g.com/ + reverse_map http://stuff.g.com/ http://www.g.com/stuff/ + +These rules result in the following translation. + +================================ ===================================== +Client Request Translated Request +================================ ===================================== +``http://www.g.com/stuff/a.gif`` ``http://external.g.com/stuff/a.gif`` +================================ ===================================== + +In the above examples, the second rule is never applied because all URLs +that match the second rule also match the first rule. The first rule +takes precedence because it appears earlier in the :file:`remap.config` +file. + +This is different if one rule does not have a host. For example consider these rules using the `Match-All`_ rule + +.. code-block:: yaml + + remap: + - type: map + from: + url: / + to: + url: http://127.0.0.1:8001/ + - type: map + from: + url: http://example.com/dist_get_user + to: + url: http://127.0.0.1:8001/denied.html + +These rules are set up to redirect requests to another local process. Using them will result in + +==================================== ===================================== +Client Request Translated Request +==================================== ===================================== +``http://example.com/a.gif`` ``http://127.0.0.1:8001/a.gif`` +``http://example.com/dist_get_user`` ``http://127.0.0.1:8001/denied.html`` +==================================== ===================================== + +For the first request the second rule host matches but the path does not and so the second rule is +not selected. The first rule is then matched in the second phase when the rules are checked without +a host value. + +The second request is matched by the second rule even though the rules have the same base +precedence. Because the first rule does not have a host it will not match in the first phase. The +second rule does have a host that matches the host in the second request along with the other parts +of the URL and is therefore selected in the first phase. + +This will yield the same results if the rules are reversed because the rule selection happens in +different phases making the order irrelevant. + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://example.com/dist_get_user + to: + url: http://127.0.0.1:8001/denied.html + - type: map + from: + url: / + to: + url: http://127.0.0.1:8001/ + +The following example shows a mapping with a path prefix specified in +the target and replacement + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://www.h.com/a/b/ + to: + url: http://server.h.com/customers/x/y + - type: reverse_map + from: + url: http://server.h.com/customers/x/y/ + to: + url: http://www.h.com/a/b/ + +This rule results in the following translation. + +===================================== ================================================== +Client Request Translated Request +===================================== ================================================== +``http://www.h.com/a/b/c/d/doc.html`` ``http://server.h.com/customers/x/y/c/d/doc.html`` +``http://www.h.com/a/index.html`` ``Translation fails`` +===================================== ================================================== + +The following example shows reverse-map rules + +.. code-block:: yaml + + remap: + - type: map + from: + url: www.x.com + to: + url: http://server.hoster.com/x/ + - type: reverse_map + from: + url: http://server.hoster.com/x/ + to: + url: http://www.x.com/ + +These rules result in the following translations. + +================================ ===================================== +Client Request Translated Request +================================ ===================================== +``http://www.x.com/Widgets`` ``http://server.hoster.com/x/Widgets`` +================================ ===================================== + + + +================================ ======================================= ============================= +Client Request Origin Server Header Translated Request +================================ ======================================= ============================= +``http://www.x.com/Widgets`` ``http://server.hoster.com/x/Widgets/`` ``http://www.x.com/Widgets/`` +================================ ======================================= ============================= + +When acting as a reverse proxy for multiple servers, Traffic Server is +unable to route to URLs from older browsers that do not send the +``Host:`` header. As a solution, set the variable :ts:cv:`proxy.config.header.parse.no_host_url_redirect` +in the :file:`records.yaml` file to the URL to which Traffic Server will redirect +requests without host headers. + + +Redirect Mapping Rules +----------------------- + +The following rule permanently redirects all HTTP requests for +``www.company.com`` to ``www.company2.com``: + +.. code-block:: yaml + + remap: + - type: redirect + from: + url: http://www.company.com/ + to: + url: http://www.company2.com/ + +The following rule *temporarily* redirects all HTTP requests for +``www.company1.com`` to ``www.company2.com``: + +.. code-block:: yaml + + remap: + - type: redirect_temporary + from: + url: http://www.company1.com/ + to: + url: http://www.company2.com/ + +Regular Expression (regex) Remap Support +========================================= + +Regular expressions can be specified in remapping rules by using the ``regex_`` prefix +for the rule type, with the same limitations as :file:`remap.config`: + +- Only the ``host`` field can contain a regex; the ``scheme``, + ``port``, and other fields cannot. For path manipulation via regexes, + use the :ref:`admin-plugins-regex-remap`. +- The number of capturing subpatterns is limited to 9. This means that + ``$0`` through ``$9`` can be used as substitution placeholders (``$0`` + will be the entire input string). +- The number of substitutions in the expansion string is limited to 10. +- There is no ``regex_`` equivalent to ``reverse_remap``, so when using + ``regex_map`` you should make sure the reverse path is clear by + setting (:ts:cv:`proxy.config.url_remap.pristine_host_hdr`) + +Examples +-------- + +.. code-block:: yaml + + remap: + - type: regex_map + from: + url: http://x([0-9]+).z.com/ + to: + url: http://real-x$1.z.com/ + - type: regex_redirect + from: + url: http://old.(.*).z.com + to: + url: http://new.$1.z.com + +map_with_recv_port +================== + +The ``map_with_recv_port`` type supports two special URL schemes, ``http+unix`` and ``https+unix``. +These are useful if you want to have different mapping rules or different plugin configuration for requests received via Unix Domain Socket. + +Examples +-------- + +.. code-block:: yaml + + remap: + - type: map_with_recv_port + from: + url: http://foo.example.com:8000/ + to: + url: http://x.example.com/ + - type: map_with_recv_port + from: + url: http://foo.example.com:8888/ + to: + url: http://y.example.com/ + +Explanation: Requests received on port 8000 and 8888 are forwarded to different servers. + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://foo.example.com/ + to: + url: http://x.example.com/ + plugins: + - name: plugin1.so + - type: map_with_recv_port + from: + url: http+unix://foo.example.com/ + to: + url: http://x.example.com/ + +Explanation: All requests are forwarded to the same server, but plugin1 does not run for requests received via Unix Domain Socket. + +map_with_referer +================ + +'redirect-URL' is a redirection URL specified according to RFC 2616 and can +contain special formatting instructions for run-time modifications of the +resulting redirection URL. All regexes Perl compatible regular expressions, +which describes the content of the "Referer" header which must be +verified. In case an actual request does not have "Referer" header or it +does not match with referer regular expression, the HTTP request will be +redirected to 'redirect-URL'. + +The ``map_with_referer`` type enables "deep linking protection" by validating +the Referer header against regular expressions. + +.. code-block:: yaml + + remap: + - type: map_with_referer + from: + url: client-URL + to: + url: origin-server-URL + redirect: + url: redirect-URL + regex: + - regex1 + - regex2 + +At least one regular expression must be specified. In order to enable the 'deep linking +protection' feature in Traffic Server, configure records.yaml with: + +.. code-block:: yaml + :linenos: + :emphasize-lines: 3 + + records: + http: + referer_filter: 1 + +In order to enable run-time formatting for redirect URL, configure: + +.. code-block:: yaml + :linenos: + :emphasize-lines: 3 + + records: + http: + referer_format_redirect: 1 + +When run-time formatting for redirect-URL is enabled the following format +symbols can be used:: + + %r - to substitute original "Referer" header string + %f - to substitute client-URL from 'map_with_referer' record + %t - to substitute origin-server-URL from 'map_with_referer' record + %o - to substitute request URL to origin server, which was created as + the result of a mapping operation + +Note: There is a special referer type "~*" that can be used to specify that the Referer header is optional in the request. If "~*" referer +was used in map_with_referer mapping, only requests with Referer header will +be verified for validity. If the "~" symbol was specified before a referer +regular expression, it means that the request with a matching referer header +will be redirected to redirectURL. It can be used to create a so-called +negative referer list. If "*" was used as a referer regular expression - +all referrers are allowed. + +Examples +-------- + +.. code-block:: yaml + + remap: + - type: map_with_referer + from: + url: http://y.foo.bar.com/x/yy/ + to: + url: http://foo.bar.com/x/yy/ + redirect: + url: http://games.bar.com/new_games + regex: + - ".*\\.bar\\.com" + - "www.bar-friends.com" + +Explanation: Referer header must be in the request, only ".*\\.bar\\.com" and "www.bar-friends.com" are allowed. + +.. code-block:: yaml + + remap: + - type: map_with_referer + from: + url: http://y.foo.bar.com/x/yy/ + to: + url: http://foo.bar.com/x/yy/ + redirect: + url: http://games.bar.com/new_games + regex: + - "*" + - "~.*\\.evil\\.com" + +Explanation: Referer header must be in the request but all referrers are allowed except ".*\\.evil\\.com". + +.. code-block:: yaml + + remap: + - type: map_with_referer + from: + url: http://y.foo.bar.com/x/yy/ + to: + url: http://foo.bar.com/x/yy/ + redirect: + url: http://games.bar.com/error + regex: + - "~*" + - "*" + - "~.*\\.evil\\.com" + +Explanation: Referer header is optional. However, if Referer header exists, only request from ".*\\.evil\\.com" will be redirected to redirect-URL. + +Plugin Chaining +=============== + +Plugins can be configured to be evaluated in a specific order, passing +the results from one to the next (unless a plugin returns 0, then the +"chain" is broken). + +Examples +-------- + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://url/path + to: + url: http://url/path + plugins: + - name: /etc/traffic_server/config/plugins/plugin1.so + params: + - "1" + - "2" + - name: /etc/traffic_server/config/plugins/plugin2.so + params: + - "3" + +This will pass "1" and "2" to plugin1.so and "3" to plugin2.so. + +NextHop Selection Strategies +============================= + +You may configure Nexthop or Parent hierarchical caching rules by remap using the +``strategy`` field. See :doc:`../configuration/hierarchical-caching.en` and :doc:`strategies.yaml.en` +for configuration details and examples. + +.. code-block:: yaml + + remap: + - type: map + from: + url: htpp://example.com + to: + url: http://backend.com + strategy: my_strategy + +ACL Filters +=========== + +In-line Filter +-------------- + +In-line filters can be created to control access of specific remap rules. The markup +is very similar to that of :file:`ip_allow.yaml`, structured as YAML mappings instead +of directive-based syntax. + +Actions +~~~~~~~ + +Each ACL filter takes one of a number of actions specified by the ``action`` field: + +- ``allow``: This behaves like the ``allow`` action in :file:`ip_allow.yaml` in which a list of allowed methods are + provided. Any request with a method in the list is allowed, while any request with a method not in the list is denied. + The exception to this is if :ts:cv:`proxy.config.url_remap.acl_behavior_policy` is set to ``0``. In this case, the + ``allow`` action is a synonym for ``add_allow``, described below. +- ``add_allow``: This action adds a list of allowed methods to whatever other methods are allowed in a subsequently + matched ACL filter or :file:`ip_allow.yaml` rule. +- ``deny``: This behaves like the ``deny`` action in :file:`ip_allow.yaml` in which a list of denied methods are + provided. Any request with a method in the list is denied, while any request with a method not in the list is allowed. + The exception to this is if :ts:cv:`proxy.config.url_remap.acl_behavior_policy` is set to ``0``. In this case, the + ``deny`` action is a synonym for ``add_deny``, described below. +- ``add_deny``: This action adds a list of denied methods to whatever other methods are denied in a subsequently matched + ACL filter or :file:`ip_allow.yaml` rule. + +Filter Fields +~~~~~~~~~~~~~ + +- ``src_ip`` -- source IP address or CIDR range (can be a list) +- ``src_ip_invert`` -- inverted source IP address or CIDR range (can be a list) +- ``src_ip_category`` -- source IP category name (string) +- ``src_ip_category_invert`` -- inverted source IP category name (string) +- ``in_ip`` -- incoming IP address or CIDR range (can be a list) +- ``in_ip_invert`` -- inverted incoming IP address or CIDR range (can be a list) +- ``method`` -- HTTP method or list of methods +- ``action`` -- action to take (allow, deny, add_allow, add_deny) +- ``internal`` -- boolean, matches internal requests only + +Examples +~~~~~~~~ + +.. code-block:: yaml + + remap: + - type: map + from: + url: http://foo.example.com/neverpost + to: + url: http://foo.example.com/neverpost + acl_filter: + action: deny + method: post + + - type: map + from: + url: http://foo.example.com/onlypost + to: + url: http://foo.example.com/onlypost + acl_filter: + action: allow + method: post + + - type: map + from: + url: http://foo.example.com/ + to: + url: http://foo.example.com/ + acl_filter: + action: deny + src_ip: 1.2.3.4 + + - type: map + from: + url: http://foo.example.com/ + to: + url: http://foo.example.com/ + acl_filter: + action: allow + src_ip: + - 10.5.2.1 + in_ip: + - 72.209.23.4 + + - type: map + from: + url: http://foo.example.com/ + to: + url: http://foo.example.com/ + acl_filter: + action: allow + src_ip: 127.0.0.1 + method: + - post + - get + - head + + - type: map + from: + url: http://foo.example.com/ + to: + url: http://foo.example.com/ + acl_filter: + action: allow + src_ip_category: ACME_INTERNAL + method: + - post + - get + - head + +Note that these ACL filters will return a 403 response if the resource is restricted. + +The difference between ``src_ip`` and ``in_ip`` is that ``src_ip`` is the client +IP and ``in_ip`` is the IP address the client is connecting to (the incoming address). +``src_ip_category`` functions like ``ip_category`` described in :file:`ip_allow.yaml`. +If no IP address is specified for ``src_ip``, ``src_ip_category``, or +``in_ip``, the filter will implicitly apply to all incoming IP addresses. This +can be explicitly stated with ``src_ip: all``. + +Named Filters +------------- + +Named filters can be defined globally in the ``acl_filters`` section and then activated +or deactivated for blocks of mappings using filter directives. + +Filter Directives +~~~~~~~~~~~~~~~~~ + +Filter directives are special entries in the ``remap`` sequence: + +- ``activate_filter: `` -- activates a named filter for subsequent rules +- ``deactivate_filter: `` -- deactivates a named filter +- ``delete_filter: `` -- removes a filter definition +- ``define_filter`` -- defines a new named filter inline + +The ``internal`` operator can be used to filter on whether a request +is generated by |TS| itself, usually by a plugin. This operator +is helpful for remapping internal requests without allowing access +to external users. By default both internal and external requests +are allowed. + +Examples +~~~~~~~~ + +.. code-block:: yaml + + acl_filters: + disable_delete_purge: + action: deny + method: + - delete + - purge + local_only: + action: allow + src_ip: + - 192.168.0.1-192.168.0.254 + - 10.0.0.1-10.0.0.254 + + remap: + - activate_filter: disable_delete_purge + + - type: map + from: + url: http://foo.example.com/ + to: + url: http://bar.example.com/ + + - activate_filter: local_only + + - type: map + from: + url: http://www.example.com/admin + to: + url: http://internal.example.com/admin + + - deactivate_filter: local_only + + - type: map + from: + url: http://www.example.com/ + to: + url: http://internal.example.com/ + + - type: map + from: + url: http://auth.example.com/ + to: + url: http://auth.internal.example.com/ + acl_filter: + action: allow + internal: true + +The filter ``disable_delete_purge`` will be applied to all of the +mapping rules after it is activated. (It is activated before any mappings +and is never deactivated.) The filter ``local_only`` will only be applied to +the ``www.example.com/admin`` mapping. + +Special Filter and ip_allow Named Filter +---------------------------------------- + +If :file:`ip_allow.yaml` has a "deny all" filter, it is treated as a special filter that is applied before remapping for +optimization. To control this for specific remap rules, a named filter called ``ip_allow`` is pre-defined. This named filter is +activated implicitly by default. To stop applying the special rule, disable the ``ip_allow`` filter as shown below. + +.. code-block:: yaml + + # ip_allow.yaml + ip_allow: + - apply: in + ip_addrs: 198.51.100.0/24 + action: deny + method: ALL + +.. code-block:: yaml + + # remap.yaml + remap: + - deactivate_filter: ip_allow + - type: map ... + - type: map ... + - activate_filter: ip_allow + +Note this entirely disables :file:`ip_allow.yaml` checks for those remap rules. + +Evaluation Order and Matching Policy +------------------------------------ + +|TS| evaluates multiple ACL filters in the following order: + +1. Special "deny all" filter in :file:`ip_allow.yaml` +2. In-line Filter in :file:`remap.yaml` +3. Named Filters in :file:`remap.yaml` +4. Filters in :file:`ip_allow.yaml` + +When a matching ACL filter is found, |TS| stops processing subsequent ACL filters. + +Note that step 1 happens at the start of the connection before any transactions are processed, unlike the other rules +here. This is an optimization: if literally all requests are denied for a source IP address via an +:file:`ip_allow.yaml` rule, then there is no need to process any content from that IP for the connection at all, so the +connection is simply denied at the start. + +.. note:: + + The ACL filter behavior in :file:`remap.yaml` is identical to that in :file:`remap.config`. + See :file:`remap.config` for details on ACL Action Behavior Changes for 10.x, Legacy Policy, + Modern Policy, and examples of ACL filter combinations. + +Including Additional Remap Files +================================= + +The ``include`` directive allows mapping rules to be spread across +multiple files. The argument to the ``include`` directive is a file path +or directory path. Unless the path is absolute, it is resolved relative to the +Traffic Server configuration directory. + +The effect of the ``include`` directive is as if the contents of +the included file(s) is included in the parent and parsing restarted +at the point of inclusion. This means that any filters defined in the +included files are global in scope, and that additional ``include`` +directives are allowed. + +.. note:: + + Included remap files are currently tracked by the configuration + subsystem. Changes to included remap files will be noticed + by online configuration changes applied by :option:`traffic_ctl config reload`. + +Examples +-------- + +In a top-level :file:`remap.yaml` file: + +.. code-block:: yaml + + remap: + - include: filters.yaml + - include: one.example.com.yaml + - include: two.example.com.yaml + - include: /path/to/remap_fragments/ # directory + +The file ``filters.yaml`` contains: + +.. code-block:: yaml + + acl_filters: + deny_purge: + action: deny + method: purge + allow_purge: + action: allow + method: purge + +The file ``one.example.com.yaml`` contains: + +.. code-block:: yaml + + remap: + - activate_filter: deny_purge + - type: map + from: + url: http://one.example.com + to: + url: http://origin-one.example.com + - deactivate_filter: deny_purge + +The file ``two.example.com.yaml`` contains: + +.. code-block:: yaml + + remap: + - activate_filter: allow_purge + - type: map + from: + url: http://two.example.com + to: + url: http://origin-two.example.com + - deactivate_filter: allow_purge + +Migration from remap.config +============================ + +The :file:`remap.yaml` format provides equivalent functionality to :file:`remap.config` +with YAML structure. Here are some common patterns: + +remap.config to remap.yaml +--------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 50 50 + + * - remap.config + - remap.yaml + * - :: + + map http://www.x.com/ http://server.com/ + - .. code-block:: yaml + + remap: + - type: map + from: + url: http://www.x.com/ + to: + url: http://server.com/ + + * - :: + + redirect http://old.com/ http://new.com/ + - .. code-block:: yaml + + remap: + - type: redirect + from: + url: http://old.com/ + to: + url: http://new.com/ + + * - :: + + map http://example.com/ http://backend.com/ \ + @plugin=plugin.so @pparam=arg1 + + - .. code-block:: yaml + + remap: + - type: map + from: + url: http://example.com/ + to: + url: http://backend.com/ + plugins: + - name: plugin.so + params: + - arg1 + + * - :: + + .definefilter my_filter @action=allow @src_ip=10.0.0.0/8 + .activatefilter my_filter + map http://example.com/ http://backend.com/ + + - .. code-block:: yaml + + acl_filters: + my_filter: + action: allow + src_ip: 10.0.0.0/8 + + remap: + - activate_filter: my_filter + - type: map + from: + url: http://example.com/ + to: + url: http://backend.com/ diff --git a/include/proxy/http/remap/AclFiltering.h b/include/proxy/http/remap/AclFiltering.h index 32fe3c0c7a5..f830a16665a 100644 --- a/include/proxy/http/remap/AclFiltering.h +++ b/include/proxy/http/remap/AclFiltering.h @@ -31,6 +31,7 @@ #include #include #include +#include // =============================================================================== // ACL like filtering defs (per one remap rule) @@ -108,8 +109,9 @@ class acl_filter_rule internal : 1; // filter internal HTTP requests // we need arguments as string array for directive processing - int argc = 0; // argument counter (only for filter defs) - char *argv[ACL_FILTER_MAX_ARGV]; // argument strings (only for filter defs) + int argc = 0; // argument counter (only for filter defs) + char *argv[ACL_FILTER_MAX_ARGV]; // argument strings (only for filter defs) + YAML::Node node; // argument node (only for filter defs) // methods bool method_restriction_enabled; @@ -133,6 +135,7 @@ class acl_filter_rule ~acl_filter_rule(); void name(const char *_name = nullptr); int add_argv(int _argc, char *_argv[]); + void add_node(const YAML::Node &_node); void print(); /** Return a description of the action. diff --git a/include/proxy/http/remap/RemapConfig.h b/include/proxy/http/remap/RemapConfig.h index 8456dd846c3..bdba861f862 100644 --- a/include/proxy/http/remap/RemapConfig.h +++ b/include/proxy/http/remap/RemapConfig.h @@ -64,7 +64,7 @@ struct BUILD_TABLE_INFO { bool ip_allow_check_enabled_p = true; bool accept_check_p = true; - acl_filter_rule *rules_list = nullptr; // all rules defined in config files as .define_filter foobar @src_ip=..... + acl_filter_rule *rules_list = nullptr; // all rules defined in config files UrlRewrite *rewrite = nullptr; // Pointer to the UrlRewrite object we are parsing for. // Clear the argument vector. @@ -92,3 +92,8 @@ bool remap_parse_config(const char *path, UrlRewrite *rewrite); using load_remap_file_func = void (*)(const char *, const char *); extern load_remap_file_func load_remap_file_cb; + +// Helper functions shared between RemapConfig.cc and RemapYamlConfig.cc +bool is_inkeylist(const char *key, ...); +void free_directory_list(int n_entries, struct dirent **entrylist); +const char *is_valid_scheme(std::string_view fromScheme, std::string_view toScheme); diff --git a/include/proxy/http/remap/RemapYamlConfig.h b/include/proxy/http/remap/RemapYamlConfig.h new file mode 100644 index 00000000000..a5ec4919c0a --- /dev/null +++ b/include/proxy/http/remap/RemapYamlConfig.h @@ -0,0 +1,73 @@ +/** @file + * + * YAML remap configuration file parsing. + * + * @section license License + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include + +#include + +#include "swoc/Errata.h" +#include "proxy/http/remap/RemapConfig.h" +#include "proxy/hdrs/URL.h" + +class acl_filter_rule; +class UrlRewrite; +class url_mapping; + +// Parse URL from YAML node into URL object +swoc::Errata parse_yaml_url(const YAML::Node &node, URL &url, bool host_check, std::string_view &url_str); + +// Parse and validate ACL filters from YAML node +swoc::Errata remap_validate_yaml_filter_args(acl_filter_rule **rule_pp, const YAML::Node &node, ACLBehaviorPolicy behavior_policy); + +// Parse map_with_referer YAML node +swoc::Errata parse_map_referer(const YAML::Node &node, url_mapping *url_mapping); + +// Parse plugin YAML node +swoc::Errata parse_yaml_plugins(const YAML::Node &node, url_mapping *url_mapping, BUILD_TABLE_INFO *bti); + +// Parse filter directive YAML nodes (activate_filter|deactivate_filter|delete_filter|define_filter) +swoc::Errata parse_yaml_filter_directive(const YAML::Node &node, BUILD_TABLE_INFO *bti); + +// Parse define filter directice YAML node (for inline and global filters) +swoc::Errata parse_yaml_define_directive(const YAML::Node &node, BUILD_TABLE_INFO *bti); + +// Parse remap filter YAML node +swoc::Errata process_yaml_filter_opt(url_mapping *mp, const YAML::Node &node, const BUILD_TABLE_INFO *bti); + +// Parse yaml subpath +swoc::Errata parse_yaml_remap_fragment(const char *path, BUILD_TABLE_INFO *bti); + +// Parse include directive for remap subpaths +swoc::Errata parse_yaml_include_directive(const std::string &include_path, BUILD_TABLE_INFO *bti); + +// Parse for single remap rule YAML node +swoc::Errata parse_yaml_remap_rule(const YAML::Node &node, BUILD_TABLE_INFO *bti); + +// Parse remap YAML node +bool remap_parse_yaml_bti(const char *path, BUILD_TABLE_INFO *bti); + +bool remap_parse_yaml(const char *path, UrlRewrite *rewrite); diff --git a/include/proxy/http/remap/UrlRewrite.h b/include/proxy/http/remap/UrlRewrite.h index dcc767ebe44..dc890a0b812 100644 --- a/include/proxy/http/remap/UrlRewrite.h +++ b/include/proxy/http/remap/UrlRewrite.h @@ -117,6 +117,18 @@ class UrlRewrite : public RefCountObjInHeap return _valid; }; + bool + is_remap_yaml() const + { + return _remap_yaml; + }; + + void + set_remap_yaml(bool yaml) + { + _remap_yaml = yaml; + }; + /// @return Number of rules defined. int rule_count() const @@ -234,6 +246,7 @@ class UrlRewrite : public RefCountObjInHeap private: bool _valid = false; + bool _remap_yaml = false; ACLBehaviorPolicy _acl_behavior_policy = ACLBehaviorPolicy::ACL_BEHAVIOR_LEGACY; bool _mappingLookup(MappingsStore &mappings, URL *request_url, int request_port, const char *request_host, int request_host_len, @@ -251,3 +264,5 @@ class UrlRewrite : public RefCountObjInHeap }; void url_rewrite_remap_request(const UrlMappingContainer &mapping_container, URL *request_url, int scheme = -1); + +mapping_type get_mapping_type(const char *type_str, BUILD_TABLE_INFO *bti); diff --git a/include/tscore/Filenames.h b/include/tscore/Filenames.h index 90ae0485df9..de6e022db98 100644 --- a/include/tscore/Filenames.h +++ b/include/tscore/Filenames.h @@ -39,6 +39,7 @@ namespace filename constexpr const char *SOCKS = "socks.config"; constexpr const char *PARENT = "parent.config"; constexpr const char *REMAP = "remap.config"; + constexpr const char *REMAP_YAML = "remap.yaml"; constexpr const char *SSL_MULTICERT = "ssl_multicert.config"; constexpr const char *SPLITDNS = "splitdns.config"; constexpr const char *SNI = "sni.yaml"; diff --git a/src/mgmt/config/AddConfigFilesHere.cc b/src/mgmt/config/AddConfigFilesHere.cc index 38ff8abad96..e7ac6a968d5 100644 --- a/src/mgmt/config/AddConfigFilesHere.cc +++ b/src/mgmt/config/AddConfigFilesHere.cc @@ -71,6 +71,7 @@ initializeRegistry() registerFile("proxy.config.cache.ip_categories.filename", ts::filename::IP_CATEGORIES, NOT_REQUIRED); registerFile("proxy.config.http.parent_proxy.file", ts::filename::PARENT, NOT_REQUIRED); registerFile("proxy.config.url_remap.filename", ts::filename::REMAP, NOT_REQUIRED); + registerFile("proxy.config.url_remap_yaml.filename", ts::filename::REMAP_YAML, NOT_REQUIRED); registerFile("", ts::filename::VOLUME, NOT_REQUIRED); registerFile("proxy.config.cache.hosting_filename", ts::filename::HOSTING, NOT_REQUIRED); registerFile("", ts::filename::PLUGIN, NOT_REQUIRED); diff --git a/src/proxy/ReverseProxy.cc b/src/proxy/ReverseProxy.cc index 341c2c7de40..e350479b2c4 100644 --- a/src/proxy/ReverseProxy.cc +++ b/src/proxy/ReverseProxy.cc @@ -71,13 +71,18 @@ init_reverse_proxy() rewrite_table = new UrlRewrite(); Note("%s loading ...", ts::filename::REMAP); - if (!rewrite_table->load()) { - Emergency("%s failed to load", ts::filename::REMAP); + Note("%s loading ...", ts::filename::REMAP_YAML); + bool status = rewrite_table->load(); + bool is_yaml = (rewrite_table->is_remap_yaml()); + + if (!status) { + Emergency("%s failed to load", is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); } else { - Note("%s finished loading", ts::filename::REMAP); + Note("%s finished loading", is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); } RecRegisterConfigUpdateCb("proxy.config.url_remap.filename", url_rewrite_CB, (void *)FILE_CHANGED); + RecRegisterConfigUpdateCb("proxy.config.url_remap_yaml.filename", url_rewrite_CB, (void *)FILE_CHANGED); RecRegisterConfigUpdateCb("proxy.config.proxy_name", url_rewrite_CB, (void *)TSNAME_CHANGED); RecRegisterConfigUpdateCb("proxy.config.reverse_proxy.enabled", url_rewrite_CB, (void *)REVERSE_CHANGED); RecRegisterConfigUpdateCb("proxy.config.http.referer_default_redirect", url_rewrite_CB, (void *)HTTP_DEFAULT_REDIRECT_CHANGED); @@ -139,9 +144,15 @@ reloadUrlRewrite() UrlRewrite *newTable, *oldTable; Note("%s loading ...", ts::filename::REMAP); + Note("%s loading ...", ts::filename::REMAP_YAML); Dbg(dbg_ctl_url_rewrite, "%s updated, reloading...", ts::filename::REMAP); + Dbg(dbg_ctl_url_rewrite, "%s updated, reloading...", ts::filename::REMAP_YAML); newTable = new UrlRewrite(); - if (newTable->load()) { + + bool status = newTable->load(); + bool is_yaml = (newTable->is_remap_yaml()); + + if (status) { static const char *msg_format = "%s finished loading"; // Hold at least one lease, until we reload the configuration @@ -155,15 +166,15 @@ reloadUrlRewrite() // Release the old one oldTable->release(); - Dbg(dbg_ctl_url_rewrite, msg_format, ts::filename::REMAP); - Note(msg_format, ts::filename::REMAP); + Dbg(dbg_ctl_url_rewrite, msg_format, is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); + Note(msg_format, is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); return true; } else { static const char *msg_format = "%s failed to load"; delete newTable; - Dbg(dbg_ctl_url_rewrite, msg_format, ts::filename::REMAP); - Error(msg_format, ts::filename::REMAP); + Dbg(dbg_ctl_url_rewrite, msg_format, is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); + Error(msg_format, is_yaml ? ts::filename::REMAP_YAML : ts::filename::REMAP); return false; } } diff --git a/src/proxy/http/remap/AclFiltering.cc b/src/proxy/http/remap/AclFiltering.cc index db48d09d32c..44ecde767de 100644 --- a/src/proxy/http/remap/AclFiltering.cc +++ b/src/proxy/http/remap/AclFiltering.cc @@ -96,6 +96,12 @@ acl_filter_rule::add_argv(int _argc, char *_argv[]) return real_cnt; } +void +acl_filter_rule::add_node(const YAML::Node &_node) +{ + node = _node; +} + void acl_filter_rule::name(const char *_name) { diff --git a/src/proxy/http/remap/CMakeLists.txt b/src/proxy/http/remap/CMakeLists.txt index ff0f63da03e..e6d4771049b 100644 --- a/src/proxy/http/remap/CMakeLists.txt +++ b/src/proxy/http/remap/CMakeLists.txt @@ -24,6 +24,7 @@ add_library( NextHopRoundRobin.cc NextHopStrategyFactory.cc RemapConfig.cc + RemapYamlConfig.cc RemapPluginInfo.cc PluginDso.cc PluginFactory.cc diff --git a/src/proxy/http/remap/RemapConfig.cc b/src/proxy/http/remap/RemapConfig.cc index a1ceac4e0ee..92ae3266dd5 100644 --- a/src/proxy/http/remap/RemapConfig.cc +++ b/src/proxy/http/remap/RemapConfig.cc @@ -76,6 +76,31 @@ UrlWhack(char *toWhack, int *origLength) return length; } +const char * +is_valid_scheme(std::string_view fromScheme, std::string_view toScheme) +{ + const char *errStr = nullptr; + // Include support for HTTPS scheme + // includes support for FILE scheme + if ((fromScheme != std::string_view{URL_SCHEME_HTTP} && fromScheme != std::string_view{URL_SCHEME_HTTPS} && + fromScheme != std::string_view{URL_SCHEME_FILE} && fromScheme != std::string_view{URL_SCHEME_TUNNEL} && + fromScheme != std::string_view{URL_SCHEME_WS} && fromScheme != std::string_view{URL_SCHEME_WSS} && + fromScheme != std::string_view{URL_SCHEME_HTTP_UDS} && fromScheme != std::string_view{URL_SCHEME_HTTPS_UDS}) || + (toScheme != std::string_view{URL_SCHEME_HTTP} && toScheme != std::string_view{URL_SCHEME_HTTPS} && + toScheme != std::string_view{URL_SCHEME_TUNNEL} && toScheme != std::string_view{URL_SCHEME_WS} && + toScheme != std::string_view{URL_SCHEME_WSS})) { + errStr = "only http, https, http+unix, https+unix, ws, wss, and tunnel remappings are supported"; + return errStr; + } + + // If mapping from WS or WSS we must map out to WS or WSS + if ((fromScheme == std::string_view{URL_SCHEME_WSS} || fromScheme == std::string_view{URL_SCHEME_WS}) && + (toScheme != std::string_view{URL_SCHEME_WSS} && toScheme != std::string_view{URL_SCHEME_WS})) { + errStr = "WS or WSS can only be mapped out to WS or WSS."; + } + return errStr; +} + /** Cleanup *char[] array - each item in array must be allocated via ats_malloc or similar "x..." function. @@ -175,7 +200,7 @@ process_filter_opt(url_mapping *mp, const BUILD_TABLE_INFO *bti, char *errStrBuf return errStr; } -static bool +bool is_inkeylist(const char *key, ...) { va_list ap; @@ -303,7 +328,7 @@ parse_deactivate_directive(const char *directive, BUILD_TABLE_INFO *bti, char *e return nullptr; } -static void +void free_directory_list(int n_entries, struct dirent **entrylist) { for (int i = 0; i < n_entries; ++i) { @@ -1155,28 +1180,8 @@ remap_parse_config_bti(const char *path, BUILD_TABLE_INFO *bti) type_id_str = is_cur_mapping_regex ? (bti->paramv[0] + 6) : bti->paramv[0]; // Check to see whether is a reverse or forward mapping - if (!strcasecmp("reverse_map", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::REVERSE_MAP"); - maptype = mapping_type::REVERSE_MAP; - } else if (!strcasecmp("map", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - %s", - ((bti->remap_optflg & REMAP_OPTFLG_MAP_WITH_REFERER) == 0) ? "mapping_type::FORWARD_MAP" : - "mapping_type::FORWARD_MAP_REFERER"); - maptype = - ((bti->remap_optflg & REMAP_OPTFLG_MAP_WITH_REFERER) == 0) ? mapping_type::FORWARD_MAP : mapping_type::FORWARD_MAP_REFERER; - } else if (!strcasecmp("redirect", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::PERMANENT_REDIRECT"); - maptype = mapping_type::PERMANENT_REDIRECT; - } else if (!strcasecmp("redirect_temporary", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::TEMPORARY_REDIRECT"); - maptype = mapping_type::TEMPORARY_REDIRECT; - } else if (!strcasecmp("map_with_referer", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::FORWARD_MAP_REFERER"); - maptype = mapping_type::FORWARD_MAP_REFERER; - } else if (!strcasecmp("map_with_recv_port", type_id_str)) { - Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::FORWARD_MAP_WITH_RECV_PORT"); - maptype = mapping_type::FORWARD_MAP_WITH_RECV_PORT; - } else { + maptype = get_mapping_type(type_id_str, bti); + if (maptype == mapping_type::NONE) { snprintf(errStrBuf, sizeof(errStrBuf), "unknown mapping type at line %d", cln + 1); errStr = errStrBuf; goto MAP_ERROR; @@ -1251,23 +1256,8 @@ remap_parse_config_bti(const char *path, BUILD_TABLE_INFO *bti) } toScheme = new_mapping->toURL.scheme_get(); - // Include support for HTTPS scheme - // includes support for FILE scheme - if ((fromScheme != std::string_view{URL_SCHEME_HTTP} && fromScheme != std::string_view{URL_SCHEME_HTTPS} && - fromScheme != std::string_view{URL_SCHEME_FILE} && fromScheme != std::string_view{URL_SCHEME_TUNNEL} && - fromScheme != std::string_view{URL_SCHEME_WS} && fromScheme != std::string_view{URL_SCHEME_WSS} && - fromScheme != std::string_view{URL_SCHEME_HTTP_UDS} && fromScheme != std::string_view{URL_SCHEME_HTTPS_UDS}) || - (toScheme != std::string_view{URL_SCHEME_HTTP} && toScheme != std::string_view{URL_SCHEME_HTTPS} && - toScheme != std::string_view{URL_SCHEME_TUNNEL} && toScheme != std::string_view{URL_SCHEME_WS} && - toScheme != std::string_view{URL_SCHEME_WSS})) { - errStr = "only http, https, http+unix, https+unix, ws, wss, and tunnel remappings are supported"; - goto MAP_ERROR; - } - - // If mapping from WS or WSS we must map out to WS or WSS - if ((fromScheme == std::string_view{URL_SCHEME_WSS} || fromScheme == std::string_view{URL_SCHEME_WS}) && - (toScheme != std::string_view{URL_SCHEME_WSS} && toScheme != std::string_view{URL_SCHEME_WS})) { - errStr = "WS or WSS can only be mapped out to WS or WSS."; + errStr = is_valid_scheme(fromScheme, toScheme); + if (errStr != nullptr) { goto MAP_ERROR; } diff --git a/src/proxy/http/remap/RemapYamlConfig.cc b/src/proxy/http/remap/RemapYamlConfig.cc new file mode 100644 index 00000000000..4cd8ad8ace6 --- /dev/null +++ b/src/proxy/http/remap/RemapYamlConfig.cc @@ -0,0 +1,1170 @@ +/** @file + * + * YAML remap configuration file parsing implementation. + * + * @section license License + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "proxy/http/remap/RemapYamlConfig.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "tscore/Diags.h" +#include "tscore/ink_string.h" +#include "tsutil/ts_errata.h" +#include "swoc/bwf_base.h" +#include "swoc/bwf_ex.h" +#include "swoc/swoc_file.h" + +#include "proxy/http/remap/UrlRewrite.h" +#include "proxy/http/remap/UrlMapping.h" +#include "proxy/http/remap/RemapConfig.h" +#include "proxy/http/remap/AclFiltering.h" +#include "records/RecCore.h" + +namespace +{ +DbgCtl dbg_ctl_remap_yaml{"remap_yaml"}; +DbgCtl dbg_ctl_url_rewrite{"url_rewrite"}; + +/** will process the regex mapping configuration and create objects in + output argument reg_map. It assumes existing data in reg_map is + inconsequential and will be perfunctorily null-ed; +*/ +static bool +process_regex_mapping_config(const char *from_host_lower, url_mapping *new_mapping, UrlRewrite::RegexMapping *reg_map) +{ + std::string_view to_host{}; + int to_host_len; + int substitution_id; + int32_t captures; + + reg_map->to_url_host_template = nullptr; + reg_map->to_url_host_template_len = 0; + reg_map->n_substitutions = 0; + + reg_map->url_map = new_mapping; + + // using from_host_lower (and not new_mapping->fromURL.host_get()) + // as this one will be nullptr-terminated (required by pcre_compile) + if (reg_map->regular_expression.compile(from_host_lower) == false) { + Warning("pcre_compile failed! Regex has error starting at %s", from_host_lower); + goto lFail; + } + + captures = reg_map->regular_expression.get_capture_count(); + if (captures == -1) { + Warning("pcre_fullinfo failed!"); + goto lFail; + } + if (captures >= UrlRewrite::MAX_REGEX_SUBS) { // off by one for $0 (implicit capture) + Warning("regex has %d capturing subpatterns (including entire regex); Max allowed: %d", captures + 1, + UrlRewrite::MAX_REGEX_SUBS); + goto lFail; + } + + to_host = new_mapping->toURL.host_get(); + to_host_len = static_cast(to_host.length()); + for (int i = 0; i < to_host_len - 1; ++i) { + if (to_host[i] == '$') { + substitution_id = to_host[i + 1] - '0'; + if ((substitution_id < 0) || (substitution_id > captures)) { + Warning("Substitution id [%c] has no corresponding capture pattern in regex [%s]", to_host[i + 1], from_host_lower); + goto lFail; + } + reg_map->substitution_markers[reg_map->n_substitutions] = i; + reg_map->substitution_ids[reg_map->n_substitutions] = substitution_id; + ++reg_map->n_substitutions; + } + } + + // so the regex itself is stored in fromURL.host; string to match + // will be in the request; string to use for substitutions will be + // in this buffer + reg_map->to_url_host_template_len = to_host_len; + reg_map->to_url_host_template = static_cast(ats_malloc(to_host_len)); + memcpy(reg_map->to_url_host_template, to_host.data(), to_host_len); + + return true; + +lFail: + ats_free(reg_map->to_url_host_template); + reg_map->to_url_host_template = nullptr; + reg_map->to_url_host_template_len = 0; + + return false; +} +} // end anonymous namespace + +swoc::Errata +parse_yaml_url(const YAML::Node &node, URL &url, bool host_check, std::string_view &url_str) +{ + if (!node || !node.IsMap()) { + return swoc::Errata("URL must be a map"); + } + url.create(nullptr); + + // Use url first if defined + ParseResult rparse; + if (node["url"]) { + url_str = node["url"].as(); + if (host_check) { + rparse = url.parse_regex(url_str); + } else { + rparse = url.parse_no_host_check(url_str); + } + if (rparse != ParseResult::DONE) { + return swoc::Errata("malformed URL: {}", url_str); + } + + return {}; + } + + // Build URL string from components + if (node["scheme"]) { + url.scheme_set(node["scheme"].as()); + } + + if (node["host"]) { + url.host_set(node["host"].as()); + } + + if (node["port"]) { + url.port_set(node["port"].as()); + } + + if (node["path"]) { + url.path_set(node["path"].as()); + } + + return {}; +} + +swoc::Errata +remap_validate_yaml_filter_args(acl_filter_rule **rule_pp, const YAML::Node &node, ACLBehaviorPolicy behavior_policy) +{ + acl_filter_rule *rule; + int j; + bool new_rule_flg = false; + + if (!rule_pp) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Invalid argument(s)"); + return swoc::Errata("Invalid argument(s)"); + } + + if (dbg_ctl_url_rewrite.on()) { + printf("validate_filter_args: "); + for (const auto &rule : node) { + printf("\"%s\" ", rule.first.as().c_str()); + } + printf("\n"); + } + + if ((rule = *rule_pp) == nullptr) { + rule = new acl_filter_rule(); + if (unlikely((*rule_pp = rule) == nullptr)) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Memory allocation error"); + return swoc::Errata("Memory allocation Error"); + } + new_rule_flg = true; + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] new acl_filter_rule class was created during remap rule processing"); + } + + if (!node || !node.IsMap()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return swoc::Errata("filters must be a map"); + } + + // Parse method + auto parse_method = [&](const std::string &method_str) { + int m = hdrtoken_tokenize(method_str.c_str(), method_str.length(), nullptr) - HTTP_WKSIDX_CONNECT; + + if (m >= 0 && m < HTTP_WKSIDX_METHODS_CNT) { + rule->standard_method_lookup[m] = true; + } else { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Using nonstandard method [%s]", method_str.c_str()); + rule->nonstandard_methods.insert(method_str); + } + rule->method_restriction_enabled = true; + }; + + if (node["method"]) { + if (node["method"].IsSequence()) { + for (const auto &method : node["method"]) { + parse_method(method.as()); + } + } else { + parse_method(node["method"].as()); + } + } + + // Parse src_ip (and src_ip_invert) + auto parse_src_ip = [&](const std::string &ip_str, bool invert) -> swoc::Errata { + if (rule->src_ip_cnt >= ACL_FILTER_MAX_SRC_IP) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Too many \"src_ip=\" filters"); + return swoc::Errata("Defined more than {} src_ip filters", ACL_FILTER_MAX_SRC_IP); + } + + src_ip_info_t *ipi = &rule->src_ip_array[rule->src_ip_cnt]; + if (invert) { + ipi->invert = true; + } + std::string_view arg{ip_str}; + if (arg == "all") { + ipi->match_all_addresses = true; + } else if (ats_ip_range_parse(arg, ipi->start, ipi->end) != 0) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Unable to parse IP value in %s", ip_str.c_str()); + return swoc::Errata("Unable to parse IP value: {}", ip_str); + } + for (j = 0; j < rule->src_ip_cnt; j++) { + if (rule->src_ip_array[j].start == ipi->start && rule->src_ip_array[j].end == ipi->end) { + ipi->reset(); + return {}; + } + } + if (ipi) { + rule->src_ip_cnt++; + rule->src_ip_valid = 1; + } + return {}; + }; + + if (node["src_ip"]) { + if (node["src_ip"].IsSequence()) { + for (const auto &src_ip : node["src_ip"]) { + auto errata = parse_src_ip(src_ip.as(), false); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } else { + auto errata = parse_src_ip(node["src_ip"].as(), false); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } + + if (node["src_ip_invert"]) { + if (node["src_ip_invert"].IsSequence()) { + for (const auto &src_ip_invert : node["src_ip_invert"]) { + auto errata = parse_src_ip(src_ip_invert.as(), true); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } else { + auto errata = parse_src_ip(node["src_ip_invert"].as(), true); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } + + // Parse src_ip_category (and src_ip_category) + auto parse_src_ip_category = [&](const std::string &ip_category, bool invert) -> swoc::Errata { + if (rule->src_ip_category_cnt >= ACL_FILTER_MAX_SRC_IP) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Too many \"src_ip_category=\" filters"); + return swoc::Errata("Defined more than {} src_ip_category filters", ACL_FILTER_MAX_SRC_IP); + } + src_ip_category_info_t *ipi = &rule->src_ip_category_array[rule->src_ip_category_cnt]; + ipi->category.assign(ip_category); + if (invert) { + ipi->invert = true; + } + for (j = 0; j < rule->src_ip_category_cnt; j++) { + if (rule->src_ip_category_array[j].category == ipi->category) { + ipi->reset(); + return {}; + } + } + if (ipi) { + rule->src_ip_category_cnt++; + rule->src_ip_category_valid = 1; + } + return {}; + }; + + if (node["src_ip_category"]) { + auto errata = parse_src_ip_category(node["src_ip_category"].as(), false); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + + if (node["src_ip_category_invert"]) { + auto errata = parse_src_ip_category(node["src_ip_category_invert"].as(), true); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + + // Parse in_ip (and in_ip_invert) + auto parse_in_ip = [&](const std::string &in_ip, bool invert) -> swoc::Errata { + if (rule->in_ip_cnt >= ACL_FILTER_MAX_IN_IP) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Too many \"in_ip=\" filters"); + return swoc::Errata("Defined more than {} in_ip filters", ACL_FILTER_MAX_IN_IP); + } + src_ip_info_t *ipi = &rule->in_ip_array[rule->in_ip_cnt]; + if (invert) { + ipi->invert = true; + } + // important! use copy of argument + std::string_view arg{in_ip}; + if (arg == "all") { + ipi->match_all_addresses = true; + } else if (ats_ip_range_parse(arg, ipi->start, ipi->end) != 0) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Unable to parse IP value in %s", in_ip.c_str()); + return swoc::Errata("Unable to parse IP value: {}", in_ip); + } + for (j = 0; j < rule->in_ip_cnt; j++) { + if (rule->in_ip_array[j].start == ipi->start && rule->in_ip_array[j].end == ipi->end) { + ipi->reset(); + return {}; + } + } + if (ipi) { + rule->in_ip_cnt++; + rule->in_ip_valid = 1; + } + return {}; + }; + + if (node["in_ip"]) { + if (node["in_ip"].IsSequence()) { + for (const auto &in_ip : node["in_ip"]) { + auto errata = parse_in_ip(in_ip.as(), false); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } else { + auto errata = parse_in_ip(node["in_ip"].as(), false); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } + + if (node["in_ip_invert"]) { + if (node["in_ip_invert"].IsSequence()) { + for (const auto &in_ip_invert : node["in_ip_invert"]) { + auto errata = parse_in_ip(in_ip_invert.as(), true); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } else { + auto errata = parse_in_ip(node["in_ip_invert"].as(), true); + if (!errata.is_ok()) { + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return errata; + } + } + } + + // Parse action + if (node["action"]) { + if (node["action"].IsSequence()) { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Only one action is allowed per remap ACL"); + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return swoc::Errata("Only one action is allowed per remap ACL"); + } + std::string action_str = node["action"].as(); + if (behavior_policy == ACLBehaviorPolicy::ACL_BEHAVIOR_MODERN) { + // With the new matching policy, we don't allow the legacy "allow" and + // "deny" actions. Users must transition to either add_allow/add_deny or + // set_allow/set_deny. + if (is_inkeylist(action_str.c_str(), "allow", "deny", nullptr)) { + Dbg(dbg_ctl_url_rewrite, + R"([validate_filter_args] "allow" and "deny" are no longer valid. Use add_allow/add_deny or set_allow/set_deny: "%s"")", + action_str.c_str()); + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return swoc::Errata("\"allow\" and \"deny\" are no longer valid. Use add_allow/add_deny or set_allow/set_deny: {}", + action_str.c_str()); + } + } + if (is_inkeylist(action_str.c_str(), "add_allow", "add_deny", nullptr)) { + rule->add_flag = 1; + } else { + rule->add_flag = 0; + } + // Remove "deny" from this list when MATCH_ON_IP_AND_METHOD is removed in 11.x. + if (is_inkeylist(action_str.c_str(), "0", "off", "deny", "set_deny", "add_deny", "disable", nullptr)) { + rule->allow_flag = 0; + // Remove "allow" from this list when MATCH_ON_IP_AND_METHOD is removed in 11.x. + } else if (is_inkeylist(action_str.c_str(), "1", "on", "allow", "set_allow", "add_allow", "enable", nullptr)) { + rule->allow_flag = 1; + } else { + Dbg(dbg_ctl_url_rewrite, "[validate_filter_args] Unknown argument \"%s\"", action_str.c_str()); + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return swoc::Errata("Unknown action: {}", action_str); + } + } + + // Parse internal + if (node["internal"] && node["internal"].as()) { + rule->internal = 1; + } + + if (dbg_ctl_url_rewrite.on()) { + rule->print(); + } + + return {}; +} + +swoc::Errata +parse_map_referer(const YAML::Node &node, url_mapping *url_mapping) +{ + if (!node || !node.IsMap()) { + return swoc::Errata("redirect must be a map"); + } + + if (!node["url"]) { + return swoc::Errata("Missing 'url' field in redirect map-with-referer"); + } + std::string url = node["url"].as(); + url_mapping->filter_redirect_url = ats_strdup(url.c_str()); + if (!strcasecmp(url.c_str(), "") || !strcasecmp(url.c_str(), "default") || + !strcasecmp(url.c_str(), "") || !strcasecmp(url.c_str(), "default_redirect_url")) { + url_mapping->default_redirect_url = true; + } + url_mapping->redir_chunk_list = redirect_tag_str::parse_format_redirect_url(ats_strdup(url.c_str())); + + if (!node["regex"] || !node["regex"].IsSequence()) { + return swoc::Errata("'regex' field must be sequence"); + } + + referer_info *ri; + for (const auto &rule : node["regex"]) { + char refinfo_error_buf[1024]; + bool refinfo_error = false; + std::string regex = rule.as(); + + ri = new referer_info(regex.c_str(), &refinfo_error, refinfo_error_buf, sizeof(refinfo_error_buf)); + if (refinfo_error) { + delete ri; + ri = nullptr; + return swoc::Errata("Incorrect Referer regular expression \"{}\" - {}", regex.c_str(), refinfo_error_buf); + } + + if (ri && ri->negative) { + if (ri->any) { + url_mapping->optional_referer = true; /* referer header is optional */ + delete ri; + ri = nullptr; + } else { + url_mapping->negative_referer = true; /* we have negative referer in list */ + } + } + if (ri) { + ri->next = url_mapping->referer_list; + url_mapping->referer_list = ri; + } + } + return {}; +} + +swoc::Errata +parse_yaml_plugins(const YAML::Node &node, url_mapping *url_mapping, BUILD_TABLE_INFO *bti) +{ + char *err; + char *pargv[1024]; + int parc = 0; + memset(pargv, 0, sizeof(pargv)); + + if (!node["name"]) { + return swoc::Errata("plugin missing 'name' field"); + } + + std::string plugin_name = node["name"].as(); + Dbg(dbg_ctl_remap_yaml, "Loading plugin: %s", plugin_name.c_str()); + + /* Prepare remap plugin parameters from the config */ + if ((err = url_mapping->fromURL.string_get(nullptr)) == nullptr) { + return swoc::Errata("Can't load fromURL from URL class"); + } + pargv[parc++] = ats_strdup(err); + ats_free(err); + + if ((err = url_mapping->toURL.string_get(nullptr)) == nullptr) { + return swoc::Errata("Can't load toURL from URL class"); + } + pargv[parc++] = ats_strdup(err); + ats_free(err); + + // Add plugin parameters + if (node["params"] && node["params"].IsSequence()) { + for (const auto ¶m : node["params"]) { + std::string pparam_str = param.as(); + pargv[parc++] = ats_strdup(pparam_str.c_str()); + Dbg(dbg_ctl_remap_yaml, " Plugin param: %s", pparam_str.c_str()); + } + } + + RemapPluginInst *pi = nullptr; + std::string error; + { + uint32_t elevate_access = 0; + elevate_access = RecGetRecordInt("proxy.config.plugin.load_elevated").value_or(0); + ElevateAccess access(elevate_access ? ElevateAccess::FILE_PRIVILEGE : 0); + + pi = + bti->rewrite->pluginFactory.getRemapPlugin(swoc::file::path(plugin_name), parc, pargv, error, isPluginDynamicReloadEnabled()); + } // done elevating access + + if (nullptr == pi) { + return swoc::Errata("failed to instantiate plugin ({}) to remap rule: {}", plugin_name.c_str(), error.c_str()); + } else { + url_mapping->add_plugin_instance(pi); + } + + ats_free(pargv[0]); // fromURL + ats_free(pargv[1]); // toURL + + return {}; +} + +swoc::Errata +parse_yaml_filter_directive(const YAML::Node &node, BUILD_TABLE_INFO *bti) +{ + acl_filter_rule *rp; + + // Check for activate_filters directive + if (node["activate_filter"]) { + std::string filter_name = node["activate_filter"].as(); + + // Check if for ip_allow filter + if (strcmp(filter_name.c_str(), "ip_allow") == 0) { + bti->ip_allow_check_enabled_p = true; + return {}; + ; + } + + if ((rp = acl_filter_rule::find_byname(bti->rules_list, filter_name.c_str())) == nullptr) { + Dbg(dbg_ctl_url_rewrite, "(Undefined filter '%s' in activate_filter directive)", filter_name.c_str()); + return swoc::Errata("(Undefined filter '{}' in activate_filter directive)", filter_name.c_str()); + } + + acl_filter_rule::requeue_in_active_list(&bti->rules_list, rp); + return {}; + } + + // Check for deactivate_filters directive + if (node["deactivate_filter"]) { + std::string filter_name = node["deactivate_filter"].as(); + + // Check if for ip_allow filter + if (strcmp(filter_name.c_str(), "ip_allow") == 0) { + bti->ip_allow_check_enabled_p = false; + return {}; + } + + if ((rp = acl_filter_rule::find_byname(bti->rules_list, filter_name.c_str())) == nullptr) { + Dbg(dbg_ctl_url_rewrite, "(Undefined filter '%s' in deactivate_filter directive)", filter_name.c_str()); + return swoc::Errata("(Undefined filter '{}' in deactivate_filter directive)", filter_name.c_str()); + } + + acl_filter_rule::requeue_in_passive_list(&bti->rules_list, rp); + return {}; + } + + // Check for delete_filters directive + if (node["delete_filter"]) { + std::string filter_name = node["delete_filter"].as(); + + acl_filter_rule::delete_byname(&bti->rules_list, filter_name.c_str()); + return {}; + } + + // Check for define_filter directive + if (node["define_filter"]) { + return parse_yaml_define_directive(node["define_filter"], bti); + } + + return swoc::Errata("Not a filter directive"); +} + +swoc::Errata +parse_yaml_define_directive(const YAML::Node &node, BUILD_TABLE_INFO *bti) +{ + bool flg; + acl_filter_rule *rp; + swoc::Errata errata; + + if (!node || !node.IsMap()) { + return swoc::Errata("named filters must be a map"); + } + + // When iterating over a YAML map, each element is a key-value pair + // We expect a single-entry map here + auto it = node.begin(); + std::string filter_name = it->first.as(); + const YAML::Node filter_spec = it->second; + + flg = ((rp = acl_filter_rule::find_byname(bti->rules_list, filter_name.c_str())) == nullptr) ? true : false; + // coverity[alloc_arg] + if ((errata = remap_validate_yaml_filter_args(&rp, filter_spec, bti->behavior_policy)).is_ok() && rp) { + if (flg) { // new filter - add to list + acl_filter_rule **rpp = nullptr; + Dbg(dbg_ctl_url_rewrite, "[parse_directive] new rule \"%s\" was created", filter_name.c_str()); + for (rpp = &bti->rules_list; *rpp; rpp = &((*rpp)->next)) { + ; + } + (*rpp = rp)->name(filter_name.c_str()); + } + Dbg(dbg_ctl_url_rewrite, "[parse_directive] %zu argument(s) were added to rule \"%s\"", filter_spec.size(), + filter_name.c_str()); + rp->add_node(filter_spec); // store string arguments for future processing + } + return errata; +} + +swoc::Errata +process_yaml_filter_opt(url_mapping *mp, const YAML::Node &node, const BUILD_TABLE_INFO *bti) +{ + acl_filter_rule *rp, **rpp; + swoc::Errata errata; + + if (unlikely(!mp || !bti)) { + Dbg(dbg_ctl_url_rewrite, "[process_yaml_filter_opt] Invalid argument(s)"); + return swoc::Errata("[process_yaml_filter_opt] Invalid argument(s)"); + } + // ACLs are processed in this order: + // 1. A remap.config ACL line for an individual remap rule. + // 2. All named ACLs in remap.config. + // 3. Rules as specified in ip_allow.yaml. + if (node["acl_filter"]) { + Dbg(dbg_ctl_url_rewrite, "[process_yaml_filter_opt] Add per remap filter"); + for (rpp = &mp->filter; *rpp; rpp = &((*rpp)->next)) { + ; + } + errata = remap_validate_yaml_filter_args(rpp, node["acl_filter"], bti->behavior_policy); + } + + for (rp = bti->rules_list; rp; rp = rp->next) { + for (rpp = &mp->filter; *rpp; rpp = &((*rpp)->next)) { + ; + } + if (rp->active_queue_flag) { + Dbg(dbg_ctl_url_rewrite, "[process_yaml_filter_opt] Add active main filter \"%s\"", + rp->filter_name ? rp->filter_name : ""); + for (rpp = &mp->filter; *rpp; rpp = &((*rpp)->next)) { + ; + } + errata = remap_validate_yaml_filter_args(rpp, rp->node, bti->behavior_policy); + if (!errata.is_ok()) { + break; + } + if (auto rule = *rpp; rule) { + // If no IP addresses are listed, treat that like `@src_ip=all`. + if (rule->src_ip_valid == 0 && rule->src_ip_cnt == 0) { + src_ip_info_t *ipi = &rule->src_ip_array[rule->src_ip_cnt]; + ipi->match_all_addresses = true; + rule->src_ip_cnt++; + rule->src_ip_valid = 1; + } + } + } + } + + // Set the ip allow flag for this rule to the current ip allow flag state + mp->ip_allow_check_enabled_p = bti->ip_allow_check_enabled_p; + + return errata; +} + +swoc::Errata +parse_yaml_remap_fragment(const char *path, BUILD_TABLE_INFO *bti) +{ + // We need to create a new bti so that we don't clobber any state in the parent parse, but we want + // to keep the ACL rules from the parent because ACLs must be global across the full set of config + // files. + BUILD_TABLE_INFO nbti; + bool success; + + if (access(path, R_OK) == -1) { + return swoc::Errata("{}: {}", path, strerror(errno)); + } + + nbti.rules_list = bti->rules_list; + nbti.rewrite = bti->rewrite; + + Dbg(dbg_ctl_url_rewrite, "[%s] including remap configuration from %s", __func__, path); + success = remap_parse_yaml_bti(path, &nbti); + + // The sub-parse might have updated the rules list, so push it up to the parent parse. + bti->rules_list = nbti.rules_list; + + if (success) { + // register the included file with the management subsystem so that we can correctly + // reload them when they change + load_remap_file_cb(ts::filename::REMAP, path); + } else { + return swoc::Errata("failed to parse included file {}", path); + } + + return {}; +} + +swoc::Errata +parse_yaml_include_directive(const std::string &include_path, BUILD_TABLE_INFO *bti) +{ + ats_scoped_str path; + swoc::Errata errata; + + // The included path is relative to SYSCONFDIR + path = RecConfigReadConfigPath(nullptr, include_path.c_str()); + + if (ink_file_is_directory(path)) { + struct dirent **entrylist; + int n_entries; + + n_entries = scandir(path, &entrylist, nullptr, alphasort); + if (n_entries == -1) { + return swoc::Errata("failed to open {}: {}", path.get(), strerror(errno)); + } + + for (int j = 0; j < n_entries; ++j) { + ats_scoped_str subpath; + + if (isdot(entrylist[j]->d_name) || isdotdot(entrylist[j]->d_name)) { + continue; + } + + subpath = Layout::relative_to(path.get(), entrylist[j]->d_name); + + if (ink_file_is_directory(subpath)) { + continue; + } + + errata = parse_yaml_remap_fragment(subpath, bti); + if (!errata.is_ok()) { + break; + } + } + + free_directory_list(n_entries, entrylist); + + } else { + errata = parse_yaml_remap_fragment(path, bti); + } + + return errata; +} + +swoc::Errata +parse_yaml_remap_rule(const YAML::Node &node, BUILD_TABLE_INFO *bti) +{ + std::string errStr; + + std::string_view fromScheme{}, toScheme{}; + std::string_view fromHost{}, toHost{}; + std::string_view fromUrl{}, toUrl{}; + std::string_view fromPath; + char *fromHost_lower = nullptr; + char *fromHost_lower_ptr = nullptr; + char fromHost_lower_buf[1024]; + mapping_type maptype; + url_mapping *new_mapping = nullptr; + + UrlRewrite::RegexMapping *reg_map = nullptr; + bool is_cur_mapping_regex; + const char *type_id_str; + + swoc::Errata errata; + const char *valid_scheme = nullptr; + + if (!node || !node.IsMap()) { + return swoc::Errata("remap rule must be a map"); + } + + // Parse for include directive first + if (node["include"]) { + return parse_yaml_include_directive(node["include"].as(), bti); + } + + // Parse for filter directives (activate/deactivate/delete/define) + if (node["activate_filter"] || node["deactivate_filter"] || node["delete_filter"] || node["define_filter"]) { + return parse_yaml_filter_directive(node, bti); + } + + // Parse rule type + if (!node["type"]) { + return swoc::Errata("remap rule missing 'type' field"); + } + std::string type_str = node["type"].as(); + + is_cur_mapping_regex = (strncasecmp("regex_", type_str.c_str(), 6) == 0); + type_id_str = is_cur_mapping_regex ? (type_str.c_str() + 6) : type_str.c_str(); + + // Check to see whether is a reverse or forward mapping + maptype = get_mapping_type(type_id_str, bti); + if (maptype == mapping_type::NONE) { + return swoc::Errata("unknown mapping type: {}", type_str); + } + + new_mapping = new url_mapping(); + + // apply filter rules if we have to + errata = process_yaml_filter_opt(new_mapping, node, bti); + if (!errata.is_ok()) { + swoc::bwprint(errStr, "Failed to process filter: {}", errata); + goto MAP_ERROR; + } + + // update sticky flag + bti->accept_check_p = bti->accept_check_p && bti->ip_allow_check_enabled_p; + + new_mapping->map_id = 0; + if (node["mapid"]) { + new_mapping->map_id = node["mapid"].as(); + } + + // Parse from URL + if (!node["from"]) { + errStr = "remap rule missing 'from' field"; + goto MAP_ERROR; + } + + errata = parse_yaml_url(node["from"], new_mapping->fromURL, true, fromUrl); + if (!errata.is_ok()) { + swoc::bwprint(errStr, "malformed From URL: {}", errata); + goto MAP_ERROR; + } + + // Parse to URL + if (!node["to"]) { + errStr = "remap rule missing 'to' field"; + goto MAP_ERROR; + } + + errata = parse_yaml_url(node["to"], new_mapping->toURL, false, toUrl); + if (!errata.is_ok()) { + swoc::bwprint(errStr, "malformed To URL: {}", errata); + goto MAP_ERROR; + } + + // Check if valid schemes + fromScheme = new_mapping->fromURL.scheme_get(); + toScheme = new_mapping->toURL.scheme_get(); + if (fromScheme.empty()) { + new_mapping->fromURL.scheme_set(std::string_view{URL_SCHEME_HTTP}); + new_mapping->wildcard_from_scheme = true; + fromScheme = new_mapping->fromURL.scheme_get(); + } + valid_scheme = is_valid_scheme(fromScheme, toScheme); + if (valid_scheme != nullptr) { + errStr = valid_scheme; + goto MAP_ERROR; + } + + // Check if map_with_referer is used + if (node["redirect"] && maptype == mapping_type::FORWARD_MAP_REFERER) { + errata = parse_map_referer(node["redirect"], new_mapping); + if (!errata.is_ok()) { + swoc::bwprint(errStr, "invalid map_with_referer: {}", errata); + goto MAP_ERROR; + } + } + + // Check to see the fromHost remapping is a relative one + fromHost = new_mapping->fromURL.host_get(); + if (fromHost.empty()) { + if (maptype == mapping_type::FORWARD_MAP || maptype == mapping_type::FORWARD_MAP_REFERER || + maptype == mapping_type::FORWARD_MAP_WITH_RECV_PORT) { + fromPath = new_mapping->fromURL.path_get(); + if ((fromPath.empty() || fromPath[0] != '/') && (fromUrl.empty() || fromUrl[0] != '/')) { + errStr = "relative remappings must begin with a /"; + goto MAP_ERROR; + } else { + fromHost = ""sv; + } + } else { + errStr = "remap source in reverse mappings requires a hostname"; + goto MAP_ERROR; + } + } + + toHost = new_mapping->toURL.host_get(); + if (toHost.empty()) { + errStr = "The remap destinations require a hostname"; + goto MAP_ERROR; + } + // Get rid of trailing slashes since they interfere + // with our ability to send redirects + + // You might be tempted to remove these lines but the new + // optimized header system will introduce problems. You + // might get two slashes occasionally instead of one because + // the rest of the system assumes that trailing slashes have + // been removed. + + if (unlikely(fromHost.length() >= sizeof(fromHost_lower_buf))) { + fromHost_lower = (fromHost_lower_ptr = static_cast(ats_malloc(fromHost.length() + 1))); + } else { + fromHost_lower = &fromHost_lower_buf[0]; + } + // Canonicalize the hostname by making it lower case + memcpy(fromHost_lower, fromHost.data(), fromHost.length()); + fromHost_lower[fromHost.length()] = 0; + LowerCaseStr(fromHost_lower); + + // set the normalized string so nobody else has to normalize this + new_mapping->fromURL.host_set({fromHost_lower, fromHost.length()}); + + if (is_cur_mapping_regex) { + reg_map = new UrlRewrite::RegexMapping(); + if (!process_regex_mapping_config(fromHost_lower, new_mapping, reg_map)) { + errStr = "could not process regex mapping config line"; + goto MAP_ERROR; + } + Dbg(dbg_ctl_url_rewrite, "Configured regex rule for host [%s]", fromHost_lower); + } + + // If a TS receives a request on a port which is set to tunnel mode + // (ie, blind forwarding) and a client connects directly to the TS, + // then the TS will use its IPv4 address and remap rules given + // to send the request to its proper destination. + // See HttpTransact::HandleBlindTunnel(). + // Therefore, for a remap with "type: map" and "scheme: tunnel", + // we also needs to convert hostname to its IPv4 addr + // and gives a new remap rule with the IPv4 addr. + if ((maptype == mapping_type::FORWARD_MAP || maptype == mapping_type::FORWARD_MAP_REFERER || + maptype == mapping_type::FORWARD_MAP_WITH_RECV_PORT) && + fromScheme == std::string_view{URL_SCHEME_TUNNEL} && (fromHost_lower[0] < '0' || fromHost_lower[0] > '9')) { + addrinfo *ai_records; // returned records. + ip_text_buffer ipb; // buffer for address string conversion. + if (0 == getaddrinfo(fromHost_lower, nullptr, nullptr, &ai_records)) { + for (addrinfo *ai_spot = ai_records; ai_spot; ai_spot = ai_spot->ai_next) { + if (ats_is_ip(ai_spot->ai_addr) && !ats_is_ip_any(ai_spot->ai_addr) && ai_spot->ai_protocol == IPPROTO_TCP) { + url_mapping *u_mapping; + + ats_ip_ntop(ai_spot->ai_addr, ipb, sizeof ipb); + u_mapping = new url_mapping; + u_mapping->fromURL.create(nullptr); + u_mapping->fromURL.copy(&new_mapping->fromURL); + u_mapping->fromURL.host_set({ipb}); + u_mapping->toURL.create(nullptr); + u_mapping->toURL.copy(&new_mapping->toURL); + + if (!bti->rewrite->InsertForwardMapping(maptype, u_mapping, ipb)) { + errStr = "unable to add mapping rule to lookup table"; + freeaddrinfo(ai_records); + goto MAP_ERROR; + } + } + } + + freeaddrinfo(ai_records); + } + } + + // check for a 'strategy' and if wire it up if one exists. + if (node["strategy"] && (maptype == mapping_type::FORWARD_MAP || maptype == mapping_type::FORWARD_MAP_REFERER || + maptype == mapping_type::FORWARD_MAP_WITH_RECV_PORT)) { + std::string strategy_name = node["strategy"].as(); + new_mapping->strategy = bti->rewrite->strategyFactory->strategyInstance(strategy_name.c_str()); + if (new_mapping->strategy == nullptr) { + errStr = "missing 'strategy' name argument, unable to add mapping rule"; + goto MAP_ERROR; + } + Dbg(dbg_ctl_url_rewrite, "mapped the 'strategy' named %s", strategy_name.c_str()); + } + + // Check "remap" plugin options and load .so object + if (node["plugins"] && (maptype == mapping_type::FORWARD_MAP || maptype == mapping_type::FORWARD_MAP_REFERER || + maptype == mapping_type::FORWARD_MAP_WITH_RECV_PORT)) { + if (!node["plugins"] || !node["plugins"].IsSequence()) { + errStr = "plugins must be a sequence"; + goto MAP_ERROR; + } + + for (const auto &plugin : node["plugins"]) { + errata = parse_yaml_plugins(plugin, new_mapping, bti); + if (!errata.is_ok()) { + swoc::bwprint(errStr, "{}", errata); + goto MAP_ERROR; + } + } + } + + // Now add the mapping to appropriate container + if (!bti->rewrite->InsertMapping(maptype, new_mapping, reg_map, fromHost_lower, is_cur_mapping_regex)) { + errStr = "unable to add mapping rule to lookup table"; + goto MAP_ERROR; + } + + ats_free_null(fromHost_lower_ptr); + + Dbg(dbg_ctl_remap_yaml, "Successfully added mapping rule"); + return {}; + +// Deal with error / warning scenarios +MAP_ERROR: + + Error("%s", errStr.c_str()); + + delete reg_map; + delete new_mapping; + return swoc::Errata(errStr); +} + +bool +remap_parse_yaml_bti(const char *path, BUILD_TABLE_INFO *bti) +{ + try { + Dbg(dbg_ctl_remap_yaml, "Parsing YAML config file: %s", path); + + YAML::Node config = YAML::LoadFile(path); + + if (config.IsNull()) { + Dbg(dbg_ctl_remap_yaml, "Empty YAML config file"); + return true; // a missing file is ok - treat as empty, no rules. + } + + Dbg(dbg_ctl_url_rewrite, "[BuildTable] UrlRewrite::BuildTable()"); + + ACLBehaviorPolicy behavior_policy = ACLBehaviorPolicy::ACL_BEHAVIOR_LEGACY; + if (!UrlRewrite::get_acl_behavior_policy(behavior_policy)) { + Warning("Failed to get ACL matching policy."); + return false; + } + bti->behavior_policy = behavior_policy; + + // Parse global filters section (optional) + if (config["acl_filters"] && config["acl_filters"].IsMap()) { + for (const auto &filter_def : config["acl_filters"]) { + auto errata = parse_yaml_define_directive(filter_def, bti); + if (!errata.is_ok()) { + Error("Failed to parse acl_filters section"); + return false; + } + } + } + + if (config["remap"].IsNull() || !config["remap"].IsSequence()) { + Error("Expected toplevel 'remap' key to be a sequence"); + return false; + } + + // Parse each remap rule + for (const auto &rule : config["remap"]) { + // Reset bti state for each rule (but keep rules_list for named filters) + bti->reset(); + + auto errata = parse_yaml_remap_rule(rule, bti); + if (!errata.is_ok()) { + Error("Failed to parse remap rule"); + return false; + } + } + + IpAllow::enableAcceptCheck(bti->accept_check_p); + + Dbg(dbg_ctl_remap_yaml, "Successfully parsed remap.yaml config"); + return true; + + } catch (YAML::Exception &ex) { + Error("YAML parsing error in %s: %s", path, ex.what()); + } catch (std::exception &ex) { + Error("Exception parsing YAML config %s: %s", path, ex.what()); + } + return false; +} + +bool +remap_parse_yaml(const char *path, UrlRewrite *rewrite) +{ + BUILD_TABLE_INFO bti; + + /* If this happens to be a config reload, the list of loaded remap plugins is non-empty, and we + * can signal all these plugins that a reload has begun. */ + rewrite->pluginFactory.indicatePreReload(); + + bti.rewrite = rewrite; + bool status = remap_parse_yaml_bti(path, &bti); + + /* Now after we parsed the configuration and (re)loaded plugins and plugin instances + * accordingly notify all plugins that we are done */ + rewrite->pluginFactory.indicatePostReload(status); + + bti.clear_acl_rules_list(); + + return status; +} diff --git a/src/proxy/http/remap/UrlRewrite.cc b/src/proxy/http/remap/UrlRewrite.cc index 6c7e3f48d6c..ddbc4e95726 100644 --- a/src/proxy/http/remap/UrlRewrite.cc +++ b/src/proxy/http/remap/UrlRewrite.cc @@ -23,6 +23,7 @@ */ #include "proxy/http/remap/UrlRewrite.h" +#include "proxy/http/remap/RemapYamlConfig.h" #include "iocore/eventsystem/ConfigProcessor.h" #include "proxy/ReverseProxy.h" #include "tscore/Layout.h" @@ -79,10 +80,19 @@ UrlRewrite::load() { ats_scoped_str config_file_path; - config_file_path = RecConfigReadConfigPath("proxy.config.url_remap.filename", ts::filename::REMAP); - if (!config_file_path) { - Warning("%s Unable to locate %s. No remappings in effect", modulePrefix, ts::filename::REMAP); - return false; + // Try remap.yaml first + config_file_path = RecConfigReadConfigPath("proxy.config.url_remap_yaml.filename", ts::filename::REMAP_YAML); + this->_remap_yaml = true; + + if (!config_file_path || !swoc::file::exists(swoc::file::path(config_file_path))) { + Dbg(dbg_ctl_url_rewrite, "%s doesn't exist yaml, fall back to remap.config", ts::filename::REMAP_YAML); + // Fall back to remap.config if remap.yaml not found + this->_remap_yaml = false; + config_file_path = RecConfigReadConfigPath("proxy.config.url_remap.filename", ts::filename::REMAP); + if (!config_file_path) { + Warning("%s Unable to locate %s. No remappings in effect", modulePrefix, ts::filename::REMAP); + return false; + } } this->ts_name = nullptr; @@ -136,6 +146,7 @@ UrlRewrite::load() int n_rules = this->rule_count(); // Minimum # of rules to be considered a valid configuration. int required_rules; required_rules = RecGetRecordInt("proxy.config.url_remap.min_rules_required").value_or(0); + Dbg(dbg_ctl_url_rewrite, "n_rules: %d, required_rules: %d", n_rules, required_rules); if (n_rules >= required_rules) { _valid = true; if (dbg_ctl_url_rewrite.on()) { @@ -835,7 +846,14 @@ UrlRewrite::BuildTable(const char *path) temporary_redirects.hash_lookup.reset(new URLTable); forward_mappings_with_recv_port.hash_lookup.reset(new URLTable); - if (!remap_parse_config(path, this)) { + bool parse_success; + if (is_remap_yaml()) { + parse_success = remap_parse_yaml(path, this); + } else { + parse_success = remap_parse_config(path, this); + } + + if (!parse_success) { return TS_ERROR; } @@ -1092,3 +1110,35 @@ UrlRewrite::_destroyList(RegexMappingList &mappings) } mappings.clear(); } + +/** + * Convert a YAML rule type string to a mapping_type enum. + */ +mapping_type +get_mapping_type(const char *type_str, BUILD_TABLE_INFO *bti) +{ + // Check to see whether is a reverse or forward mapping + if (!strcasecmp("reverse_map", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::REVERSE_MAP"); + return mapping_type::REVERSE_MAP; + } else if (!strcasecmp("map", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - %s", + ((bti->remap_optflg & REMAP_OPTFLG_MAP_WITH_REFERER) == 0) ? "mapping_type::FORWARD_MAP" : + "mapping_type::FORWARD_MAP_REFERER"); + return ((bti->remap_optflg & REMAP_OPTFLG_MAP_WITH_REFERER) == 0) ? mapping_type::FORWARD_MAP : + mapping_type::FORWARD_MAP_REFERER; + } else if (!strcasecmp("redirect", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::PERMANENT_REDIRECT"); + return mapping_type::PERMANENT_REDIRECT; + } else if (!strcasecmp("redirect_temporary", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::TEMPORARY_REDIRECT"); + return mapping_type::TEMPORARY_REDIRECT; + } else if (!strcasecmp("map_with_referer", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::FORWARD_MAP_REFERER"); + return mapping_type::FORWARD_MAP_REFERER; + } else if (!strcasecmp("map_with_recv_port", type_str)) { + Dbg(dbg_ctl_url_rewrite, "[BuildTable] - mapping_type::FORWARD_MAP_WITH_RECV_PORT"); + return mapping_type::FORWARD_MAP_WITH_RECV_PORT; + } + return mapping_type::NONE; +} diff --git a/src/proxy/http/remap/unit-tests/CMakeLists.txt b/src/proxy/http/remap/unit-tests/CMakeLists.txt index 4de87e8f08d..cb312c59063 100644 --- a/src/proxy/http/remap/unit-tests/CMakeLists.txt +++ b/src/proxy/http/remap/unit-tests/CMakeLists.txt @@ -288,3 +288,21 @@ target_link_libraries( ) add_catch2_test(NAME test_RemapRules COMMAND $) + +### test_RemapRulesYaml ######################################################################## +add_executable(test_RemapRulesYaml "${PROJECT_SOURCE_DIR}/src/iocore/cache/unit_tests/stub.cc" test_RemapRulesYaml.cc) + +target_link_libraries( + test_RemapRulesYaml + PRIVATE Catch2::Catch2WithMain + ts::http + ts::hdrs # transitive + logging # transitive + ts::http_remap # transitive + ts::proxy + inkdns # transitive + ts::inknet + ts::jsonrpc_protocol +) + +add_catch2_test(NAME test_RemapRulesYaml COMMAND $) diff --git a/src/proxy/http/remap/unit-tests/test_RemapRulesYaml.cc b/src/proxy/http/remap/unit-tests/test_RemapRulesYaml.cc new file mode 100644 index 00000000000..ed98f9dd5c9 --- /dev/null +++ b/src/proxy/http/remap/unit-tests/test_RemapRulesYaml.cc @@ -0,0 +1,256 @@ +/** @file + + Unit tests for a class that deals with remap rules + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + @section details Details + + Implements code necessary for Reverse Proxy which mostly consists of + general purpose hostname substitution in URLs. + + */ + +#include "proxy/hdrs/HdrHeap.h" +#include "proxy/http/remap/RemapYamlConfig.h" +#include "proxy/http/remap/UrlMapping.h" +#include "proxy/http/remap/UrlRewrite.h" +#include "records/RecordsConfig.h" +#include "swoc/swoc_file.h" +#include "ts/apidefs.h" +#include "tscore/BaseLogFile.h" +#include "tsutil/PostScript.h" + +#include +#include + +#include /* catch unit-test framework */ +#include +#include +#include + +struct TestListener : Catch::EventListenerBase { + using EventListenerBase::EventListenerBase; + + void + testRunStarting(Catch::TestRunInfo const & /* testRunInfo ATS_UNUSED */) override + { + Thread *main_thread = new EThread(); + main_thread->set_specific(); + + DiagsPtr::set(new Diags("test_RemapRulesYaml", "*", "", new BaseLogFile("stderr"))); + // diags()->activate_taglist(".*", DiagsTagType_Debug); + // diags()->config.enabled(DiagsTagType_Debug, 1); + diags()->show_location = SHOW_LOCATION_DEBUG; + + url_init(); + mime_init(); + http_init(); + Layout::create(); + RecProcessInit(diags()); + LibRecordsConfigInit(); + } +}; + +CATCH_REGISTER_LISTENER(TestListener); + +swoc::file::path +write_test_remap(const std::string &config, const std::string &tag) +{ + auto tmpdir = swoc::file::temp_directory_path(); + auto path = tmpdir / swoc::file::path(tag + ".yaml"); + + std::ofstream f(path.c_str(), std::ios::trunc); + f.write(config.data(), config.size()); + f.close(); + + return path; +} + +SCENARIO("Parsing ACL named filters", "[proxy][remap]") +{ + GIVEN("Named filter definitions with multiple actions") + { + BUILD_TABLE_INFO bti{}; + ts::PostScript acl_rules_defer([&]() -> void { bti.clear_acl_rules_list(); }); + UrlRewrite rewrite{}; + + bti.rewrite = &rewrite; + + WHEN("filter rule definition has multiple @action") + { + std::string config = R"RMCFG( + remap: + - define_filter: + deny_methods: + action: [deny, allow] + method: [CONNECT, PUT, DELETE] + )RMCFG"; + auto cpath = write_test_remap(config, "test2"); + THEN("The remap parse fails with an error") + { + REQUIRE(remap_parse_yaml_bti(cpath.c_str(), &bti) == false); + } + } + + WHEN("filter rule redefine has multiple @action") + { + std::string config = R"RMCFG( + remap: + - define_filter: + deny_methods: + action: deny + method: CONNECT + deny_methods: + action: allow + method: [PUT, DELETE] + )RMCFG"; + auto cpath = write_test_remap(config, "test2"); + THEN("The rule uses the first action specified") + { + REQUIRE(remap_parse_yaml_bti(cpath.c_str(), &bti) == true); + REQUIRE((bti.rules_list != nullptr && bti.rules_list->next == nullptr)); + REQUIRE((bti.rules_list != nullptr && bti.rules_list->allow_flag == false)); + ; + } + } + } +} + +struct EasyURL { + URL url; + HdrHeap *heap; + + EasyURL(std::string_view s) + { + heap = new_HdrHeap(); + url.create(heap); + url.parse(s); + } + ~EasyURL() { heap->destroy(); } +}; + +SCENARIO("Parsing UrlRewrite", "[proxy][remap]") +{ + GIVEN("A named remap rule without ips") + { + std::unique_ptr urlrw = std::make_unique(); + urlrw->set_remap_yaml(true); + + std::string config = R"RMCFG( + remap: + - define_filter: + deny_methods: + action: deny + method: [CONNECT, PUT, DELETE] + - activate_filter: deny_methods + - type: map + from: + url: https://h1.example.com + to: + url: https://h2.example.com + - deactivate_filter: deny_methods + )RMCFG"; + + auto cpath = write_test_remap(config, "test1"); + printf("wrote config to path: %s\n", cpath.c_str()); + int rc = urlrw->BuildTable(cpath.c_str()); + EasyURL url("https://h1.example.com"); + const char *host = "h1.example.com"; + + THEN("the remap rules has an ip=all") + { + REQUIRE(rc == TS_SUCCESS); + REQUIRE(urlrw->rule_count() == 1); + UrlMappingContainer urlmap; + + REQUIRE(urlrw->forwardMappingLookup(&url.url, 443, host, strlen(host), urlmap)); + REQUIRE(urlmap.getMapping()->filter); + REQUIRE(urlmap.getMapping()->filter->src_ip_cnt == 1); + REQUIRE(urlmap.getMapping()->filter->src_ip_valid); + REQUIRE(urlmap.getMapping()->filter->src_ip_array[0].match_all_addresses); + } + } + GIVEN("map_with_recv_port keyword with a special URL scheme for Unix Domain Socket") + { + std::unique_ptr urlrw = std::make_unique(); + urlrw->set_remap_yaml(true); + + std::string config = R"RMCFG( + remap: + - type: map_with_recv_port + from: + url: http+unix://front.example.com + to: + url: http://origin.example.com + )RMCFG"; + + auto cpath = write_test_remap(config, "unix-scheme"); + printf("wrote config to path: %s\n", cpath.c_str()); + int rc = urlrw->BuildTable(cpath.c_str()); + EasyURL url("http+unix://front.example.com"); + const char *host = "front.example.com"; + + THEN("only requests via unix domain socket matches") + { + // Checck if the rule is loaded + REQUIRE(rc == TS_SUCCESS); + REQUIRE(urlrw->rule_count() == 1); + UrlMappingContainer urlmap; + + // The rule must not match if a port number is available (the request is made on IP interface) + REQUIRE(urlrw->forwardMappingWithRecvPortLookup(&url.url, 80, host, strlen(host), urlmap) == false); + // The rule must match if a port number is unavailable (the request is made on Unix Domain Socket) + REQUIRE(urlrw->forwardMappingWithRecvPortLookup(&url.url, 0, host, strlen(host), urlmap) == true); + } + } + GIVEN("map_with_recv_port keyword with a regular URL scheme") + { + std::unique_ptr urlrw = std::make_unique(); + urlrw->set_remap_yaml(true); + + std::string config = R"RMCFG( + remap: + - type: map_with_recv_port + from: + url: http://front.example.com + to: + url: http://origin.example.com + )RMCFG"; + + auto cpath = write_test_remap(config, "regular-scheme"); + printf("wrote config to path: %s\n", cpath.c_str()); + int rc = urlrw->BuildTable(cpath.c_str()); + EasyURL url("http://front.example.com"); + const char *host = "front.example.com"; + + THEN("only request via IP interface matches") + { + // Checck if the rule is loaded + REQUIRE(rc == TS_SUCCESS); + REQUIRE(urlrw->rule_count() == 1); + UrlMappingContainer urlmap; + + // The rule must match if a port number is available (the request is made on IP interface) + REQUIRE(urlrw->forwardMappingWithRecvPortLookup(&url.url, 80, host, strlen(host), urlmap) == true); + // The rule must not match if a port number is unavailable (the request is made on Unix Domain Socket) + REQUIRE(urlrw->forwardMappingWithRecvPortLookup(&url.url, 0, host, strlen(host), urlmap) == false); + } + } +} diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc index 68bfb893d64..7bc518b215a 100644 --- a/src/records/RecordsConfig.cc +++ b/src/records/RecordsConfig.cc @@ -1096,6 +1096,8 @@ static constexpr RecordElement RecordsConfig[] = , {RECT_CONFIG, "proxy.config.url_remap.filename", RECD_STRING, ts::filename::REMAP, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , + {RECT_CONFIG, "proxy.config.url_remap_yaml.filename", RECD_STRING, ts::filename::REMAP_YAML, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + , {RECT_CONFIG, "proxy.config.url_remap.strategies.filename", RECD_STRING, "strategies.yaml", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , {RECT_CONFIG, "proxy.config.url_remap.remap_required", RECD_INT, "1", RECU_DYNAMIC, RR_NULL, RECC_INT, "[0-1]", RECA_NULL} diff --git a/tests/autest.sh b/tests/autest.sh index 55f02825763..a5979bab75c 100755 --- a/tests/autest.sh +++ b/tests/autest.sh @@ -30,7 +30,7 @@ cd "$SCRIPT_DIR" ./prepare_proxy_verifier.sh || fail "Failed to install Proxy Verifier." export PYTHONPATH=$(pwd):$PYTHONPATH -export PYTHONPATH=$(pwd)/gold_tests/remap:$PYTHONPATH +export PYTHONPATH=$(pwd)/gold_tests/remap:$(pwd)/gold_tests/remap_yaml:$PYTHONPATH ./test-env-check.sh || fail "Failed Python environment checks." hash nc 2>/dev/null || fail "Netcat is not installed." # this is for rhel or centos systems diff --git a/tests/autest.sh.in b/tests/autest.sh.in index 813f3ccfa3f..84ac1ac0286 100755 --- a/tests/autest.sh.in +++ b/tests/autest.sh.in @@ -5,7 +5,7 @@ # export LD_LIBRARY_PATH=${CMAKE_INSTALL_PREFIX}/lib -export PYTHONPATH=${CMAKE_CURRENT_SOURCE_DIR}/gold_tests/remap:${CMAKE_CURRENT_SOURCE_DIR}/gold_tests/lib:$PYTHONPATH +export PYTHONPATH=${CMAKE_CURRENT_SOURCE_DIR}/gold_tests/remap:${CMAKE_CURRENT_SOURCE_DIR}/gold_tests/remap_yaml:${CMAKE_CURRENT_SOURCE_DIR}/gold_tests/lib:$PYTHONPATH # Define tests to skip for CURL_UDS_FLAG if [ -n "${CURL_UDS_FLAG}" ]; then diff --git a/tests/gold_tests/autest-site/trafficserver.test.ext b/tests/gold_tests/autest-site/trafficserver.test.ext index a41f8948ced..e5a212f78d9 100755 --- a/tests/gold_tests/autest-site/trafficserver.test.ext +++ b/tests/gold_tests/autest-site/trafficserver.test.ext @@ -295,6 +295,10 @@ def MakeATSProcess( tmpname = os.path.join(config_dir, fname) p.Disk.File(tmpname, id=make_id(fname), typename="ats:config") + fname = "remap.yaml" + tmpname = os.path.join(config_dir, fname) + p.Disk.File(tmpname, id=make_id(fname), typename="ats:config") + fname = "socks.config" tmpname = os.path.join(config_dir, fname) p.Disk.File(tmpname, id=make_id(fname), typename="ats:config") diff --git a/tests/gold_tests/remap_yaml/all_acl_combinations_yaml.py b/tests/gold_tests/remap_yaml/all_acl_combinations_yaml.py new file mode 100644 index 00000000000..8fe1e1d8296 --- /dev/null +++ b/tests/gold_tests/remap_yaml/all_acl_combinations_yaml.py @@ -0,0 +1,162 @@ +''' +Verify remap.yaml acl behavior. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ALLOW_GET_AND_POST = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: set_allow + methods: [GET, POST] +''' + +ALLOW_GET = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: set_allow + methods: [GET] +''' + +DENY_GET = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: set_deny + methods: [GET] +''' + +DENY_GET_AND_POST = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: set_deny + methods: [GET, POST] +''' + +# yapf: disable +keys = ["index", "policy", "inline", "named_acl", "ip_allow", "GET response", "POST response"] +all_acl_combinations_yaml = [ + [ 0, "legacy", [], [], ALLOW_GET_AND_POST, 200, 200, ], + [ 1, "legacy", [], [], ALLOW_GET, 200, 403, ], + [ 2, "legacy", [], [], DENY_GET, 403, 200, ], + [ 3, "legacy", [], [], DENY_GET_AND_POST, 403, 403, ], + [ 4, "legacy", [], ['action: allow', 'method: GET'], ALLOW_GET_AND_POST, 200, 403, ], + [ 5, "legacy", [], ['action: allow', 'method: GET'], ALLOW_GET, 200, 403, ], + [ 6, "legacy", [], ['action: allow', 'method: GET'], DENY_GET, 403, 403, ], + [ 7, "legacy", [], ['action: allow', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 8, "legacy", [], ['action: deny', 'method: GET'], ALLOW_GET_AND_POST, 403, 200, ], + [ 9, "legacy", [], ['action: deny', 'method: GET'], ALLOW_GET, 403, 403, ], + [ 10, "legacy", [], ['action: deny', 'method: GET'], DENY_GET, 403, 200, ], + [ 11, "legacy", [], ['action: deny', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 12, "legacy", ['action: allow', 'method: GET'], [], ALLOW_GET_AND_POST, 200, 403, ], + [ 13, "legacy", ['action: allow', 'method: GET'], [], ALLOW_GET, 200, 403, ], + [ 14, "legacy", ['action: allow', 'method: GET'], [], DENY_GET, 403, 403, ], + [ 15, "legacy", ['action: allow', 'method: GET'], [], DENY_GET_AND_POST, 403, 403, ], + [ 16, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: GET'], ALLOW_GET_AND_POST, 200, 403, ], + [ 17, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: GET'], ALLOW_GET, 200, 403, ], + [ 18, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: GET'], DENY_GET, 403, 403, ], + [ 19, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 20, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: GET'], ALLOW_GET_AND_POST, 403, 403, ], + [ 21, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: GET'], ALLOW_GET, 403, 403, ], + [ 22, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: GET'], DENY_GET, 403, 403, ], + [ 23, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 24, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: POST'], ALLOW_GET_AND_POST, 403, 403, ], + [ 25, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: POST'], ALLOW_GET, 403, 403, ], + [ 26, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: POST'], DENY_GET, 403, 403, ], + [ 27, "legacy", ['action: allow', 'method: GET'], ['action: allow', 'method: POST'], DENY_GET_AND_POST, 403, 403, ], + [ 28, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: POST'], ALLOW_GET_AND_POST, 200, 403, ], + [ 29, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: POST'], ALLOW_GET, 200, 403, ], + [ 30, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: POST'], DENY_GET, 403, 403, ], + [ 31, "legacy", ['action: allow', 'method: GET'], ['action: deny', 'method: POST'], DENY_GET_AND_POST, 403, 403, ], + [ 32, "legacy", ['action: deny', 'method: GET'], [], ALLOW_GET_AND_POST, 403, 200, ], + [ 33, "legacy", ['action: deny', 'method: GET'], [], ALLOW_GET, 403, 403, ], + [ 34, "legacy", ['action: deny', 'method: GET'], [], DENY_GET, 403, 200, ], + [ 35, "legacy", ['action: deny', 'method: GET'], [], DENY_GET_AND_POST, 403, 403, ], + [ 36, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: GET'], ALLOW_GET_AND_POST, 403, 403, ], + [ 37, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: GET'], ALLOW_GET, 403, 403, ], + [ 38, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: GET'], DENY_GET, 403, 403, ], + [ 39, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 40, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: GET'], ALLOW_GET_AND_POST, 403, 200, ], + [ 41, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: GET'], ALLOW_GET, 403, 403, ], + [ 42, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: GET'], DENY_GET, 403, 200, ], + [ 43, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: GET'], DENY_GET_AND_POST, 403, 403, ], + [ 44, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: POST'], ALLOW_GET_AND_POST, 403, 200, ], + [ 45, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: POST'], ALLOW_GET, 403, 403, ], + [ 46, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: POST'], DENY_GET, 403, 200, ], + [ 47, "legacy", ['action: deny', 'method: GET'], ['action: allow', 'method: POST'], DENY_GET_AND_POST, 403, 403, ], + [ 48, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: POST'], ALLOW_GET_AND_POST, 403, 403, ], + [ 49, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: POST'], ALLOW_GET, 403, 403, ], + [ 50, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: POST'], DENY_GET, 403, 403, ], + [ 51, "legacy", ['action: deny', 'method: GET'], ['action: deny', 'method: POST'], DENY_GET_AND_POST, 403, 403, ], + [ 52, "modern", [], [], ALLOW_GET_AND_POST, 200, 200, ], + [ 53, "modern", [], [], ALLOW_GET, 200, 403, ], + [ 54, "modern", [], [], DENY_GET, 403, 200, ], + [ 55, "modern", [], [], DENY_GET_AND_POST, 403, 403, ], + [ 56, "modern", [], ['action: set_allow', 'method: GET'], ALLOW_GET_AND_POST, 200, 403, ], + [ 57, "modern", [], ['action: set_allow', 'method: GET'], ALLOW_GET, 200, 403, ], + [ 58, "modern", [], ['action: set_allow', 'method: GET'], DENY_GET, 200, 403, ], + [ 59, "modern", [], ['action: set_allow', 'method: GET'], DENY_GET_AND_POST, 200, 403, ], + [ 60, "modern", [], ['action: set_deny', 'method: GET'], ALLOW_GET_AND_POST, 403, 200, ], + [ 61, "modern", [], ['action: set_deny', 'method: GET'], ALLOW_GET, 403, 200, ], + [ 62, "modern", [], ['action: set_deny', 'method: GET'], DENY_GET, 403, 200, ], + [ 63, "modern", [], ['action: set_deny', 'method: GET'], DENY_GET_AND_POST, 403, 200, ], + [ 64, "modern", ['action: set_allow', 'method: GET'], [], ALLOW_GET_AND_POST, 200, 403, ], + [ 65, "modern", ['action: set_allow', 'method: GET'], [], ALLOW_GET, 200, 403, ], + [ 66, "modern", ['action: set_allow', 'method: GET'], [], DENY_GET, 200, 403, ], + [ 67, "modern", ['action: set_allow', 'method: GET'], [], DENY_GET_AND_POST, 200, 403, ], + [ 68, "modern", ['action: set_allow', 'method: GET'], ['action: set_allow', 'method: GET'], ALLOW_GET_AND_POST, 200, 403, ], + [ 69, "modern", ['action: set_allow', 'method: GET'], ['action: set_allow', 'method: GET'], ALLOW_GET, 200, 403, ], + [ 70, "modern", ['action: set_allow', 'method: GET'], ['action: set_allow', 'method: GET'], DENY_GET, 200, 403, ], + [ 71, "modern", ['action: set_allow', 'method: GET'], ['action: set_allow', 'method: GET'], DENY_GET_AND_POST, 200, 403, ], + [ 72, "modern", ['action: set_allow', 'method: GET'], ['action: set_deny', 'method: GET'], ALLOW_GET_AND_POST, 200, 403, ], + [ 73, "modern", ['action: set_allow', 'method: GET'], ['action: set_deny', 'method: GET'], ALLOW_GET, 200, 403, ], + [ 74, "modern", ['action: set_allow', 'method: GET'], ['action: set_deny', 'method: GET'], DENY_GET, 200, 403, ], + [ 75, "modern", ['action: set_allow', 'method: GET'], ['action: set_deny', 'method: GET'], DENY_GET_AND_POST, 200, 403, ], + [ 76, "modern", ['action: set_deny', 'method: GET'], [], ALLOW_GET_AND_POST, 403, 200, ], + [ 77, "modern", ['action: set_deny', 'method: GET'], [], ALLOW_GET, 403, 200, ], + [ 78, "modern", ['action: set_deny', 'method: GET'], [], DENY_GET, 403, 200, ], + [ 79, "modern", ['action: set_deny', 'method: GET'], [], DENY_GET_AND_POST, 403, 200, ], + [ 80, "modern", ['action: set_deny', 'method: GET'], ['action: set_allow', 'method: GET'], ALLOW_GET_AND_POST, 403, 200, ], + [ 81, "modern", ['action: set_deny', 'method: GET'], ['action: set_allow', 'method: GET'], ALLOW_GET, 403, 200, ], + [ 82, "modern", ['action: set_deny', 'method: GET'], ['action: set_allow', 'method: GET'], DENY_GET, 403, 200, ], + [ 83, "modern", ['action: set_deny', 'method: GET'], ['action: set_allow', 'method: GET'], DENY_GET_AND_POST, 403, 200, ], + [ 84, "modern", ['action: set_deny', 'method: GET'], ['action: set_deny', 'method: GET'], ALLOW_GET_AND_POST, 403, 200, ], + [ 85, "modern", ['action: set_deny', 'method: GET'], ['action: set_deny', 'method: GET'], ALLOW_GET, 403, 200, ], + [ 86, "modern", ['action: set_deny', 'method: GET'], ['action: set_deny', 'method: GET'], DENY_GET, 403, 200, ], + [ 87, "modern", ['action: set_deny', 'method: GET'], ['action: set_deny', 'method: GET'], DENY_GET_AND_POST, 403, 200, ], + [ 88, "legacy", ['action: allow', 'src_ip: 127.0.0.1'], [], ALLOW_GET_AND_POST, 200, 200, ], + [ 89, "legacy", ['action: allow', 'src_ip: 127.0.0.1'], [], ALLOW_GET, 200, 403, ], + [ 90, "legacy", ['action: allow', 'src_ip: 127.0.0.1'], [], DENY_GET, 403, 200, ], + [ 91, "legacy", ['action: allow', 'src_ip: 127.0.0.1'], [], DENY_GET_AND_POST, 403, 403, ], + [ 92, "legacy", ['action: deny', 'src_ip: 127.0.0.1'], [], ALLOW_GET_AND_POST, 403, 403, ], + [ 93, "legacy", ['action: deny', 'src_ip: 127.0.0.1'], [], ALLOW_GET, 403, 403, ], + [ 94, "legacy", ['action: deny', 'src_ip: 127.0.0.1'], [], DENY_GET, 403, 403, ], + [ 95, "legacy", ['action: deny', 'src_ip: 127.0.0.1'], [], DENY_GET_AND_POST, 403, 403, ], + [ 96, "legacy", ['action: allow', 'src_ip: 192.0.2.1/24'], [], ALLOW_GET_AND_POST, 403, 403, ], + [ 97, "legacy", ['action: allow', 'src_ip: 192.0.2.1/24'], [], ALLOW_GET, 403, 403, ], + [ 98, "legacy", ['action: allow', 'src_ip: 192.0.2.0/24'], [], DENY_GET, 403, 403, ], + [ 99, "legacy", ['action: allow', 'src_ip: 192.0.2.0/24'], [], DENY_GET_AND_POST, 403, 403, ], + [100, "legacy", ['action: deny', 'src_ip: 192.0.2.1/24'], [], ALLOW_GET_AND_POST, 200, 200, ], + [101, "legacy", ['action: deny', 'src_ip: 192.0.2.1/24'], [], ALLOW_GET, 200, 403, ], + [102, "legacy", ['action: deny', 'src_ip: 192.0.2.0/24'], [], DENY_GET, 403, 200, ], + [103, "legacy", ['action: deny', 'src_ip: 192.0.2.0/24'], [], DENY_GET_AND_POST, 403, 403, ], +] +# yapf: enable + +all_acl_combination_tests_yaml = [dict(zip(keys, test)) for test in all_acl_combinations_yaml] diff --git a/tests/gold_tests/remap_yaml/base.replay.yaml b/tests/gold_tests/remap_yaml/base.replay.yaml new file mode 100644 index 00000000000..b236481ffe4 --- /dev/null +++ b/tests/gold_tests/remap_yaml/base.replay.yaml @@ -0,0 +1,67 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that denies HEAD and POST, but allows all other +# methods. + +meta: + version: '1.0' + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: GET + version: '1.1' + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + proxy-response: + status: 200 + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + + - client-request: + method: POST + version: '1.1' + url: /test/ip_allow/test_post + headers: + fields: + - [ Content-Length, 10 ] + - [ uuid, post ] + - [ X-Request, post ] + + proxy-response: + status: 200 + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] diff --git a/tests/gold_tests/remap_yaml/basic_conf_remap_yaml_yaml.test.py b/tests/gold_tests/remap_yaml/basic_conf_remap_yaml_yaml.test.py new file mode 100644 index 00000000000..5cff98416c1 --- /dev/null +++ b/tests/gold_tests/remap_yaml/basic_conf_remap_yaml_yaml.test.py @@ -0,0 +1,203 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test conf_remap using a yaml file. +''' + +Test.ContinueOnFail = True + + +class conf_remap_yaml_load_test: + """Test conf_remap using a yaml file.""" + + client_counter: int = 0 + ts_counter: int = 0 + server_counter: int = 0 + + def __init__(self, name: str, gold_file="", remap_filename="", remap_content=""): + """Initialize the test. + :param name: The name of the test. + :param gold_file: Gold file to be checked. + :param remap_filename: Remap yaml filename. + :param remap_content: remap yaml file content. + """ + self.name = name + self.gold_file = gold_file + self._remap_filename = remap_filename + self._remap_content = remap_content + + def _configure_server(self, tr: 'TestRun'): + """Configure the server. + + :param tr: The TestRun object to associate the server process with. + """ + server = Test.MakeOriginServer(f"server-{conf_remap_yaml_load_test.ts_counter}", lookup_key="{%Host}{PATH}") + request_header2 = { + "headers": "GET /test HTTP/1.1\r\nHost: www.testexample.com\r\n\r\n", + "timestamp": "1469733493.993", + "body": "" + } + response_header2 = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} + + server.addResponse("sessionfile.log", request_header2, response_header2) + conf_remap_yaml_load_test.server_counter += 1 + self._server = server + + def _configure_traffic_server(self, tr: 'TestRun'): + """Configure Traffic Server. + + :param tr: The TestRun object to associate the ts process with. + """ + ts = Test.MakeATSProcess(f"ts-{conf_remap_yaml_load_test.ts_counter}") + + conf_remap_yaml_load_test.ts_counter += 1 + ts.Disk.records_config.update( + ''' + diags: + debug: + enabled: 1 + tags: conf_remap + dns: + resolv_conf: NULL + http: + referer_filter: 1 + url_remap: + pristine_host_hdr: 0 # make sure is 0 + + ''') + self._ts = ts + + def run(self, diags_fail_exp="", ts_retcode=0): + """Run the test. + :param diags_fail_exp: Text to be included to validate the error. + :param ts_retcode: Expected return code from TS. + """ + tr = Test.AddTestRun(self.name) + self._configure_server(tr) + self._configure_traffic_server(tr) + + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + + self._ts.ReturnCode = ts_retcode + + if ts_retcode > 0: # we could have errors logged and yet, we still want to move on. + self._ts.Ready = 0 + + if diags_fail_exp != "": + # some error logs will be written to the diags. + self._ts.Disk.diags_log.Content = Testers.IncludesExpression(diags_fail_exp, "Have a look.") + else: + tr.Processes.Default.ReturnCode = 0 + + if self.gold_file: + tr.Processes.Default.Streams.stderr = self.gold_file + + if self._remap_filename != "" and self._remap_content != "": + self._ts.Disk.MakeConfigFile(self._remap_filename).update(self._remap_content) + self._ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://www.testexample.com/ + to: + url: http://127.0.0.1:{self._server.Variables.Port} + plugins: + - name: conf_remap.so + params: + - {self._remap_filename} + '''.split("\n")) + if Condition.CurlUsingUnixDomainSocket(): + tr.MakeCurlCommand( + '-H "Host: www.testexample.com" "http://127.0.0.1:{0}/test" --verbose'.format(self._ts.Variables.port), ts=self._ts) + else: + tr.MakeCurlCommand( + '--proxy 127.0.0.1:{0} "http://www.testexample.com/test" -H "Host: www.testexample.com" --verbose'.format( + self._ts.Variables.port), + ts=self._ts) + conf_remap_yaml_load_test.client_counter += 1 + + +gold_file = "gold/200OK_test.gold" +if Condition.CurlUsingUnixDomainSocket(): + gold_file = "gold/200OK_test_uds.gold" + +test0 = conf_remap_yaml_load_test( + "Test success", + gold_file=gold_file, + remap_filename="testexample_remap.yaml", + remap_content=''' + records: + url_remap: + pristine_host_hdr: 1 + ''') +test0.run() + +test1 = conf_remap_yaml_load_test( + "Test mismatch type", + remap_filename="mismatch_field_type_remap.yaml", + remap_content=''' + records: + url_remap: + pristine_host_hdr: !!float '1' + ''') +test1.run(diags_fail_exp="'proxy.config.url_remap.pristine_host_hdr' variable type mismatch", ts_retcode=33) + +test2 = conf_remap_yaml_load_test( + "Test invalid variable", + remap_filename="invalid1_field_type_remap.yaml", + remap_content=''' + records: + plugin: + dynamic_reload_mode: 1 + ''') + +test2.run( + diags_fail_exp="'proxy.config.plugin.dynamic_reload_mode' is not a configuration variable or cannot be overridden", + ts_retcode=33) + +# We let the conf_remap parse two fields, only one is valid, we expect ATS to start and the invalid fields ignored. +test3 = conf_remap_yaml_load_test( + "Test success", + gold_file=gold_file, + remap_filename="testexample2_remap.yaml", + remap_content=''' + records: + plugin: + dynamic_reload_mode: 1 + + url_remap: + pristine_host_hdr: 1 + ''') +test3.run(diags_fail_exp="'proxy.config.plugin.dynamic_reload_mode' is not a configuration variable or cannot be overridden") + +# Check null values +test4 = conf_remap_yaml_load_test( + "Test success - with NULL variable", + gold_file=gold_file, + remap_filename="testexample_remap.yaml", + remap_content=''' + records: + url_remap: + pristine_host_hdr: 1 + hostdb: + ip_resolve: "NULL" # We want to make sure this gets read as it should. "NULL" could be the value of this field. + ''') +test4.run() diff --git a/tests/gold_tests/remap_yaml/conf_remap_float_yaml.test.py b/tests/gold_tests/remap_yaml/conf_remap_float_yaml.test.py new file mode 100644 index 00000000000..bf0edbc3108 --- /dev/null +++ b/tests/gold_tests/remap_yaml/conf_remap_float_yaml.test.py @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test command: traffic_ctl config describe proxy.config.http.background_fill_completed_threshold (YTSATS-3309) +''' +Test.testName = 'Float in conf_remap Config Test' + +ts = Test.MakeATSProcess("ts") + +ts.Disk.MakeConfigFile('conf_remap.yaml').update(''' +records: + http: + background_fill_completed_threshold: !!float '0.5' +''') + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://cdn.example.com/ + to: + url: http://origin.example.com/ + plugins: + - name: conf_remap.so + params: + - {Test.RunDirectory}/ts/config/conf_remap.yaml + '''.split("\n")) + +tr = Test.AddTestRun("traffic_ctl command") +tr.Env = ts.Env +tr.TimeOut = 5 +tr.StillRunningAfter = ts + +p = tr.Processes.Default +p.Command = f"traffic_ctl config describe proxy.config.http.background_fill_completed_threshold" +p.ReturnCode = 0 +p.StartBefore(Test.Processes.ts) diff --git a/tests/gold_tests/remap_yaml/deactivate_ip_allow_yaml.py b/tests/gold_tests/remap_yaml/deactivate_ip_allow_yaml.py new file mode 100644 index 00000000000..5c710abc568 --- /dev/null +++ b/tests/gold_tests/remap_yaml/deactivate_ip_allow_yaml.py @@ -0,0 +1,126 @@ +''' +Verify remap.yaml acl behavior. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ALLOW_GET_AND_POST = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: allow + methods: [GET, POST] +''' + +ALLOW_GET = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: allow + methods: [GET] +''' + +DENY_GET = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: deny + methods: [GET] +''' + +DENY_GET_AND_POST = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: deny + methods: [GET, POST] +''' + +# Optimized ACL filter on accept +DENY_ALL = f''' +ip_allow: + - apply: in + ip_addrs: [0/0, ::/0] + action: deny + methods: ALL +''' + +# yapf: disable +keys = ["index", "policy", "inline", "named_acl", "deactivate_ip_allow", "ip_allow", "GET response", "POST response"] +deactivate_ip_allow_combinations_yaml = [ + [ 0, "legacy", [], [], False, ALLOW_GET_AND_POST, 200, 200, ], + [ 1, "legacy", [], [], False, ALLOW_GET, 200, 403, ], + [ 2, "legacy", [], [], False, DENY_GET, 403, 200, ], + [ 3, "legacy", [], [], False, DENY_GET_AND_POST, 403, 403, ], + [ 4, "legacy", [], [], False, DENY_ALL, None, None, ], + [ 5, "legacy", [], [], True, ALLOW_GET_AND_POST, 200, 200, ], + [ 6, "legacy", [], [], True, ALLOW_GET, 200, 200, ], + [ 7, "legacy", [], [], True, DENY_GET, 200, 200, ], + [ 8, "legacy", [], [], True, DENY_GET_AND_POST, 200, 200, ], + [ 9, "legacy", [], [], True, DENY_ALL, 200, 200, ], + [ 10, "legacy", ["action: allow", "method: GET"], [], False, ALLOW_GET_AND_POST, 200, 403, ], + [ 11, "legacy", ["action: allow", "method: GET"], [], False, ALLOW_GET, 200, 403, ], + [ 12, "legacy", ["action: allow", "method: GET"], [], False, DENY_GET, 403, 403, ], + [ 13, "legacy", ["action: allow", "method: GET"], [], False, DENY_GET_AND_POST, 403, 403, ], + [ 14, "legacy", ["action: allow", "method: GET"], [], False, DENY_ALL, None, None, ], + [ 15, "legacy", ["action: allow", "method: GET"], [], True, ALLOW_GET_AND_POST, 200, 403, ], + [ 16, "legacy", ["action: allow", "method: GET"], [], True, ALLOW_GET, 200, 403, ], + [ 17, "legacy", ["action: allow", "method: GET"], [], True, DENY_GET, 200, 403, ], + [ 18, "legacy", ["action: allow", "method: GET"], [], True, DENY_GET_AND_POST, 200, 403, ], + [ 19, "legacy", ["action: allow", "method: GET"], [], True, DENY_ALL, 200, 403, ], + [ 20, "legacy", ["action: deny", "method: GET"], [], False, ALLOW_GET_AND_POST, 403, 200, ], + [ 21, "legacy", ["action: deny", "method: GET"], [], False, ALLOW_GET, 403, 403, ], + [ 22, "legacy", ["action: deny", "method: GET"], [], False, DENY_GET, 403, 200, ], + [ 23, "legacy", ["action: deny", "method: GET"], [], False, DENY_GET_AND_POST, 403, 403, ], + [ 24, "legacy", ["action: deny", "method: GET"], [], False, DENY_ALL, None, None, ], + [ 25, "legacy", ["action: deny", "method: GET"], [], True, ALLOW_GET_AND_POST, 403, 200, ], + [ 26, "legacy", ["action: deny", "method: GET"], [], True, ALLOW_GET, 403, 200, ], + [ 27, "legacy", ["action: deny", "method: GET"], [], True, DENY_GET, 403, 200, ], + [ 28, "legacy", ["action: deny", "method: GET"], [], True, DENY_GET_AND_POST, 403, 200, ], + [ 29, "legacy", ["action: deny", "method: GET"], [], True, DENY_ALL, 403, 200, ], + [ 30, "legacy", ["action: allow", "src_ip: 127.0.0.1"], [], False, DENY_ALL, None, None, ], + [ 31, "legacy", ["action: allow", "src_ip: 127.0.0.1"], [], True, DENY_ALL, 200, 200, ], + [ 32, "legacy", ["action: deny", "src_ip: 127.0.0.1"], [], False, DENY_ALL, None, None, ], + [ 33, "legacy", ["action: deny", "src_ip: 127.0.0.1"], [], True, DENY_ALL, 403, 403, ], + [ 34, "legacy", ["action: allow", "src_ip: 192.0.2.1/24"], [], False, DENY_ALL, None, None, ], + [ 35, "legacy", ["action: allow", "src_ip: 192.0.2.1/24"], [], True, DENY_ALL, 403, 403, ], + [ 36, "legacy", ["action: deny", "src_ip: 192.0.2.0/24"], [], False, DENY_ALL, None, None, ], + [ 37, "legacy", ["action: deny", "src_ip: 192.0.2.0/24"], [], True, DENY_ALL, 200, 200, ], + + # Verify in legacy mode that add_allow acts just like allow, and add_deny acts just like deny. + [ 38, "legacy", ["action: add_allow", "method: GET"], [], False, ALLOW_GET_AND_POST, 200, 403, ], + [ 39, "legacy", ["action: add_allow", "method: GET"], [], False, ALLOW_GET, 200, 403, ], + [ 40, "legacy", ["action: add_allow", "method: GET"], [], False, DENY_GET, 403, 403, ], + [ 41, "legacy", ["action: add_allow", "method: GET"], [], False, DENY_GET_AND_POST, 403, 403, ], + [ 42, "legacy", ["action: add_allow", "method: GET"], [], False, DENY_ALL, None, None, ], + [ 43, "legacy", ["action: add_allow", "method: GET"], [], True, ALLOW_GET_AND_POST, 200, 403, ], + [ 44, "legacy", ["action: add_allow", "method: GET"], [], True, ALLOW_GET, 200, 403, ], + [ 45, "legacy", ["action: add_allow", "method: GET"], [], True, DENY_GET, 200, 403, ], + [ 46, "legacy", ["action: add_allow", "method: GET"], [], True, DENY_GET_AND_POST, 200, 403, ], + [ 47, "legacy", ["action: add_allow", "method: GET"], [], True, DENY_ALL, 200, 403, ], + [ 48, "legacy", ["action: add_deny", "method: GET"], [], False, ALLOW_GET_AND_POST, 403, 200, ], + [ 49, "legacy", ["action: add_deny", "method: GET"], [], False, ALLOW_GET, 403, 403, ], + [ 50, "legacy", ["action: add_deny", "method: GET"], [], False, DENY_GET, 403, 200, ], + [ 51, "legacy", ["action: add_deny", "method: GET"], [], False, DENY_GET_AND_POST, 403, 403, ], + [ 52, "legacy", ["action: add_deny", "method: GET"], [], False, DENY_ALL, None, None, ], + [ 53, "legacy", ["action: add_deny", "method: GET"], [], True, ALLOW_GET_AND_POST, 403, 200, ], + [ 54, "legacy", ["action: add_deny", "method: GET"], [], True, ALLOW_GET, 403, 200, ], + [ 55, "legacy", ["action: add_deny", "method: GET"], [], True, DENY_GET, 403, 200, ], + [ 56, "legacy", ["action: add_deny", "method: GET"], [], True, DENY_GET_AND_POST, 403, 200, ], + [ 57, "legacy", ["action: add_deny", "method: GET"], [], True, DENY_ALL, 403, 200, ], +] +all_deactivate_ip_allow_tests_yaml = [dict(zip(keys, test)) for test in deactivate_ip_allow_combinations_yaml] +# yapf: enable diff --git a/tests/gold_tests/remap_yaml/deny_head_post.replay.yaml b/tests/gold_tests/remap_yaml/deny_head_post.replay.yaml new file mode 100644 index 00000000000..f814b534779 --- /dev/null +++ b/tests/gold_tests/remap_yaml/deny_head_post.replay.yaml @@ -0,0 +1,98 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that denies HEAD and POST, but allows all other +# methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "HEAD" + version: "1.1" + url: /test/ip_allow/test_head + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, head ] + - [ X-Request, head ] + + <<: *standard_response + + proxy-response: + status: 403 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 403 + + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + proxy-response: + status: 200 diff --git a/tests/gold_tests/remap_yaml/gold/200OK_test.gold b/tests/gold_tests/remap_yaml/gold/200OK_test.gold new file mode 100644 index 00000000000..887d5fb3ac0 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/200OK_test.gold @@ -0,0 +1,13 @@ +`` +> GET http://www.testexample.com/test`` +> Host: www.testexample.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +`` diff --git a/tests/gold_tests/remap_yaml/gold/200OK_test_uds.gold b/tests/gold_tests/remap_yaml/gold/200OK_test_uds.gold new file mode 100644 index 00000000000..5d588e039e5 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/200OK_test_uds.gold @@ -0,0 +1,13 @@ +`` +> GET /test`` +> Host: www.testexample.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Connection: keep-alive +< Server: ATS/`` +`` diff --git a/tests/gold_tests/remap_yaml/gold/lookupTest.gold b/tests/gold_tests/remap_yaml/gold/lookupTest.gold new file mode 100644 index 00000000000..e6f1707580e --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/lookupTest.gold @@ -0,0 +1,14 @@ +`` +> GET ``/test`` +> Host: www.testexample.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/map-with-recv-port-ip.gold b/tests/gold_tests/remap_yaml/gold/map-with-recv-port-ip.gold new file mode 100644 index 00000000000..b7949ccc68c --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/map-with-recv-port-ip.gold @@ -0,0 +1,13 @@ +`` +> GET /`` +> Host: test.example.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Content-Length: 2 +< Date: `` +< Age: `` +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/map-with-recv-port-unix.gold b/tests/gold_tests/remap_yaml/gold/map-with-recv-port-unix.gold new file mode 100644 index 00000000000..a83cd71adee --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/map-with-recv-port-unix.gold @@ -0,0 +1,13 @@ +`` +> GET /`` +> Host: test.example.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Content-Length: 4 +< Date: `` +< Age: `` +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-200.gold b/tests/gold_tests/remap_yaml/gold/remap-200.gold new file mode 100644 index 00000000000..d44a26ad06c --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-200.gold @@ -0,0 +1,14 @@ +`` +> GET `` +> Host: www.example.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-404.gold b/tests/gold_tests/remap_yaml/gold/remap-404.gold new file mode 100644 index 00000000000..133518e2d02 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-404.gold @@ -0,0 +1,12 @@ +`` +> GET `` HTTP/1.1 +> Host: `` +> User-Agent: curl/`` +`` +< HTTP/1.1 404 Not Found`` +< Date: `` +< Proxy-Connection: keep-alive +< Server: ATS/`` +`` +< Content-Type: text/html +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-DNS-200.gold b/tests/gold_tests/remap_yaml/gold/remap-DNS-200.gold new file mode 100644 index 00000000000..9fafc8ee1d6 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-DNS-200.gold @@ -0,0 +1,14 @@ +`` +> GET `` +> Host: testDNS.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-DNS-ipv6-200.gold b/tests/gold_tests/remap_yaml/gold/remap-DNS-ipv6-200.gold new file mode 100644 index 00000000000..f926c5b93fb --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-DNS-ipv6-200.gold @@ -0,0 +1,14 @@ +`` +> GET `` +> Host: testDNS2.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-hitATS-404.gold b/tests/gold_tests/remap_yaml/gold/remap-hitATS-404.gold new file mode 100644 index 00000000000..67a81bc4f9a --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-hitATS-404.gold @@ -0,0 +1,11 @@ +`` +> GET / HTTP/1.1 +> Host: `` +> User-Agent: curl/`` +`` +< HTTP/1.1 404 Not Found on Accelerator +< Date: `` +< Connection: `` +< Server: ATS/`` +< Content-Type: text/html +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-https-200.gold b/tests/gold_tests/remap_yaml/gold/remap-https-200.gold new file mode 100644 index 00000000000..9cd42fc43a0 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-https-200.gold @@ -0,0 +1,13 @@ +`` +> GET / HTTP/1.1 +> Host: www.example.com`` +> User-Agent: curl/`` +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-https-200_2.gold b/tests/gold_tests/remap_yaml/gold/remap-https-200_2.gold new file mode 100644 index 00000000000..4bb24ea4e34 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-https-200_2.gold @@ -0,0 +1,13 @@ +`` +> GET / HTTP/1.1 +> Host: www.anotherexample.com`` +> User-Agent: curl/`` +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-https-200_3.gold b/tests/gold_tests/remap_yaml/gold/remap-https-200_3.gold new file mode 100644 index 00000000000..f5a6798af88 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-https-200_3.gold @@ -0,0 +1,13 @@ +`` +> GET / HTTP/1.1 +> Host: www.example3.com`` +> User-Agent: curl/`` +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-ip-resolve.gold b/tests/gold_tests/remap_yaml/gold/remap-ip-resolve.gold new file mode 100644 index 00000000000..c83df3f4ca3 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-ip-resolve.gold @@ -0,0 +1,14 @@ +`` +> GET http://testDNS.com/`` +> Host: testDNS.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Server: ATS/`` +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-redirect.gold b/tests/gold_tests/remap_yaml/gold/remap-redirect.gold new file mode 100644 index 00000000000..27203b00d45 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-redirect.gold @@ -0,0 +1,15 @@ +`` +> GET `` +> Host: test3.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 301 Redirect +< Date: `` +< Proxy-Connection: `` +< Server: ATS/`` +< Cache-Control: `` +< Location: http://httpbin.org/ +`` +< Content-Length: `` +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-referer-hit.gold b/tests/gold_tests/remap_yaml/gold/remap-referer-hit.gold new file mode 100644 index 00000000000..17e39666026 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-referer-hit.gold @@ -0,0 +1,14 @@ +`` +> GET `` +> Host: test4.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-referer-miss.gold b/tests/gold_tests/remap_yaml/gold/remap-referer-miss.gold new file mode 100644 index 00000000000..0a99becbda1 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-referer-miss.gold @@ -0,0 +1,15 @@ +`` +> GET `` +> Host: test4.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 302 Redirect +< Date: `` +< Proxy-Connection: `` +< Server: ATS/`` +< Cache-Control: `` +< Location: http://httpbin.org +`` +< Content-Length: `` +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-ws-metrics-uds.gold b/tests/gold_tests/remap_yaml/gold/remap-ws-metrics-uds.gold new file mode 100644 index 00000000000..d88a74be335 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-ws-metrics-uds.gold @@ -0,0 +1,21 @@ +proxy.process.http.total_incoming_connections 2 +proxy.process.http.total_client_connections 2 +proxy.process.http.total_client_connections_ipv4 0 +proxy.process.http.total_client_connections_ipv6 0 +proxy.process.http.total_server_connections 1 +proxy.process.http2.total_client_connections 0 +proxy.process.http.connect_requests 0 +proxy.process.tunnel.total_client_connections_blind_tcp 1 +proxy.process.tunnel.current_client_connections_blind_tcp 0 +proxy.process.tunnel.total_server_connections_blind_tcp 1 +proxy.process.tunnel.current_server_connections_blind_tcp 0 +proxy.process.tunnel.total_client_connections_tls_tunnel 0 +proxy.process.tunnel.current_client_connections_tls_tunnel 0 +proxy.process.tunnel.total_client_connections_tls_forward 0 +proxy.process.tunnel.current_client_connections_tls_forward 0 +proxy.process.tunnel.total_client_connections_tls_partial_blind 0 +proxy.process.tunnel.current_client_connections_tls_partial_blind 0 +proxy.process.tunnel.total_client_connections_tls_http 0 +proxy.process.tunnel.current_client_connections_tls_http 0 +proxy.process.tunnel.total_server_connections_tls 0 +proxy.process.tunnel.current_server_connections_tls 0 diff --git a/tests/gold_tests/remap_yaml/gold/remap-ws-metrics.gold b/tests/gold_tests/remap_yaml/gold/remap-ws-metrics.gold new file mode 100644 index 00000000000..cf39c20a91c --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-ws-metrics.gold @@ -0,0 +1,21 @@ +proxy.process.http.total_incoming_connections 3 +proxy.process.http.total_client_connections 3 +proxy.process.http.total_client_connections_ipv4 3 +proxy.process.http.total_client_connections_ipv6 0 +proxy.process.http.total_server_connections 2 +proxy.process.http2.total_client_connections 0 +proxy.process.http.connect_requests 0 +proxy.process.tunnel.total_client_connections_blind_tcp 1 +proxy.process.tunnel.current_client_connections_blind_tcp 0 +proxy.process.tunnel.total_server_connections_blind_tcp 2 +proxy.process.tunnel.current_server_connections_blind_tcp 0 +proxy.process.tunnel.total_client_connections_tls_tunnel 0 +proxy.process.tunnel.current_client_connections_tls_tunnel 0 +proxy.process.tunnel.total_client_connections_tls_forward 0 +proxy.process.tunnel.current_client_connections_tls_forward 0 +proxy.process.tunnel.total_client_connections_tls_partial_blind 0 +proxy.process.tunnel.current_client_connections_tls_partial_blind 0 +proxy.process.tunnel.total_client_connections_tls_http 1 +proxy.process.tunnel.current_client_connections_tls_http 0 +proxy.process.tunnel.total_server_connections_tls 0 +proxy.process.tunnel.current_server_connections_tls 0 diff --git a/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade-400.gold b/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade-400.gold new file mode 100644 index 00000000000..a46a2490de7 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade-400.gold @@ -0,0 +1,7 @@ +`` +> GET /chat HTTP/1.1 +> Host: `` +> User-Agent: curl/`` +`` +< HTTP/1.1 400 Invalid Upgrade Request +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade.gold b/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade.gold new file mode 100644 index 00000000000..317fee1f53c --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-ws-upgrade.gold @@ -0,0 +1,11 @@ +`` +> GET /chat HTTP/1.1 +> Host: `` +> User-Agent: curl/`` +`` +< HTTP/1.1 101 Switching Protocols +< Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= +< Date: `` +< Connection: Upgrade +< Upgrade: websocket +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap-zero-200.gold b/tests/gold_tests/remap_yaml/gold/remap-zero-200.gold new file mode 100644 index 00000000000..bcad7913071 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap-zero-200.gold @@ -0,0 +1,7 @@ +`` +> GET / HTTP/1.1 +> Host: zero.one.two.three.com +`` +< HTTP/1.1 200 OK +< Date: `` +`` diff --git a/tests/gold_tests/remap_yaml/gold/remap2-200.gold b/tests/gold_tests/remap_yaml/gold/remap2-200.gold new file mode 100644 index 00000000000..19ec0035312 --- /dev/null +++ b/tests/gold_tests/remap_yaml/gold/remap2-200.gold @@ -0,0 +1,14 @@ +`` +> GET `` +> Host: www.example2.com`` +> User-Agent: curl/`` +> Accept: */* +`` +< HTTP/1.1 200 OK +< Date: `` +< Age: `` +< Transfer-Encoding: chunked +< Proxy-Connection: keep-alive +< Server: ATS/`` +< +`` diff --git a/tests/gold_tests/remap_yaml/map_with_recv_port_yaml.test.py b/tests/gold_tests/remap_yaml/map_with_recv_port_yaml.test.py new file mode 100644 index 00000000000..5513533a1f2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/map_with_recv_port_yaml.test.py @@ -0,0 +1,87 @@ +''' +Verify correct behavior of regex_map in remap.config. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify correct behavior of map_with_recv_port in remap.config. +''' + +Test.ContinueOnFail = True +ts = Test.MakeATSProcess("ts") +server = Test.MakeOriginServer("server") +dns = Test.MakeDNServer("dns", default='127.0.0.1') + +Test.testName = "" +request_header_ip = {"headers": "GET /ip HTTP/1.1\r\nHost: origin.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +response_header_ip = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": "ip"} +request_header_unix = { + "headers": "GET /unix HTTP/1.1\r\nHost: origin.example.com\r\n\r\n", + "timestamp": "1469733493.993", + "body": "" +} +response_header_unix = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": "unix"} +request_header_error = { + "headers": "GET /error HTTP/1.1\r\nHost: origin.example.com\r\n\r\n", + "timestamp": "1469733493.993", + "body": "" +} +response_header_error = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": "error"} +server.addResponse("sessionfile.log", request_header_ip, response_header_ip) +server.addResponse("sessionfile.log", request_header_unix, response_header_unix) +server.addResponse("sessionfile.log", request_header_error, response_header_error) + +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|dns', + 'proxy.config.dns.nameservers': '127.0.0.1:{0}'.format(dns.Variables.Port), + 'proxy.config.dns.resolv_conf': 'NULL' + }) + +# This map rule should not match +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://test.example.com + to: + url: http://origin.example.com:{server.Variables.Port}/error + - type: map_with_recv_port + from: + url: http://test.example.com:{ts.Variables.port}/ + to: + url: http://origin.example.com:{server.Variables.Port}/ip + - type: map_with_recv_port + from: + url: http+unix://test.example.com + to: + url: http://origin.example.com:{server.Variables.Port}/unix + '''.split("\n")) + +tr = Test.AddTestRun() +tr.MakeCurlCommand('-H"Host: test.example.com" http://127.0.0.1:{0}/ --verbose'.format(ts.Variables.port), ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(dns) +tr.Processes.Default.StartBefore(Test.Processes.ts) +if Condition.CurlUsingUnixDomainSocket(): + tr.Processes.Default.Streams.stderr = "gold/map-with-recv-port-unix.gold" +else: + tr.Processes.Default.Streams.stderr = "gold/map-with-recv-port-ip.gold" +tr.StillRunningAfter = server diff --git a/tests/gold_tests/remap_yaml/regex_map_yaml.test.py b/tests/gold_tests/remap_yaml/regex_map_yaml.test.py new file mode 100644 index 00000000000..1565337b1d3 --- /dev/null +++ b/tests/gold_tests/remap_yaml/regex_map_yaml.test.py @@ -0,0 +1,65 @@ +''' +Verify correct behavior of regex_map in remap.yaml. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify correct behavior of regex_map in remap.config. +''' + +Test.ContinueOnFail = True +ts = Test.MakeATSProcess("ts") +server = Test.MakeOriginServer("server") +dns = Test.MakeDNServer("dns", default='127.0.0.1') + +Test.testName = "" +request_header = {"headers": "GET / HTTP/1.1\r\nHost: zero.one.two.three.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +response_header = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +server.addResponse("sessionfile.log", request_header, response_header) + +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http.*|dns|conf_remap', + 'proxy.config.http.referer_filter': 1, + 'proxy.config.dns.nameservers': '127.0.0.1:{0}'.format(dns.Variables.Port), + 'proxy.config.dns.resolv_conf': 'NULL' + }) + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: regex_map + from: + url: http://(.*)?one\.two\.three\.com/ + to: + url: http://$1reactivate.four.five.six.com:{server.Variables.Port}/ + - type: regex_map + from: + url: https://\b(?!(.*one|two|three|four|five|six)).+\b\.seven\.eight\.nine\.com/blah12345.html + to: + url: https://www.example.com:{server.Variables.Port}/one/two/three/blah12345.html + '''.split("\n")) + +tr = Test.AddTestRun() +tr.MakeCurlCommand('-H"Host: zero.one.two.three.com" http://127.0.0.1:{0}/ --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(dns) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Streams.stderr = "gold/remap-zero-200.gold" +tr.StillRunningAfter = server diff --git a/tests/gold_tests/remap_yaml/reload_1.replay.yaml b/tests/gold_tests/remap_yaml/reload_1.replay.yaml new file mode 100644 index 00000000000..a65574262e2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/reload_1.replay.yaml @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This replay file assumes that caching is enabled and +# proxy.config.http.cache.ignore_client_cc_max_age is set to 0 so that we can +# test max-age in the client requests. +# + +meta: + version: "1.0" + +sessions: +- transactions: + - all: { headers: { fields: [[ uuid, success ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /index.html + headers: + fields: + - [ Host, charlie.ex ] + + proxy-response: + status: 200 diff --git a/tests/gold_tests/remap_yaml/reload_2.replay.yaml b/tests/gold_tests/remap_yaml/reload_2.replay.yaml new file mode 100644 index 00000000000..a65574262e2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/reload_2.replay.yaml @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This replay file assumes that caching is enabled and +# proxy.config.http.cache.ignore_client_cc_max_age is set to 0 so that we can +# test max-age in the client requests. +# + +meta: + version: "1.0" + +sessions: +- transactions: + - all: { headers: { fields: [[ uuid, success ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /index.html + headers: + fields: + - [ Host, charlie.ex ] + + proxy-response: + status: 200 diff --git a/tests/gold_tests/remap_yaml/reload_3.replay.yaml b/tests/gold_tests/remap_yaml/reload_3.replay.yaml new file mode 100644 index 00000000000..ac49c7b6548 --- /dev/null +++ b/tests/gold_tests/remap_yaml/reload_3.replay.yaml @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This replay file assumes that caching is enabled and +# proxy.config.http.cache.ignore_client_cc_max_age is set to 0 so that we can +# test max-age in the client requests. +# + +meta: + version: "1.0" + +sessions: +- transactions: + - all: { headers: { fields: [[ uuid, success ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /index.html + headers: + fields: + - [ Host, charlie.ex ] + + proxy-response: + status: 404 diff --git a/tests/gold_tests/remap_yaml/reload_4.replay.yaml b/tests/gold_tests/remap_yaml/reload_4.replay.yaml new file mode 100644 index 00000000000..fc1e00261de --- /dev/null +++ b/tests/gold_tests/remap_yaml/reload_4.replay.yaml @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This replay file assumes that caching is enabled and +# proxy.config.http.cache.ignore_client_cc_max_age is set to 0 so that we can +# test max-age in the client requests. +# + +meta: + version: "1.0" + +sessions: +- transactions: + - all: { headers: { fields: [[ uuid, success ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /index.html + headers: + fields: + - [ Host, golf.ex ] + + proxy-response: + status: 200 diff --git a/tests/gold_tests/remap_yaml/reload_server.replay.yaml b/tests/gold_tests/remap_yaml/reload_server.replay.yaml new file mode 100644 index 00000000000..c7045e63fb2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/reload_server.replay.yaml @@ -0,0 +1,51 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This replay file assumes that caching is enabled and +# proxy.config.http.cache.ignore_client_cc_max_age is set to 0 so that we can +# test max-age in the client requests. +# + +meta: + version: "1.0" + + blocks: + - 200_ok_response: &200_ok_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + +sessions: +- transactions: + - all: { headers: { fields: [[ uuid, success ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /index.html + headers: + fields: + - [ Host, charlie.ex ] + + <<: *200_ok_response + + proxy-response: + status: 200 diff --git a/tests/gold_tests/remap_yaml/remap_acl_all_allowed.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_all_allowed.replay.yaml new file mode 100644 index 00000000000..06501c9ed6f --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_all_allowed.replay.yaml @@ -0,0 +1,120 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config to allow all requests. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + # Not received. + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + proxy-response: + status: 400 diff --git a/tests/gold_tests/remap_yaml/remap_acl_all_denied.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_all_denied.replay.yaml new file mode 100644 index 00000000000..3e727f5bcb5 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_all_denied.replay.yaml @@ -0,0 +1,124 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that denies all request methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + # Not received. + <<: *standard_response + + proxy-response: + status: 403 + + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + # Not received. + <<: *standard_response + + proxy-response: + status: 403 + + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + # Not received. + <<: *standard_response + + proxy-response: + status: 403 + + # DELETE rejected + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + proxy-response: + status: 403 diff --git a/tests/gold_tests/remap_yaml/remap_acl_get_allowed.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_get_allowed.replay.yaml new file mode 100644 index 00000000000..648fdc137a2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_get_allowed.replay.yaml @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that allows GET, but denies all other +# methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 403 + + # PUT rejected + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + # Not received. + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 + + # DELETE rejected + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + # Verify that ATS rejects the DELETE. + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 diff --git a/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed.replay.yaml new file mode 100644 index 00000000000..26e6297c4c9 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed.replay.yaml @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that allows GET and POST, but denies all other +# methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + # POST also is in the allow list. + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 200 + + # PUT rejected + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + # Not received. + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 + + # DELETE rejected + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + # Verify that ATS rejects the DELETE. + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 diff --git a/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed_pp.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed_pp.replay.yaml new file mode 100644 index 00000000000..1e8b9323b0f --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_get_post_allowed_pp.replay.yaml @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that allows GET and POST, but denies all other +# methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: proxy-protocol + version: 2 + src-addr: "1.2.3.4:1111" + dst-addr: "5.6.7.8:2222" + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + # POST also is in the allow list. + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 200 + + # PUT rejected + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + # Not received. + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 + + # DELETE rejected + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + # Verify that ATS rejects the DELETE. + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 diff --git a/tests/gold_tests/remap_yaml/remap_acl_get_post_denied.replay.yaml b/tests/gold_tests/remap_yaml/remap_acl_get_post_denied.replay.yaml new file mode 100644 index 00000000000..9093a8bb363 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_get_post_denied.replay.yaml @@ -0,0 +1,122 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This expects a remap.config that denies GET and POST, but allows all other +# methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 403 + + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 403 + + # All other methods are allowed. + - client-request: + method: "PUT" + version: "1.1" + url: /test/ip_allow/test_put + headers: + fields: + - [ Host, example.com ] + - [ uuid, put ] + - [ X-Request, put ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Host, example.com ] + - [ uuid, delete ] + - [ X-Request, delete ] + - [ Content-Length, 0 ] + + <<: *standard_response + + proxy-response: + status: 200 + + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # ATS will allow the PUSH, but issue a 400 saying that it can't cache it. + proxy-response: + status: 400 diff --git a/tests/gold_tests/remap_yaml/remap_acl_yaml.test.py b/tests/gold_tests/remap_yaml/remap_acl_yaml.test.py new file mode 100644 index 00000000000..a200fa229a7 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_acl_yaml.test.py @@ -0,0 +1,662 @@ +''' +Verify remap.yaml acl behavior. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# # http://www.apache.org/licenses/LICENSE-2.0 # +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import io +import re +import inspect +import tempfile +from yaml import load, dump +from yaml import CLoader as Loader +from typing import List, Tuple + +from ports import get_port + +Test.Summary = ''' +Verify remap.yaml acl behavior. +''' + + +def update_config_file(path1: str, content1: str, path2: str, content2: str) -> None: + """Update two config files. + + This is used for some of the updates to the config files between test runs. + + :param path1: The path to the first config file. + :param content1: The content to write to the first config file. + :param path2: The path to the second config file. + :param content2: The content to write to the second config file. + """ + with open(path1, 'w') as f: + f.write(content1 + '\n') + with open(path2, 'w') as f: + f.write(content2 + '\n') + + +class Test_remap_acl: + """Configure a test to verify remap.config acl behavior.""" + + _ts: 'TestProcess' = None + _ts_reload_counter: int = 0 + _ts_is_started: bool = False + + _server_counter: int = 0 + _client_counter: int = 0 + + def __init__( + self, name: str, replay_file: str, ip_allow_content: str, deactivate_ip_allow: bool, acl_behavior_policy: int, + acl_configuration: List[str], named_acls: List[Tuple[str, + List[str]]], expected_responses: List[int], proxy_protocol: bool): + """Initialize the test. + + :param name: The name of the test. + :param replay_file: The replay file to be used. + :param ip_allow_content: The ip_allow configuration to be used. + :param deactivate_ip_allow: Whether to deactivate the ip_allow filter. + :param acl_configuration: The ACL configuration to be used. + :param named_acls: The set of named ACLs to configure and use. + :param expect_responses: The in-order expected responses from the proxy. + """ + self._replay_file = replay_file + self._ip_allow_lines = ip_allow_content.split("\n") + self._deactivate_ip_allow = deactivate_ip_allow + self._acl_behavior_policy = acl_behavior_policy + self._acl_configuration = acl_configuration + self._named_acls = named_acls + self._expected_responses = expected_responses + + # Usually we configure the server first and use the server port to + # configure ATS to remap to it. In this case, though, we want a + # long-lived ATS process that spans TestRuns. So we let ATS choose an + # arbitrary availble server port, and then tell the TestRun-specific + # server to use that port. + server_port = self._configure_traffic_server() + tr = Test.AddTestRun(name) + self._configure_server(tr, server_port) + self._configure_client(tr, proxy_protocol) + + def _configure_server(self, tr: 'TestRun', server_port: int) -> None: + """Configure the server. + """ + name = f"server-{Test_remap_acl._server_counter}" + server = tr.AddVerifierServerProcess(name, self._replay_file, http_ports=[server_port]) + Test_remap_acl._server_counter += 1 + self._server = server + + def _configure_traffic_server(self) -> int: + """Configure Traffic Server. + + :return: The listening port that the server should use. + """ + + call_reload: bool = False + if Test_remap_acl._ts is not None: + ts = Test_remap_acl._ts + call_reload = True + else: + ts = Test.MakeATSProcess("ts", enable_cache=False, enable_proxy_protocol=True, enable_uds=False) + Test_remap_acl._ts = ts + self._ts = ts + port_name = f'ServerPort-{Test_remap_acl._ts_reload_counter}' + server_port: int = get_port(ts, port_name) + + remap_yaml_lines = ['remap:'] + if self._deactivate_ip_allow: + remap_yaml_lines.append('- deactivate_filter: ip_allow') + + # First, define the name ACLs (filters). + defined_filters = [] + for name, definition in self._named_acls: + if len(definition) > 0: + remap_yaml_lines.append('- define_filter:') + remap_yaml_lines.append(f' {name}:') + for d in definition: + remap_yaml_lines.append(f' {d}') + defined_filters.append(name) + # Now activate them. + for name in defined_filters: + remap_yaml_lines.append(f'- activate_filter: {name}') + + remap_yaml_lines.append('- type: map') + remap_yaml_lines.append(' from:') + remap_yaml_lines.append(' url: /') + remap_yaml_lines.append(' to:') + remap_yaml_lines.append(f' url: http://127.0.0.1:{server_port}') + + if len(self._acl_configuration) > 0: + remap_yaml_lines.append(' acl_filter:') + for f in self._acl_configuration: + remap_yaml_lines.append(f' {f}') + + if call_reload: + # + # Update the ATS configuration. + # + tr = Test.AddTestRun("Change the ATS configuration") + p = tr.Processes.Default + p.Command = ( + f'traffic_ctl config set proxy.config.http.connect_ports {server_port} && ' + f'traffic_ctl config set proxy.config.url_remap.acl_behavior_policy {self._acl_behavior_policy}') + + p.Env = ts.Env + tr.StillRunningAfter = ts + + remap_yaml_path = os.path.join(ts.Variables.CONFIGDIR, 'remap.yaml') + ip_allow_path = os.path.join(ts.Variables.CONFIGDIR, 'ip_allow.yaml') + p.Setup.Lambda( + lambda: update_config_file( + remap_yaml_path, '\n'.join(remap_yaml_lines), ip_allow_path, '\n'.join(self._ip_allow_lines))) + + # + # Kick off the ATS config reload. + # + tr = Test.AddTestRun("Reload the ATS configuration") + p = tr.Processes.Default + p.Command = 'traffic_ctl config reload' + p.Env = ts.Env + tr.StillRunningAfter = ts + + # + # Await the config reload to finish. + # + tr = Test.AddTestRun("Await config reload") + p = tr.Processes.Default + p.Command = 'echo awaiting config reload' + p.Env = ts.Env + Test_remap_acl._ts_reload_counter += 1 + count = Test_remap_acl._ts_reload_counter + await_config_reload = tr.Processes.Process(f'config_reload_succeeded_{count}', 'sleep 30') + await_config_reload.Ready = When.FileContains(ts.Disk.diags_log.Name, "remap.yaml finished loading", count) + p.StartBefore(await_config_reload) + + else: + record_config = { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|url|remap|ip_allow|proxyprotocol', + 'proxy.config.http.push_method_enabled': 1, + 'proxy.config.http.connect_ports': server_port, + 'proxy.config.url_remap.acl_behavior_policy': self._acl_behavior_policy, + 'proxy.config.acl.subjects': 'PROXY,PEER', + } + + ts.Disk.records_config.update(record_config) + ts.Disk.remap_yaml.AddLines(remap_yaml_lines) + ts.Disk.ip_allow_yaml.AddLines(self._ip_allow_lines) + + return server_port + + def _configure_client(self, tr: 'TestRun', proxy_protocol: bool) -> None: + """Run the test. + + :param tr: The TestRun object to associate the client process with. + """ + + name = f"client-{Test_remap_acl._client_counter}" + ts = Test_remap_acl._ts + port = ts.Variables.port if proxy_protocol == False else ts.Variables.proxy_protocol_port + p = tr.AddVerifierClientProcess(name, self._replay_file, http_ports=[port]) + Test_remap_acl._client_counter += 1 + p.StartBefore(self._server) + if not Test_remap_acl._ts_is_started: + p.StartBefore(ts) + Test_remap_acl._ts_is_started = True + + if self._expected_responses == [None, None]: + # If there are no expected responses, expect the Warning about the rejected ip. + self._ts.Disk.diags_log.Content += Testers.ContainsExpression( + "client '127.0.0.1' prohibited by ip-allow policy", "Verify the client rejection warning message.") + + # Also, the client will complain about the broken connections. + p.ReturnCode = 1 + + else: + codes = [str(code) for code in self._expected_responses] + p.Streams.stdout += Testers.ContainsExpression( + '.*'.join(codes), "Verifying the expected order of responses", reflags=re.DOTALL | re.MULTILINE) + + +class Test_old_action: + _ts_counter: int = 0 + + def __init__(self, name: str, acl_filter: List[str], ip_allow_content: str) -> None: + '''Test that ATS fails with a FATAL message if an old action is used with modern ACL filter policy. + + :param name: The name of the test run. + :param acl_filter: The ACL filter to use. + :param ip_allow_content: The ip_allow configuration to use. + ''' + + tr = Test.AddTestRun(name) + ts = self._configure_traffic_server(tr, acl_filter, ip_allow_content) + + def _configure_traffic_server(self, tr: 'TestRun', acl_filter: List[str], ip_allow_content: str) -> 'Process': + '''Configure Traffic Server process + + :param tr: The TestRun object to associate the Traffic Server process with. + :param acl_filter: The ACL filter to configure in remap.config. + :param ip_allow_content: The ip_allow configuration to use. + :return: The Traffic Server process. + ''' + name = f"ts-old-action-{Test_old_action._ts_counter}" + Test_old_action._ts_counter += 1 + ts = tr.MakeATSProcess(name, enable_uds=False) + self._ts = ts + + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|url|remap|ip_allow', + 'proxy.config.url_remap.acl_behavior_policy': 1, + }) + + ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: / + to: + url: http://127.0.0.1:8080 + acl_filter: '''.split("\n")) + + for f in acl_filter: + ts.Disk.remap_yaml.AddLine(f' {f}') + + if ip_allow_content: + ts.Disk.ip_allow_yaml.AddLines(ip_allow_content.split("\n")) + + if len(acl_filter) > 0: + expected_error = '"allow" and "deny" are no longer valid.' + else: + expected_error = 'Legacy action name of' + + # We have to wait upon TS to emit the expected log message, but it cannot be + # the ts Ready criteria because autest might detect the process going away + # before it detects the log message. So we add a separate process that waits + # upon the log message. + watcher = tr.Processes.Process("watcher") + watcher.Command = "sleep 10" + watcher.Ready = When.FileContains(ts.Disk.diags_log.Name, expected_error) + watcher.StartBefore(ts) + + tr.Processes.Default.Command = 'printf "Fatal Shutdown Test"' + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.StartBefore(watcher) + + tr.Timeout = 5 + ts.ReturnCode = Any(33, 70) + ts.Ready = 0 + ts.Disk.diags_log.Content = Testers.IncludesExpression(expected_error, 'ATS should fatal with the old actions.') + + return ts + + +IP_ALLOW_OLD_ACTION = f''' +ip_categories: + - name: ACME_LOCAL + ip_addrs: 127.0.0.1 + - name: ACME_EXTERNAL + ip_addrs: 5.6.7.8 + +ip_allow: + - apply: in + ip_addrs: 0/0 + action: allow + methods: + - GET +''' + +IP_ALLOW_CONTENT = f''' +ip_categories: + - name: ACME_LOCAL + ip_addrs: 127.0.0.1 + - name: ACME_EXTERNAL + ip_addrs: 5.6.7.8 + +ip_allow: + - apply: in + ip_addrs: 0/0 + action: set_allow + methods: + - GET +''' + +Test_old_action("Verify allow is reject in modern policy", ['action: allow', 'method: GET'], IP_ALLOW_CONTENT) +Test_old_action("Verify deny is reject in modern policy", ['action: deny', 'method: GET'], IP_ALLOW_CONTENT) +Test_old_action("Verify deny is reject in modern policy", [], IP_ALLOW_OLD_ACTION) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify non-allowed methods are blocked.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods_pp = Test_remap_acl( + "Verify non-allowed methods are blocked (PP).", + replay_file='remap_acl_get_post_allowed_pp.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 1.2.3.4', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=True) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify add_allow adds an allowed method.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: add_allow', 'src_ip: 127.0.0.1', 'method: POST'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify add_allow adds allowed methods.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: add_allow', 'src_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify if no ACLs match, ip_allow.yaml is used.", + replay_file='remap_acl_get_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 1.2.3.4', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify @src_ip=all works.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: all', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify @src_ip_category works.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip_category: ACME_LOCAL', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify no @src_ip implies all IP addresses.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify denied methods are blocked.", + replay_file='remap_acl_get_post_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_deny', 'src_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[403, 403, 200, 200, 400], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify add_deny adds blocked methods.", + replay_file='remap_acl_all_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: add_deny', 'src_ip: 127.0.0.1', 'method: GET'], + named_acls=[], + expected_responses=[403, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify a default deny filter rule works.", + replay_file='remap_acl_all_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 1.2.3.4', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[403, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify inverting @src_ip works.", + replay_file='remap_acl_all_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip_invert: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[403, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify inverting @src_ip works with the rule matching.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip_invert: 3.4.5.6', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify inverting @src_ip_category works.", + replay_file='remap_acl_all_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip_category_invert: ACME_LOCAL', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[403, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify inverting @src_ip_category works with the rule matching.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip_category_invert: ACME_EXTERNAL', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify @src_ip and @src_ip_category AND together.", + replay_file='remap_acl_all_denied.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + # The rule will not match because, while @src_ip matches, @src_ip_category does not. + acl_configuration=['action: set_allow', 'src_ip: 127.0.0.1', 'src_ip_category: ACME_EXTERNAL', 'method: [GET, POST]'], + # Therefore, this named deny filter will block. + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[403, 403, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify defined in-line ACLS are evaluated before named ones.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[('deny', ['action: set_deny'])], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify remap.config line overrides ip_allow rule.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'src_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify we can deactivate the ip_allow filter.", + replay_file='remap_acl_all_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=True, + acl_behavior_policy=1, + # This won't match, so nothing will match since ip_allow.yaml is off. + acl_configuration=['action: set_allow', 'src_ip: 1.2.3.4', 'method: [GET, POST]'], + named_acls=[], + # Nothing will block the request since ip_allow.yaml is off. + expected_responses=[200, 200, 200, 200, 400], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify in_ip matches on IP as expected.", + replay_file='remap_acl_get_post_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'in_ip: 127.0.0.1', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 200, 403, 403, 403], + proxy_protocol=False) + +test_ip_allow_optional_methods = Test_remap_acl( + "Verify in_ip rules do not match on other IPs.", + replay_file='remap_acl_get_allowed.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=['action: set_allow', 'in_ip: 3.4.5.6', 'method: [GET, POST]'], + named_acls=[], + expected_responses=[200, 403, 403, 403, 403], + proxy_protocol=False) + +test_named_acl_deny = Test_remap_acl( + "Verify a named ACL is applied if an in-line ACL is absent.", + replay_file='deny_head_post.replay.yaml', + ip_allow_content=IP_ALLOW_CONTENT, + deactivate_ip_allow=False, + acl_behavior_policy=1, + acl_configuration=[], + named_acls=[('deny', ['action: set_deny', 'method: [HEAD, POST]'])], + expected_responses=[200, 403, 403, 403], + proxy_protocol=False) + + +def replay_proxy_response(filename, replay_file, get_proxy_response, post_proxy_response): + """ + replay_proxy_response writes the given replay file (which expects a single GET & POST client-request) + with the given proxy_response value. This is only used to support the tests in the combination table. + """ + + current_dir = os.path.dirname(inspect.getfile(inspect.currentframe())) + path = os.path.join(current_dir, filename) + data = None + with open(path) as f: + data = load(f, Loader=Loader) + for session in data["sessions"]: + for transaction in session["transactions"]: + method = transaction["client-request"]["method"] + if method == "GET": + transaction["proxy-response"]["status"] = 403 if get_proxy_response == None else get_proxy_response + elif method == "POST": + transaction["proxy-response"]["status"] = 403 if post_proxy_response == None else post_proxy_response + else: + raise Exception("Expected to find GET or POST request, found %s", method) + with open(replay_file, "w") as f: + f.write(dump(data)) + + +from deactivate_ip_allow_yaml import all_deactivate_ip_allow_tests_yaml +from all_acl_combinations_yaml import all_acl_combination_tests_yaml +""" +Test all acl combinations +""" +for idx, test in enumerate(all_acl_combination_tests_yaml): + (_, replay_file_name) = tempfile.mkstemp(suffix="acl_table_test_{}.replay".format(idx)) + replay_proxy_response( + "base.replay.yaml", + replay_file_name, + test["GET response"], + test["POST response"], + ) + Test_remap_acl( + "allcombo-{0} {1} {2} {3}".format(idx, test["inline"], test["named_acl"], test["ip_allow"]), + replay_file=replay_file_name, + ip_allow_content=test["ip_allow"], + deactivate_ip_allow=False, + acl_behavior_policy=0 if test["policy"] == "legacy" else 1, + acl_configuration=test["inline"], + named_acls=[("acl", test["named_acl"])] if test["named_acl"] != "" else [], + expected_responses=[test["GET response"], test["POST response"]], + proxy_protocol=False, + ) +""" +Test all ACL combinations +""" +for idx, test in enumerate(all_deactivate_ip_allow_tests_yaml): + try: + test["deactivate_ip_allow"] + except: + print(test) + (_, replay_file_name) = tempfile.mkstemp(suffix="deactivate_ip_allow_table_test_{}.replay".format(idx)) + replay_proxy_response( + "base.replay.yaml", + replay_file_name, + test["GET response"], + test["POST response"], + ) + Test_remap_acl( + "ipallow-{0} {1} {2} {3}".format(idx, test["inline"], test["named_acl"], test["ip_allow"]), + replay_file=replay_file_name, + ip_allow_content=test["ip_allow"], + deactivate_ip_allow=test["deactivate_ip_allow"], + acl_behavior_policy=0 if test["policy"] == "legacy" else 1, + acl_configuration=test["inline"], + named_acls=[("acl", test["named_acl"])] if test["named_acl"] != "" else [], + expected_responses=[test["GET response"], test["POST response"]], + proxy_protocol=False, + ) diff --git a/tests/gold_tests/remap_yaml/remap_http_yaml.test.py b/tests/gold_tests/remap_yaml/remap_http_yaml.test.py new file mode 100644 index 00000000000..204246fae99 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_http_yaml.test.py @@ -0,0 +1,193 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test a basic remap of a http connection +''' + +Test.ContinueOnFail = True +# Define default ATS +ts = Test.MakeATSProcess("ts") +server = Test.MakeOriginServer("server") +server2 = Test.MakeOriginServer("server2", lookup_key="{%Host}{PATH}") +dns = Test.MakeDNServer("dns") + +Test.testName = "" +request_header = {"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +# expected response from the origin server +response_header = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} + +request_header2 = {"headers": "GET /test HTTP/1.1\r\nHost: www.testexample.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +# expected response from the origin server +response_header2 = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} + +# add response to the server dictionary +server.addResponse("sessionfile.log", request_header, response_header) +server2.addResponse("sessionfile.log", request_header2, response_header2) +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http.*|dns|conf_remap|remap_yaml', + 'proxy.config.http.referer_filter': 1, + 'proxy.config.dns.nameservers': '127.0.0.1:{0}'.format(dns.Variables.Port), + 'proxy.config.dns.resolv_conf': 'NULL' + }) + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://www.example.com + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: map_with_recv_port + from: + url: http://www.example2.com:{ts.Variables.port} + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: map + from: + url: http://www.example.com:8080 + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: redirect + from: + url: http://test3.com + to: + url: http://httpbin.org + - type: map_with_referer + from: + url: http://test4.com + to: + url: http://127.0.0.1:{server.Variables.Port} + redirect: + url: http://httpbin.org + regex: + - (.*[.])?persia[.]com + - type: map + from: + url: http://testDNS.com + to: + url: http://audrey.hepburn.com:{server.Variables.Port} + - type: map + from: + url: http://www.testexample.com + to: + url: http://127.0.0.1:{server2.Variables.Port} + plugins: + - name: conf_remap.so + params: + - proxy.config.url_remap.pristine_host_hdr=1 + '''.split("\n")) + +dns.addRecords(records={"audrey.hepburn.com.": ["127.0.0.1"]}) +dns.addRecords(records={"whatever.com.": ["127.0.0.1"]}) + +# call localhost straight +tr = Test.AddTestRun() +tr.MakeCurlCommand('"http://127.0.0.1:{0}/" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(dns) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Streams.stderr = "gold/remap-hitATS-404.gold" +tr.StillRunningAfter = server + +# www.example.com host +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--proxy 127.0.0.1:{0} "http://www.example.com" -H "Proxy-Connection: keep-alive" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-200.gold" + +# www.example2.com host (match on receive port) +# map_with_recv_port doesn't work with UDS +if not Condition.CurlUsingUnixDomainSocket(): + tr = Test.AddTestRun() + tr.MakeCurlCommand( + '--proxy 127.0.0.1:{0} "http://www.example2.com" -H "Proxy-Connection: keep-alive" --verbose'.format(ts.Variables.port), + ts=ts) + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stderr = "gold/remap2-200.gold" + +# www.example.com:80 host +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://www.example.com:80/" -H "Proxy-Connection: keep-alive" --verbose'.format(ts.Variables.port), + ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-200.gold" + +# www.example.com:8080 host +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://www.example.com:8080" -H "Proxy-Connection: keep-alive" --verbose'.format(ts.Variables.port), + ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-200.gold" + +# no rule for this +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://www.test.com/" -H "Proxy-Connection: keep-alive" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-404.gold" + +# redirect result +tr = Test.AddTestRun() +tr.MakeCurlCommand(' --proxy 127.0.0.1:{0} "http://test3.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-redirect.gold" + +# referer hit +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://test4.com" --header "Referer: persia.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-referer-hit.gold" + +# referer miss +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://test4.com" --header "Referer: monkey.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-referer-miss.gold" + +# referer hit +tr = Test.AddTestRun() +tr.MakeCurlCommand( + ' --proxy 127.0.0.1:{0} "http://test4.com" --header "Referer: www.persia.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-referer-hit.gold" + +# DNS test +tr = Test.AddTestRun() +tr.MakeCurlCommand(' --proxy 127.0.0.1:{0} "http://testDNS.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-DNS-200.gold" + +# microserver lookup test +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--proxy 127.0.0.1:{0} "http://www.testexample.com/test" -H "Host: www.testexample.com" --verbose'.format(ts.Variables.port), + ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server2) +tr.Processes.Default.Streams.stderr = "gold/lookupTest.gold" +tr.StillRunningAfter = server2 diff --git a/tests/gold_tests/remap_yaml/remap_https_yaml.test.py b/tests/gold_tests/remap_yaml/remap_https_yaml.test.py new file mode 100644 index 00000000000..91da4d012fe --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_https_yaml.test.py @@ -0,0 +1,136 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test a basic remap of a http connection +''' + +Test.SkipIf(Condition.CurlUsingUnixDomainSocket()) +Test.ContinueOnFail = True +# Define default ATS +ts = Test.MakeATSProcess("ts", enable_tls=True) +server = Test.MakeOriginServer("server") +server2 = Test.MakeOriginServer("server2", ssl=True) + +# **testname is required** +testName = "" +request_header = {"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +# desired response form the origin server +response_header = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +server.addResponse("sessionlog.json", request_header, response_header) +server2.addResponse("sessionlog.json", request_header, response_header) + +# add ssl materials like key, certificates for the server +ts.addDefaultSSLFiles() + +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'lm|ssl', + 'proxy.config.ssl.server.cert.path': '{0}'.format(ts.Variables.SSLDir), + 'proxy.config.ssl.server.private_key.path': '{0}'.format(ts.Variables.SSLDir), + # enable ssl port + 'proxy.config.http.server_ports': '{0} {1}:proto=http2;http:ssl'.format(ts.Variables.port, ts.Variables.ssl_port), + 'proxy.config.ssl.client.verify.server.policy': 'PERMISSIVE', + }) + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: https://www.example.com + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: map + from: + url: https://www.example.com:{ts.Variables.ssl_port} + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: map_with_recv_port + from: + url: https://www.example3.com:{ts.Variables.ssl_port} + to: + url: http://127.0.0.1:{server.Variables.Port} + - type: map + from: + url: https://www.anotherexample.com + to: + url: https://127.0.0.1:{server2.Variables.SSL_Port} + '''.split("\n")) + +ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') + +# call localhost straight +tr = Test.AddTestRun() +tr.MakeCurlCommand('--http1.1 -k https://127.0.0.1:{0} --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 + +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(server2) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Streams.stderr = "gold/remap-hitATS-404.gold" +tr.StillRunningAfter = server +tr.StillRunningAfter = ts + +# www.example.com host +tr = Test.AddTestRun() +tr.MakeCurlCommand('--http1.1 -k https://127.0.0.1:{0} -H "Host: www.example.com" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-https-200.gold" + +# www.example.com:80 host +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--http1.1 -k https://127.0.0.1:{0} -H "Host: www.example.com:443" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-https-200.gold" + +# www.example.com:8080 host +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--http1.1 -k https://127.0.0.1:{0} -H "Host: www.example.com:{0}" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-https-200.gold" + +# www.example3.com (match on receive port) +tr = Test.AddTestRun() +tr.MakeCurlCommand('--http1.1 -k https://127.0.0.1:{0} -H "Host: www.example3.com" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-https-200_3.gold" + +# no rule for this +tr = Test.AddTestRun() +tr.MakeCurlCommand('--http1.1 -k https://127.0.0.1:{0} -H "Host: www.test.com" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-hitATS-404.gold" + +# bad port +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--http1.1 -k https://127.0.0.1:{0} -H "Host: www.example.com:1234" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-hitATS-404.gold" + +# map www.anotherexample.com to https://.com +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--http1.1 -k https://127.0.0.1:{0} -H "Host: www.anotherexample.com" --verbose'.format(ts.Variables.ssl_port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-https-200_2.gold" +tr.StillRunningAfter = server2 diff --git a/tests/gold_tests/remap_yaml/remap_ip_resolve_yaml.test.py b/tests/gold_tests/remap_yaml/remap_ip_resolve_yaml.test.py new file mode 100644 index 00000000000..4cdc3c0122d --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_ip_resolve_yaml.test.py @@ -0,0 +1,89 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test a basic ip_resolve override using an ipv6 server +''' + +Test.ContinueOnFail = True +# Define default ATS +ts = Test.MakeATSProcess("ts") +server = Test.MakeOriginServer("server") +server_v6 = Test.MakeOriginServer("server_v6", None, None, '::1', 0) + +dns = Test.MakeDNServer("dns") + +Test.testName = "" +request_header = {"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} +# expected response from the origin server +response_header = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} + +# add response to the server dictionary +server.addResponse("sessionfile.log", request_header, response_header) +server_v6.addResponse("sessionfile.log", request_header, response_header) +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http.*|dns|conf_remap', + 'proxy.config.http.referer_filter': 1, + 'proxy.config.dns.nameservers': '127.0.0.1:{0}'.format(dns.Variables.Port), + 'proxy.config.dns.resolv_conf': 'NULL', + 'proxy.config.hostdb.ip_resolve': 'ipv4' + }) + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://testDNS.com + to: + url: http://test.ipv4.only.com:{server.Variables.Port} + plugins: + - name: conf_remap.so + params: + - proxy.config.hostdb.ip_resolve=ipv6;ipv4;client + - type: map + from: + url: http://testDNS2.com + to: + url: http://test.ipv6.only.com:{server_v6.Variables.Port} + plugins: + - name: conf_remap.so + params: + - proxy.config.hostdb.ip_resolve=ipv6;only + '''.split("\n")) + +dns.addRecords(records={"test.ipv4.only.com.": ["127.0.0.1"]}) +dns.addRecords(records={"test.ipv6.only.com": ["127.0.0.1", "::1"]}) + +tr = Test.AddTestRun() +tr.MakeCurlCommand(' --proxy 127.0.0.1:{0} "http://testDNS.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(dns) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Streams.stderr = "gold/remap-DNS-200.gold" +tr.StillRunningAfter = server + +tr = Test.AddTestRun() +tr.MakeCurlCommand(' --proxy 127.0.0.1:{0} "http://testDNS2.com" --verbose'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(server_v6) +tr.Processes.Default.Streams.stderr = "gold/remap-DNS-ipv6-200.gold" +tr.StillRunningAfter = server_v6 diff --git a/tests/gold_tests/remap_yaml/remap_load_empty_failure_yaml.test.py b/tests/gold_tests/remap_yaml/remap_load_empty_failure_yaml.test.py new file mode 100644 index 00000000000..03991886426 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_load_empty_failure_yaml.test.py @@ -0,0 +1,42 @@ +''' +Verify correct behavior of regex_map in remap.yaml. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test minimum rules on load - fail on missing file. +''' +ts = Test.MakeATSProcess("ts") +ts.Disk.remap_yaml.AddLine(f"") # empty file +ts.Disk.records_config.update({'proxy.config.url_remap.min_rules_required': 1}) +ts.ReturnCode = 33 # expect to Emergency fail due to empty "remap.config". +ts.Ready = 0 + +tr = Test.AddTestRun("test") + +# We have to wait upon TS to emit the expected log message, but it cannot be +# the ts Ready criteria because autest might detect the process going away +# before it detects the log message. So we add a separate process that waits +# upon the log message. +watcher = Test.Processes.Process("watcher") +watcher.Command = "sleep 10" +watcher.Ready = When.FileContains(ts.Disk.diags_log.Name, "remap.yaml failed to load") +watcher.StartBefore(ts) + +tr.Processes.Default.Command = "echo howdy" +tr.TimeOut = 5 +tr.Processes.Default.StartBefore(watcher) diff --git a/tests/gold_tests/remap_yaml/remap_load_empty_success_yaml.test.py b/tests/gold_tests/remap_yaml/remap_load_empty_success_yaml.test.py new file mode 100644 index 00000000000..596973bde61 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_load_empty_success_yaml.test.py @@ -0,0 +1,33 @@ +''' +Verify correct behavior of regex_map in remap.yaml. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test minimum rules on load - succeed on empty file. +''' + +ts = Test.MakeATSProcess("ts") +ts.Disk.remap_yaml.AddLine(f"") # empty file +ts.Disk.records_config.update({f'proxy.config.url_remap.min_rules_required': 0}) + +tr = Test.AddTestRun("startup") +p = tr.Processes.Default +p.Command = "sh -c echo" +p.ReturnCode = 0 +p.StartBefore(Test.Processes.ts) +tr.StillRunningAfter = ts diff --git a/tests/gold_tests/remap_yaml/remap_load_missing_failure_yaml.test.py b/tests/gold_tests/remap_yaml/remap_load_missing_failure_yaml.test.py new file mode 100644 index 00000000000..8f38e7644e2 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_load_missing_failure_yaml.test.py @@ -0,0 +1,29 @@ +''' +Verify correct behavior of regex_map in remap.yaml. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test minimum rules on load - fail on missing file. +''' +ts = Test.MakeATSProcess("ts") +ts.Disk.records_config.update({'proxy.config.url_remap.min_rules_required': 1}) +ts.ReturnCode = 33 # expect to Emergency fail due to empty "remap.config". + +tr = Test.AddTestRun("test") +tr.Processes.Default.Command = "echo" +tr.Processes.Default.StartAfter(ts, ready=When.FileExists(ts.Disk.diags_log)) diff --git a/tests/gold_tests/remap_yaml/remap_load_missing_success_yaml.test.py b/tests/gold_tests/remap_yaml/remap_load_missing_success_yaml.test.py new file mode 100644 index 00000000000..1b2b519c645 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_load_missing_success_yaml.test.py @@ -0,0 +1,33 @@ +''' +Verify correct behavior of regex_map in remap.yaml. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test minimum rules on load - succeed on missing file. +''' + +ts = Test.MakeATSProcess("ts") + +ts.Disk.records_config.update({f'proxy.config.url_remap.min_rules_required': 0}) + +tr = Test.AddTestRun("startup") +p = tr.Processes.Default +p.Command = "sh -c echo" +p.ReturnCode = 0 +p.StartBefore(Test.Processes.ts) +tr.StillRunningAfter = ts diff --git a/tests/gold_tests/remap_yaml/remap_reload_yaml.test.py b/tests/gold_tests/remap_yaml/remap_reload_yaml.test.py new file mode 100644 index 00000000000..cd006903cc3 --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_reload_yaml.test.py @@ -0,0 +1,165 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os + + +def update_remap_yaml(path: str, lines: list) -> None: + """Update the remap.yaml file. + + This is used to update the config file between test runs without + triggering framework warnings about overriding file objects. + + :param path: The path to the remap.config file. + :param lines: The list of lines to write to the file. + """ + with open(path, 'w') as f: + f.write('\n'.join(lines) + '\n') + + +Test.Summary = ''' +Test remap reloading +''' +Test.testName = 'remap_reload' + +replay_file_1 = "reload_1.replay.yaml" +replay_file_2 = "reload_2.replay.yaml" +replay_file_3 = "reload_3.replay.yaml" +replay_file_4 = "reload_4.replay.yaml" + +tm = Test.MakeATSProcess("ts") +tm.Disk.diags_log.Content = Testers.ContainsExpression("remap.yaml failed to load", "Remap should fail to load") +remap_cfg_path = os.path.join(tm.Variables.CONFIGDIR, 'remap.yaml') + +pv = Test.MakeVerifierServerProcess("pv", "reload_server.replay.yaml") +pv_port = pv.Variables.http_port + +tm.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: http://alpha.ex + to: + url: http://alpha.ex:{pv_port} + - type: map + from: + url: http://bravo.ex + to: + url: http://bravo.ex:{pv_port} + - type: map + from: + url: http://charlie.ex + to: + url: http://charlie.ex:{pv_port} + - type: map + from: + url: http://delta.ex + to: + url: http://delta.ex:{pv_port} + '''.split("\n")) + +tm.Disk.records_config.update({'proxy.config.url_remap.min_rules_required': 3}) + +nameserver = Test.MakeDNServer("dns", default='127.0.0.1') +tm.Disk.records_config.update( + { + 'proxy.config.dns.nameservers': f"127.0.0.1:{nameserver.Variables.Port}", + 'proxy.config.dns.resolv_conf': 'NULL' + }) + +tr = Test.AddTestRun("verify load") +tr.Processes.Default.StartBefore(pv) +tr.Processes.Default.StartBefore(nameserver) +tr.Processes.Default.StartBefore(tm) +tr.AddVerifierClientProcess("client", replay_file_1, http_ports=[tm.Variables.port]) + +tr = Test.AddTestRun("Change remap.yaml to have only two remap rules") +p = tr.Processes.Default +p.Env = tm.Env +p.Command = 'echo "Change remap.yaml, two lines"' +p.Setup.Lambda( + lambda: update_remap_yaml( + remap_cfg_path, f''' +remap: + - type: map + from: + url: http://alpha.ex + to: + url: http://alpha.ex:{pv_port} + - type: map + from: + url: http://bravo.ex + to: + url: http://bravo.ex:{pv_port} + '''.split("\n"))) + +tr = Test.AddTestRun("remap_yaml reload, fails") +tr.Processes.Default.Env = tm.Env +tr.Processes.Default.Command = 'sleep 2; traffic_ctl config reload' + +tr = Test.AddTestRun("after first reload") +await_config_reload = tr.Processes.Process('config_reload_failed', 'sleep 30') +await_config_reload.Ready = When.FileContains(tm.Disk.diags_log.Name, "configuration is invalid") +tr.AddVerifierClientProcess("client_2", replay_file_2, http_ports=[tm.Variables.port]) +tr.Processes.Default.StartBefore(await_config_reload) + +tr = Test.AddTestRun("Change remap.yaml to have more than three remap rules") +p = tr.Processes.Default +p.Env = tm.Env +p.Command = 'echo "Change remap.yaml, more than three lines"' +p.Setup.Lambda( + lambda: update_remap_yaml( + remap_cfg_path, f''' +remap: + - type: map + from: + url: http://echo.ex + to: + url: http://echo.ex:{pv_port} + - type: map + from: + url: http://foxtrot.ex + to: + url: http://foxtrot.ex:{pv_port} + - type: map + from: + url: http://golf.ex + to: + url: http://golf.ex:{pv_port} + - type: map + from: + url: http://hotel.ex + to: + url: http://hotel.ex:{pv_port} + - type: map + from: + url: http://india.ex + to: + url: http://india.ex:{pv_port} + '''.split("\n"))) + +tr = Test.AddTestRun("remap_yaml reload, succeeds") +tr.Processes.Default.Env = tm.Env +tr.Processes.Default.Command = 'sleep 2; traffic_ctl config reload' + +tr = Test.AddTestRun("post update charlie") +await_config_reload = tr.Processes.Process('config_reload_succeeded', 'sleep 30') +await_config_reload.Ready = When.FileContains(tm.Disk.diags_log.Name, "remap.yaml finished loading", 2) +tr.Processes.Default.StartBefore(await_config_reload) +tr.AddVerifierClientProcess("client_3", replay_file_3, http_ports=[tm.Variables.port]) + +tr = Test.AddTestRun("post update golf") +tr.AddVerifierClientProcess("client_4", replay_file_4, http_ports=[tm.Variables.port]) diff --git a/tests/gold_tests/remap_yaml/remap_ws_yaml.test.py b/tests/gold_tests/remap_yaml/remap_ws_yaml.test.py new file mode 100644 index 00000000000..3de1350ae7e --- /dev/null +++ b/tests/gold_tests/remap_yaml/remap_ws_yaml.test.py @@ -0,0 +1,129 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test a basic remap of a websocket connections +''' + +Test.ContinueOnFail = True + +ts = Test.MakeATSProcess("ts", enable_tls=True) +server = Test.MakeOriginServer("server") + +testName = "Test WebSocket Remaps" +request_header = { + "headers": "GET /chat HTTP/1.1\r\nHost: www.example.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n", + "body": None +} +response_header = { + "headers": + "HTTP/1.1 101 OK\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=\r\n\r\n", + "body": None +} +server.addResponse("sessionlog.json", request_header, response_header) + +ts.addDefaultSSLFiles() + +ts.Disk.records_config.update( + { + 'proxy.config.ssl.server.cert.path': '{0}'.format(ts.Variables.SSLDir), + 'proxy.config.ssl.server.private_key.path': '{0}'.format(ts.Variables.SSLDir), + }) + +ts.Disk.remap_yaml.AddLines( + f''' +remap: + - type: map + from: + url: ws://www.example.com:{ts.Variables.port} + to: + url: ws://127.0.0.1:{server.Variables.Port} + - type: map + from: + url: wss://www.example.com:{ts.Variables.ssl_port} + to: + url: ws://127.0.0.1:{server.Variables.Port} + '''.split("\n")) + +ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') + +if not Condition.CurlUsingUnixDomainSocket(): + # wss mapping + tr = Test.AddTestRun() + tr.Processes.Default.StartBefore(server) + tr.Processes.Default.StartBefore(Test.Processes.ts, ready=1) + tr.MakeCurlCommand( + '--max-time 2 -v -s -q -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" -H "Sec-WebSocket-Version: 13" --http1.1 --resolve www.example.com:{0}:127.0.0.1 -k https://www.example.com:{0}/chat' + .format(ts.Variables.ssl_port), + ts=ts) + tr.Processes.Default.ReturnCode = 28 + tr.Processes.Default.Streams.stderr = "gold/remap-ws-upgrade.gold" + tr.StillRunningAfter = server + tr.StillRunningAfter = ts + +# ws mapping +tr = Test.AddTestRun() +if Condition.CurlUsingUnixDomainSocket(): + tr.Processes.Default.StartBefore(server) + tr.Processes.Default.StartBefore(Test.Processes.ts, ready=1) +tr.MakeCurlCommand( + '--max-time 2 -v -s -q -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" -H "Sec-WebSocket-Version: 13" --http1.1 --resolve www.example.com:{0}:127.0.0.1 -k http://www.example.com:{0}/chat' + .format(ts.Variables.port), + ts=ts) +tr.Processes.Default.ReturnCode = 28 +tr.Processes.Default.Streams.stderr = "gold/remap-ws-upgrade.gold" +tr.StillRunningAfter = server +tr.StillRunningAfter = ts + +# Missing required headers (should result in 400) +tr = Test.AddTestRun() +tr.MakeCurlCommand( + '--max-time 2 -v -s -q -H "Connection: Upgrade" -H "Upgrade: websocket" --http1.1 --resolve www.example.com:{0}:127.0.0.1 -k http://www.example.com:{0}/chat' + .format(ts.Variables.port), + ts=ts) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stderr = "gold/remap-ws-upgrade-400.gold" +tr.StillRunningAfter = server +tr.StillRunningAfter = ts + +# Test metrics +tr = Test.AddTestRun() +if Condition.CurlUsingUnixDomainSocket(): + metrics_gold_file = 'remap-ws-metrics-uds.gold' +else: + metrics_gold_file = 'remap-ws-metrics.gold' +tr.Processes.Default.Command = ( + f"{Test.Variables.AtsTestToolsDir}/stdout_wait" + " 'traffic_ctl metric get" + + " proxy.process.http.total_incoming_connections" + " proxy.process.http.total_client_connections" + + " proxy.process.http.total_client_connections_ipv4" + " proxy.process.http.total_client_connections_ipv6" + + " proxy.process.http.total_server_connections" + " proxy.process.http2.total_client_connections" + + " proxy.process.http.connect_requests" + " proxy.process.tunnel.total_client_connections_blind_tcp" + + " proxy.process.tunnel.current_client_connections_blind_tcp" + " proxy.process.tunnel.total_server_connections_blind_tcp" + + " proxy.process.tunnel.current_server_connections_blind_tcp" + " proxy.process.tunnel.total_client_connections_tls_tunnel" + + " proxy.process.tunnel.current_client_connections_tls_tunnel" + " proxy.process.tunnel.total_client_connections_tls_forward" + + " proxy.process.tunnel.current_client_connections_tls_forward" + + " proxy.process.tunnel.total_client_connections_tls_partial_blind" + + " proxy.process.tunnel.current_client_connections_tls_partial_blind" + + " proxy.process.tunnel.total_client_connections_tls_http" + " proxy.process.tunnel.current_client_connections_tls_http" + + " proxy.process.tunnel.total_server_connections_tls" + " proxy.process.tunnel.current_server_connections_tls'" + + f" {Test.TestDirectory}/gold/{metrics_gold_file}") +# Need to copy over the environment so traffic_ctl knows where to find the unix domain socket +tr.Processes.Default.Env = ts.Env +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = server +tr.StillRunningAfter = ts