[记录]解决亿级访问量Wordpress集群的幽灵故障

Shuai · 2023 年5 月 25 日 05:43

背景

手里的一个Wordpress站点最近经常(每个月都有几次)莫名其妙down掉，已经尝试很多次没法解决。
由于每次也都会自动恢复，也就一直没有找到这种故障的真实原因，所以我称之为幽灵故障
今天终于解决了，所以顺手对整个过程做个记录

故障表象

CPU打满
流量异常
NFSD进程异常
磁盘IO正常

架构

先说下这个Wordpress的架构

围绕整个东西向链路详细解释一下:

整个服务器集群最前端是Cloudflare, Cloudflare这里主要作为 CDN&filter&SSL 使用
LB是云自身的，是4层的. 主要功能是用来观测节点情况和自动伸缩负载均衡用
Nginx最开始有4台，后续因为故障的发生，最高伸缩至了12台，依然没有解决问题，后来降至6台
Nginx作为计算节点，自带运行有php，可向LB直接返回数据
自建的storage存储节点，通过nfs进行业务文件共享
Redis作为Wordpress的对象存储数据库，Wordpress使用插件对热门数据对象进行缓存，降低数据库压力
DB有两台，主从结构，主要是主操作，从作为read only进行负载分摊和热数据备份节点

排查

表象

从服务器指标上来看，最异常的主要就是 CPU/流量。

存储节点
- CPUs占用率80%以上,
- 带宽占用1GB+, 大部分是VPC内的局域网流量。
- 并且大部分流量的source就是这台存储服务器
计算节点
- 故障初期CPUs 100%占满，导致业务不可用
- 流量异常，主要是接收的数据来自存储节点

排查CPU

CPU的异常通过top查询，主要是来自nfsd的进程，该进程是集群网络文件共享的核心。每个进程占vCPU大概50-60%。
整个Wordpress的文件都通过nfsd在集群计算节点之前共享。但是排查了nfsd几个小时，依然没有发现问题。
日志正常，就是CPU高得离谱。通过lsof查询进程，也是一无所获，根本没查到预想的文件死锁之类的情况。

root@storage:~# lsof -p 2222
COMMAND  PID USER   FD      TYPE DEVICE SIZE/OFF NODE NAME
nfsd    2222 root  cwd       DIR  252,1     4096    2 /
nfsd    2222 root  rtd       DIR  252,1     4096    2 /
nfsd    2222 root  txt   unknown                      /proc/2222/exe

然后通过开关计算计算，想通过logs或者特征的方式，找出cpu异常的愿意，依然是没有结果

排查流量异常

通过iftop等分析，总结出主要是存储节点在给计算节点发包导致的。所以流量最大的就是存储节点，因为需要向多个计算节点发包。
所以随即通过tcpdump对存储节点进行抓包，并使用Wireshark进行网络数据包分析.

但是由于存储节点每秒上GB的数据进行传输，抓出来的包杂乱无章。只是知道充斥来很多NFS的数据包，但是并不知道具体发生了什么事情。

我甚至认为真的是NFS出现了漏洞，导致了这种类似于反射放大的DDOS。
但是如果是DDOS，只攻击内网的服务器，也不符合逻辑和我的预期。
以前发生这种幽灵故障的时候，基本都是止步于这个阶段。

计算节点流量排查

这次我对结算节点进行了单独的抓包

由于少了很多其他数据包的干扰，更清楚的看到了包的内容和传输过程。
之前只是知道请求了很多READDIR的TCP请求，这次大概知道了为什么。
海量READDIR的请求，然后返回一个数据包。造成了类似反射放大DDOS的效果。
由于Wireshark UI上看到的包都不大，但是由于TCP分片的效果。通过tcp follow分析，导致实际每个请求都是反射放大了N倍的。

那问题来了，到底是不是攻击？如果不是攻击那到底是什么原因？
然后我复制数据包中反问的内容，在存储Wordpress文件夹中模糊搜索相关字符串。发现文件处于这个目录下:

/wp-content/cache/tmpWpfc/

然后发现这个内容是由Wordpress的 WP fastest cache 插件造成的，我是买的正版插件啊，怎么还有这种bug??先不管，继续排查。。为什么NFS一个READDIR请求，就导致网络炸掉的原因也找到了。
因为在 /wp-content/cache/tmpWpfc/xxx/tag 目录下，有4.5万个文件夹
通过 ls -l | wc -l 发现在/wp-content/cache/tmpWpfc/xxx/tag 目录下有 4.5万个文件夹。
也就是说，计算节点一个READDIR请求，会有4.5万个文件夹的名字内容从存储节点返回给计算节点。
这也是为啥看起来像是反射DDOS攻击的原因了，确实就是，不过这个是来自于信任的TCP协议。

WHY

那为什么这个 WP fastest cache 插件会有这个问题呢？
通过查阅官方文档终于有了答案:

原文: Clear Cache Process

也就是说，这个插件不会对过期cache进行定时完整的物理删除。而且通过目录移动的方式，进行每次的小批量删除。
小批量删除的时候，就会读取目录树，进行遍历式的删除。由于我使用了网络存储，导致整个网络流量发生了几何形事的倍增，最终导致故障发生。我甚至怀疑由于是集群环境，甚至出现了文件死锁的问题。

如何解决

既然找到了是插件的原因，那么摆在我面前的就是如何解决这个问题。我心中有两个方案：

找个该静态页面生成缓存插件的替代品，但是测试成本高
不使用该类型插件，肯定会增加服务器负载。
找到更好的方式解决FPC(full page cache)的问题。

如何解决

我最终选择了使用更改整个服务器集群架构的方式，解决该问题。
我在整个集群架构中增加一层中间件，专门用作FPC。最终架构如下:

使用varnish替代LB的位置有以下几个好处

方便切换，只需要更改CF的解析。立即就可以从老的架构上更换到新的架构上
方便测试，只需要新开三台服务器，就可以做线上测试。不需要原始环境停机解决
快速恢复，如果架构更换期间出现问题，可以做快速恢复
速度更快，由于varnish相当于是在webserver直接做了memory cache，相对于插件这种形式，还需要进入php的逻辑，理论上速度已经快了百倍
抗压能力更好，由于网站访问量巨大。 varnish的response复用形式，可以更好的解决服务器并发压力的问题
成本更低，使用varnish后可以降低服务器使用量，撤销LB，更低的成本做更多的事，而且架构更简单
可观测性，由于varnish提供了丰富的接口，可对cache的实际情况做更多的观测。

遇到的坑

上面说了这么多好处，实际配置中还是遇到了比较多的坑。
首先我一直是varnish的忠实用户，有一定的使用经验，所以才让整个架构迁移过程比较平稳快速。
但是我是第一次给Wordpress配置varnish，而且还是集群环境下。
这里给出我的vcl文件配置，希望能帮助大家。我的配置也可能不适合你的环境，环境你在下面留言。

vcl 4.1;

import std;
import directors;

backend node {
    .host = "10.108.0.13";
    .port = "80";
}
backend node1 {
    .host = "10.108.0.14";
    .port = "80";
}
backend node2 {
    .host = "10.108.0.12";
    .port = "80";
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "10.108.0.0/24";
    "localhost";
    "127.0.0.1";
    "::1";
}

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(node);
    vdir.add_backend(node1);
    vdir.add_backend(node2);
}

sub vcl_recv {   
    # Remove empty query string parameters
    # e.g.: www.example.com/index.html?
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

    # Sorts query string parameters alphabetically for cache normalization purposes
    set req.url = std.querysort(req.url);

    # Remove the proxy header to mitigate the httpoxy vulnerability
    # See https://httpoxy.org/
    unset req.http.proxy;
    
    # Add X-Forwarded-Proto header when using https
    if (!req.http.X-Forwarded-Proto) {
            set req.http.X-Forwarded-Proto = "https";
    }
   # add UA Diff
if (req.http.User-Agent ~ "Mobile") {
        set req.http.X-Varnish-Device = "mobile";
    } else {
        set req.http.X-Varnish-Device = "desktop";
    }
# Set the client IP based on X-Forwarded-For header
if (req.http.X-Forwarded-For) {
    set req.http.X-Forwarded-For = regsub(req.http.X-Forwarded-For, ",.*$", "");
    set req.http.X-Real-IP = regsub(req.http.X-Forwarded-For, ".*,\s*", "");
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = req.http.X-Real-IP;
}
    # Purge logic to remove objects from the cache. 
    # Tailored to the Proxy Cache Purge WordPress plugin
    # See https://wordpress.org/plugins/varnish-http-purge/
    if(req.method == "PURGE") {
        if(!client.ip ~ purge) {
            return(synth(405,"PURGE not allowed for this IP address"));
        }
        if (req.http.X-Purge-Method == "regex") {
            ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host == " + req.http.host);
            return(synth(200, "Purged"));
        }
        ban("obj.http.x-url == " + req.url + " && obj.http.x-host == " + req.http.host);
        return(synth(200, "Purged"));
    }

    # Only handle relevant HTTP request methods
    if (
        req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE"
    ) {
        return (pipe);
    }

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
    }

    # Only cache GET and HEAD requests
    if (req.method != "GET" && req.method != "HEAD") {
        set req.http.X-Cacheable = "NO:REQUEST-METHOD";
        return(pass);
    }

    # Mark static files with the X-Static-File header, and remove any cookies
    # X-Static-File is also used in vcl_backend_response to identify static files
    if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
        set req.http.X-Static-File = "true";
        unset req.http.Cookie;
        return(hash);
    }

    # No caching of special URLs, logged in users and some plugins
    if (
        req.http.Cookie ~ "wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID" ||
        req.http.Authorization ||
        req.url ~ "add_to_cart" ||
        req.url ~ "edd_action" ||
        req.url ~ "nocache" ||
        req.url ~ "^/addons" ||
        req.url ~ "^/bb-admin" ||
        req.url ~ "^/bb-login.php" ||
        req.url ~ "^/bb-reset-password.php" ||
        req.url ~ "^/cart" ||
        req.url ~ "^/checkout" ||
        req.url ~ "^/control.php" ||
        req.url ~ "^/login" ||
        req.url ~ "^/logout" ||
        req.url ~ "^/lost-password" ||
        req.url ~ "^/my-account" ||
        req.url ~ "^/product" ||
        req.url ~ "^/register" ||
        req.url ~ "^/register.php" ||
        req.url ~ "^/server-status" ||
        req.url ~ "^/signin" ||
        req.url ~ "^/signup" ||
        req.url ~ "^/stats" ||
        req.url ~ "^/wc-api" ||
        req.url ~ "^/wp-admin" ||
        req.url ~ "^/wp-comments-post.php" ||
        req.url ~ "^/wp-cron.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^/wp-activate.php" ||
        req.url ~ "^/wp-mail.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^\?add-to-cart=" ||
        req.url ~ "^\?wc-api=" ||
        req.url ~ "^/preview=" ||
        req.url ~ "^/\.well-known/acme-challenge/"
    ) {
	     set req.http.X-Cacheable = "NO:Logged in/Got Sessions";
	     if(req.http.X-Requested-With == "XMLHttpRequest") {
		     set req.http.X-Cacheable = "NO:Ajax";
	     }
        return(pass);
    }

    # Remove any cookies left
    unset req.http.Cookie;
    return(hash);
  set req.backend_hint = vdir.backend();
}

sub vcl_hash {
    if(req.http.X-Varnish-Device) {
        # Create cache variations depending on the request protocol       
        hash_data(req.http.X-Varnish-Device);
    }
}

sub vcl_backend_response {
    # Inject URL & Host header into the object for asynchronous banning purposes
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;

    # If we dont get a Cache-Control header from the backend
    # we default to 1h cache for all objects
    if (!beresp.http.Cache-Control) {
        set beresp.ttl = 1h;
        set beresp.http.X-Cacheable = "YES:Forced";
    }

    # If the file is marked as static we cache it for 1 day
    if (bereq.http.X-Static-File == "true") {
        unset beresp.http.Set-Cookie;
        set beresp.http.X-Cacheable = "YES:Forced";
        set beresp.ttl = 1d;
    }

	# Remove the Set-Cookie header when a specific Wordfence cookie is set
    if (beresp.http.Set-Cookie ~ "wfvt_|wordfence_verifiedHuman") {
	    unset beresp.http.Set-Cookie;
	 }
	
    if (beresp.http.Set-Cookie) {
        set beresp.http.X-Cacheable = "NO:Got Cookies";
    } elseif(beresp.http.Cache-Control ~ "private") {
        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
    }	
}

sub vcl_deliver {
    # Debug header
    if(req.http.X-Cacheable) {
        set resp.http.X-Cacheable = req.http.X-Cacheable;    
    } elseif(obj.uncacheable) {
        if(!resp.http.X-Cacheable) {
            set resp.http.X-Cacheable = "NO:UNCACHEABLE";        
        }
    } elseif(!resp.http.X-Cacheable) {
        set resp.http.X-Cacheable = "YES";
    }
    
    # Cleanup of headers
    unset resp.http.x-url;
    unset resp.http.x-host;    
}

注意，该配置文件是适配于我集群环境下的配置文件。如果你想复用，请根据自己的环境修改。

总结

NFS性能低
wp fattest插件有大坑
插件文件夹遍历的问题+NFS网络传输，最终导致了类似于反射放大DDOS的效果。
集群分散的架构方便后续维护升级时的处理和故障感知。
varnish大法好
wp代码和架构真心烂
最初的架构设计也有问题，程序代码不应该放在NFS，应该和静态资源做分离。但是工作量大，也没去做

Shuai · 2023 年6 月 7 日 15:14

之前的版本，在处理相同URL得pc端和移动端的request时候，虽然有处理，但是处理得并不好。
引用了github大神的配置配置文件，得到解决：

我的具体配置如下：
大神的原始文件未动：
https://raw.githubusercontent.com/varnishcache/varnish-devicedetect/master/devicedetect.vcl

我Wordpress的配置文件

vcl 4.1;

import std;
import directors;
include "devicedetect.vcl";

backend node {
    .host = "10.108.0.13";
    .port = "80";
}
backend node1 {
    .host = "10.108.0.14";
    .port = "80";
}
backend node2 {
    .host = "10.108.0.12";
    .port = "80";
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "10.108.0.0/24";
    "localhost";
    "127.0.0.1";
    "::1";
}

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(node);
    vdir.add_backend(node1);
    vdir.add_backend(node2);
    vdir.policy = client.ip;
}

sub vcl_recv {   
    # Remove empty query string parameters
    # e.g.: www.example.com/index.html?
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

    # Sorts query string parameters alphabetically for cache normalization purposes
    set req.url = std.querysort(req.url);

    # Remove the proxy header to mitigate the httpoxy vulnerability
    # See https://httpoxy.org/
    unset req.http.proxy;
    
    # Add X-Forwarded-Proto header when using https
    if (!req.http.X-Forwarded-Proto) {
            set req.http.X-Forwarded-Proto = "https";
    }
    #    # add UA Diff
    # if (req.http.User-Agent ~ "Mobile") {
    #         set req.http.X-Varnish-Device = "mobile";
    #     } else {
    #         set req.http.X-Varnish-Device = "desktop";
    #     }
    call devicedetect;
    if(req.http.X-UA-Device ~ "^(mobile|tablet)\-.+$") {
        set req.http.X-UA-Device = "mobile";
    } else {
        set req.http.X-UA-Device = "desktop";
    }
    # Purge logic to remove objects from the cache. 
    # Tailored to the Proxy Cache Purge WordPress plugin
    # See https://wordpress.org/plugins/varnish-http-purge/
    if(req.method == "PURGE") {
        if(!client.ip ~ purge) {
            return(synth(405,"PURGE not allowed for this IP address"));
        }
        if (req.http.X-Purge-Method == "regex") {
            ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host == " + req.http.host);
            return(synth(200, "Purged"));
        }
        ban("obj.http.x-url == " + req.url + " && obj.http.x-host == " + req.http.host);
        return(synth(200, "Purged"));
    }

    # Only handle relevant HTTP request methods
    if (
        req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE"
    ) {
        return (pipe);
    }

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
    }

    # Only cache GET and HEAD requests
    if (req.method != "GET" && req.method != "HEAD") {
        set req.http.X-Cacheable = "NO:REQUEST-METHOD";
        return(pass);
    }

    # Mark static files with the X-Static-File header, and remove any cookies
    # X-Static-File is also used in vcl_backend_response to identify static files
    if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
        set req.http.X-Static-File = "true";
        unset req.http.Cookie;
        return(hash);
    }

    # No caching of special URLs, logged in users and some plugins
    if (
        req.http.Cookie ~ "wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID" ||
        req.http.Authorization ||
        req.url ~ "add_to_cart" ||
        req.url ~ "edd_action" ||
        req.url ~ "nocache" ||
        req.url ~ "^/addons" ||
        req.url ~ "^/bb-admin" ||
        req.url ~ "^/bb-login.php" ||
        req.url ~ "^/bb-reset-password.php" ||
        req.url ~ "^/cart" ||
        req.url ~ "^/checkout" ||
        req.url ~ "^/control.php" ||
        req.url ~ "^/login" ||
        req.url ~ "^/logout" ||
        req.url ~ "^/lost-password" ||
        req.url ~ "^/my-account" ||
        req.url ~ "^/product" ||
        req.url ~ "^/register" ||
        req.url ~ "^/register.php" ||
        req.url ~ "^/server-status" ||
        req.url ~ "^/signin" ||
        req.url ~ "^/signup" ||
        req.url ~ "^/stats" ||
        req.url ~ "^/wc-api" ||
        req.url ~ "^/wp-admin" ||
        req.url ~ "^/wp-comments-post.php" ||
        req.url ~ "^/wp-cron.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^/wp-activate.php" ||
        req.url ~ "^/wp-mail.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^\?add-to-cart=" ||
        req.url ~ "^\?wc-api=" ||
        req.url ~ "^/preview=" ||
        req.url ~ "^/\.well-known/acme-challenge/"
    ) {
	     set req.http.X-Cacheable = "NO:Logged in/Got Sessions";
	     if(req.http.X-Requested-With == "XMLHttpRequest") {
		     set req.http.X-Cacheable = "NO:Ajax";
	     }
        return(pass);
    }

    # Remove any cookies left
    unset req.http.Cookie;
    return(hash);
  set req.backend_hint = vdir.backend();
}

sub vcl_hash {
    hash_data(req.http.X-UA-Device);
}

sub vcl_backend_response {
    # Inject URL & Host header into the object for asynchronous banning purposes
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;

    # If we dont get a Cache-Control header from the backend
    # we default to 1h cache for all objects
    if (!beresp.http.Cache-Control) {
        set beresp.ttl = 1h;
        set beresp.http.X-Cacheable = "YES:Forced";
    }

    # If the file is marked as static we cache it for 1 day
    if (bereq.http.X-Static-File == "true") {
        unset beresp.http.Set-Cookie;
        set beresp.http.X-Cacheable = "YES:Forced";
        set beresp.ttl = 1d;
    }

	# Remove the Set-Cookie header when a specific Wordfence cookie is set
    if (beresp.http.Set-Cookie ~ "wfvt_|wordfence_verifiedHuman") {
	    unset beresp.http.Set-Cookie;
	 }
	
    if (beresp.http.Set-Cookie) {
        set beresp.http.X-Cacheable = "NO:Got Cookies";
    } elseif(beresp.http.Cache-Control ~ "private") {
        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
    }	
}

sub vcl_deliver {
    # Debug header
    if(req.http.X-Cacheable) {
        set resp.http.X-Cacheable = req.http.X-Cacheable;    
    } elseif(obj.uncacheable) {
        if(!resp.http.X-Cacheable) {
            set resp.http.X-Cacheable = "NO:UNCACHEABLE";        
        }
    } elseif(!resp.http.X-Cacheable) {
        set resp.http.X-Cacheable = "YES";
    }
    
    # Cleanup of headers
    unset resp.http.x-url;
    unset resp.http.x-host;    
}