------------------------ HAProxy Management Guide ------------------------ version 1.8 This document describes how to start, stop, manage, and troubleshoot HAProxy, as well as some known limitations and traps to avoid. It does not describe how to configure it (for this please read configuration.txt). Note to documentation contributors : This document is formatted with 80 columns per line, with even number of spaces for indentation and without tabs. Please follow these rules strictly so that it remains easily printable everywhere. If you add sections, please update the summary below for easier searching. Summary ------- 1. Prerequisites 2. Quick reminder about HAProxy's architecture 3. Starting HAProxy 4. Stopping and restarting HAProxy 5. File-descriptor limitations 6. Memory management 7. CPU usage 8. Logging 9. Statistics and monitoring 9.1. CSV format 9.2. Typed output format 9.3. Unix Socket commands 10. Tricks for easier configuration management 11. Well-known traps to avoid 12. Debugging and performance issues 13. Security considerations 1. Prerequisites ---------------- In this document it is assumed that the reader has sufficient administration skills on a UNIX-like operating system, uses the shell on a daily basis and is familiar with troubleshooting utilities such as strace and tcpdump. 2. Quick reminder about HAProxy's architecture ---------------------------------------------- HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is uses event multiplexing to schedule all of its activities instead of relying on the system to schedule between multiple activities. Most of the time it runs as a single process, so the output of "ps aux" on a system will report only one "haproxy" process, unless a soft reload is in progress and an older process is finishing its job in parallel to the new one. It is thus always easy to trace its activity using the strace utility. HAProxy is designed to isolate itself into a chroot jail during startup, where it cannot perform any file-system access at all. This is also true for the libraries it depends on (eg: libc, libssl, etc). The immediate effect is that a running process will not be able to reload a configuration file to apply changes, instead a new process will be started using the updated configuration file. Some other less obvious effects are that some timezone files or resolver files the libc might attempt to access at run time will not be found, though this should generally not happen as they're not needed after startup. A nice consequence of this principle is that the HAProxy process is totally stateless, and no cleanup is needed after it's killed, so any killing method that works will do the right thing. HAProxy doesn't write log files, but it relies on the standard syslog protocol to send logs to a remote server (which is often located on the same system). HAProxy uses its internal clock to enforce timeouts, that is derived from the system's time but where unexpected drift is corrected. This is done by limiting the time spent waiting in poll() for an event, and measuring the time it really took. In practice it never waits more than one second. This explains why, when running strace over a completely idle process, periodic calls to poll() (or any of its variants) surrounded by two gettimeofday() calls are noticed. They are normal, completely harmless and so cheap that the load they imply is totally undetectable at the system scale, so there's nothing abnormal there. Example : 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0 HAProxy is a TCP proxy, not a router. It deals with established connections that have been validated by the kernel, and not with packets of any form nor with sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence may prevent it from binding a port. It relies on the system to accept incoming connections and to initiate outgoing connections. An immediate effect of this is that there is no relation between packets observed on the two sides of a forwarded connection, which can be of different size, numbers and even family. Since a connection may only be accepted from a socket in LISTEN state, all the sockets it is listening to are necessarily visible using the "netstat" utility to show listening sockets. Example : # netstat -ltnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy 3. Starting HAProxy ------------------- HAProxy is started by invoking the "haproxy" program with a number of arguments passed on the command line. The actual syntax is : $ haproxy []* where []* is any number of options. An option always starts with '-' followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one if the "global" section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is : -- * : all the arguments following "--" are paths to configuration file/directory to be loaded and processed in the declaration order. It is mostly useful when relying on the shell to load many files that are numerically ordered. See also "-f". The difference between "--" and "-f" is that one "-f" must be placed before each file name, while a single "--" is needed before all file names. Both options can be used together, the command line ordering still applies. When more than one file is specified, each file must start on a section boundary, so the first keyword of each file must be one of "global", "defaults", "peers", "listen", "frontend", "backend", and so on. A file cannot contain just a server list for example. -f : adds to the list of configuration files to be loaded. If is a directory, all the files (and only files) it contains are added in lexical order (using LC_COLLATE=C) to the list of configuration files to be loaded ; only files with ".cfg" extension are added, only non hidden files (not prefixed with ".") are added. Configuration files are loaded and processed in their declaration order. This option may be specified multiple times to load multiple files. See also "--". The difference between "--" and "-f" is that one "-f" must be placed before each file name, while a single "--" is needed before all file names. Both options can be used together, the command line ordering still applies. When more than one file is specified, each file must start on a section boundary, so the first keyword of each file must be one of "global", "defaults", "peers", "listen", "frontend", "backend", and so on. A file cannot contain just a server list for example. -C : changes to directory before loading configuration files. This is useful when using relative paths. Warning when using wildcards after "--" which are in fact replaced by the shell before starting haproxy. -D : start as a daemon. The process detaches from the current terminal after forking, and errors are not reported anymore in the terminal. It is equivalent to the "daemon" keyword in the "global" section of the configuration. It is recommended to always force it in any init script so that a faulty configuration doesn't prevent the system from booting. -L : change the local peer name to , which defaults to the local hostname. This is used only with peers replication. -N : sets the default per-proxy maxconn to instead of the builtin default value (usually 2000). Only useful for debugging. -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or "quiet". -W : master-worker mode. It is equivalent to the "master-worker" keyword in the "global" section of the configuration. This mode will launch a "master" which will monitor the "workers". Using this mode, you can reload HAProxy directly by sending a SIGUSR2 signal to the master. The master-worker mode is compatible either with the foreground or daemon mode. It is recommended to use this mode with multiprocess and systemd. -Ws : master-worker mode with support of `notify` type of systemd service. This option is only available when HAProxy was built with `USE_SYSTEMD` build option enabled. -c : only performs a check of the configuration files and exits before trying to bind. The exit status is zero if everything is OK, or non-zero if an error is encountered. -d : enable debug mode. This disables daemon mode, forces the process to stay in foreground and to show incoming and outgoing events. It is equivalent to the "global" section's "debug" keyword. It must never be used in an init script. -dG : disable use of getaddrinfo() to resolve host names into addresses. It can be used when suspecting that getaddrinfo() doesn't work as expected. This option was made available because many bogus implementations of getaddrinfo() exist on various systems and cause anomalies that are difficult to troubleshoot. -dM[] : forces memory poisoning, which means that each and every memory region allocated with malloc() or pool_alloc() will be filled with before being passed to the caller. When is not specified, it defaults to 0x50 ('P'). While this slightly slows down operations, it is useful to reliably trigger issues resulting from missing initializations in the code that cause random crashes. Note that -dM0 has the effect of turning any malloc() into a calloc(). In any case if a bug appears or disappears when using this option it means there is a bug in haproxy, so please report it. -dS : disable use of the splice() system call. It is equivalent to the "global" section's "nosplice" keyword. This may be used when splice() is suspected to behave improperly or to cause performance issues, or when using strace to see the forwarded data (which do not appear when using splice()). -dV : disable SSL verify on the server side. It is equivalent to having "ssl-server-verify none" in the "global" section. This is useful when trying to reproduce production issues out of the production environment. Never use this in an init script as it degrades SSL security to the servers. -db : disable background mode and multi-process mode. The process remains in foreground. It is mainly used during development or during small tests, as Ctrl-C is enough to stop the process. Never use it in an init script. -de : disable the use of the "epoll" poller. It is equivalent to the "global" section's keyword "noepoll". It is mostly useful when suspecting a bug related to this poller. On systems supporting epoll, the fallback will generally be the "poll" poller. -dk : disable the use of the "kqueue" poller. It is equivalent to the "global" section's keyword "nokqueue". It is mostly useful when suspecting a bug related to this poller. On systems supporting kqueue, the fallback will generally be the "poll" poller. -dp : disable the use of the "poll" poller. It is equivalent to the "global" section's keyword "nopoll". It is mostly useful when suspecting a bug related to this poller. On systems supporting poll, the fallback will generally be the "select" poller, which cannot be disabled and is limited to 1024 file descriptors. -dr : ignore server address resolution failures. It is very common when validating a configuration out of production not to have access to the same resolvers and to fail on server address resolution, making it difficult to test a configuration. This option simply appends the "none" method to the list of address resolution methods for all servers, ensuring that even if the libc fails to resolve an address, the startup sequence is not interrupted. -m : limit the total allocatable memory to megabytes across all processes. This may cause some connection refusals or some slowdowns depending on the amount of memory needed for normal operations. This is mostly used to force the processes to work in a constrained resource usage scenario. It is important to note that the memory is not shared between processes, so in a multi-process scenario, this value is first divided by global.nbproc before forking. -n : limits the per-process connection limit to . This is equivalent to the global section's keyword "maxconn". It has precedence over this keyword. This may be used to quickly force lower limits to avoid a service outage on systems where resource limits are too low. -p : write all processes' pids into during startup. This is equivalent to the "global" section's keyword "pidfile". The file is opened before entering the chroot jail, and after doing the chdir() implied by "-C". Each pid appears on its own line. -q : set "quiet" mode. This disables some messages during the configuration parsing and during startup. It can be used in combination with "-c" to just check if a configuration file is valid or not. -sf * : send the "finish" signal (SIGUSR1) to older processes after boot completion to ask them to finish what they are doing and to leave. is a list of pids to signal (one per argument). The list ends on any option starting with a "-". It is not a problem if the list of pids is empty, so that it can be built on the fly based on the result of a command like "pidof" or "pgrep". -st * : send the "terminate" signal (SIGTERM) to older processes after boot completion to terminate them immediately without finishing what they were doing. is a list of pids to signal (one per argument). The list is ends on any option starting with a "-". It is not a problem if the list of pids is empty, so that it can be built on the fly based on the result of a command like "pidof" or "pgrep". -v : report the version and build date. -vv : display the version, build options, libraries versions and usable pollers. This output is systematically requested when filing a bug report. -x : connect to the specified socket and try to retrieve any listening sockets from the old process, and use them instead of trying to bind new ones. This is useful to avoid missing any new connection when reloading the configuration on Linux. The capability must be enable on the stats socket using "expose-fd listeners" in your configuration. A safe way to start HAProxy from an init file consists in forcing the daemon mode, storing existing pids to a pid file and using this pid file to notify older processes to finish before leaving : haproxy -f /etc/haproxy.cfg \ -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) When the configuration is split into a few specific files (eg: tcp vs http), it is recommended to use the "-f" option : haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \ -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \ -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \ -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) When an unknown number of files is expected, such as customer-specific files, it is recommended to assign them a name starting with a fixed-size sequence number and to use "--" to load them, possibly after loading some defaults : haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \ -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \ -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \ -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \ -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/* Sometimes a failure to start may happen for whatever reason. Then it is important to verify if the version of HAProxy you are invoking is the expected version and if it supports the features you are expecting (eg: SSL, PCRE, compression, Lua, etc). This can be verified using "haproxy -vv". Some important information such as certain build options, the target system and the versions of the libraries being used are reported there. It is also what you will systematically be asked for when posting a bug report : $ haproxy -vv HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08 Copyright 2000-2015 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \ -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.6 Compression algorithms supported : identity("identity"), deflate("deflate"), \ raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.12 2011-01-15 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with Lua version : Lua 5.3.1 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. The relevant information that many non-developer users can verify here are : - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit ID "a088d3" which is the 4th one after after official version "1.6-dev7". Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in fact "1.6-dev7". This is the 7th development version of what will become version 1.6 in the future. A development version not suitable for use in production (unless you know exactly what you are doing). A stable version will show as a 3-numbers version, such as "1.5.14-16f863", indicating the 14th level of fix on top of version 1.5. This is a production-ready version. - the release date : 2015/10/08. It is represented in the universal year/month/day format. Here this means August 8th, 2015. Given that stable releases are issued every few months (1-2 months at the beginning, sometimes 6 months once the product becomes very stable), if you're seeing an old date here, it means you're probably affected by a number of bugs or security issues that have since been fixed and that it might be worth checking on the official site. - build options : they are relevant to people who build their packages themselves, they can explain why things are not behaving as expected. For example the development version above was built for Linux 2.6.28 or later, targeting a generic CPU (no CPU-specific optimizations), and lacks any code optimization (-O0) so it will perform poorly in terms of performance. - libraries versions : zlib version is reported as found in the library itself. In general zlib is considered a very stable product and upgrades are almost never needed. OpenSSL reports two versions, the version used at build time and the one being used, as found on the system. These ones may differ by the last letter but never by the numbers. The build date is also reported because most OpenSSL bugs are security issues and need to be taken seriously, so this library absolutely needs to be kept up to date. Seeing a 4-months old version here is highly suspicious and indeed an update was missed. PCRE provides very fast regular expressions and is highly recommended. Certain of its extensions such as JIT are not present in all versions and still young so some people prefer not to build with them, which is why the build status is reported as well. Regarding the Lua scripting language, HAProxy expects version 5.3 which is very young since it was released a little time before HAProxy 1.6. It is important to check on the Lua web site if some fixes are proposed for this branch. - Available polling systems will affect the process's scalability when dealing with more than about one thousand of concurrent connections. These ones are only available when the correct system was indicated in the TARGET variable during the build. The "epoll" mechanism is highly recommended on Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them will result in poll() or even select() being used, causing a high CPU usage when dealing with a lot of connections. 4. Stopping and restarting HAProxy ---------------------------------- HAProxy supports a graceful and a hard stop. The hard stop is simple, when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed. The graceful stop is triggered when the SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding from listening ports, but continue to process existing connections until they close. Once the last connection is closed, the process leaves. The hard stop method is used for the "stop" or "restart" actions of the service management script. The graceful stop is used for the "reload" action which tries to seamlessly reload a new configuration in a new process. Both of these signals may be sent by the new haproxy process itself during a reload or restart, so that they are sent at the latest possible moment and only if absolutely required. This is what is performed by the "-st" (hard) and "-sf" (graceful) options respectively. In master-worker mode, it is not needed to start a new haproxy process in order to reload the configuration. The master process reacts to the SIGUSR2 signal by reexecuting itself with the -sf parameter followed by the PIDs of the workers. The master will then parse the configuration file and fork new workers. To understand better how these signals are used, it is important to understand the whole restart mechanism. First, an existing haproxy process is running. The administrator uses a system specific command such as "/etc/init.d/haproxy reload" to indicate he wants to take the new configuration file into effect. What happens then is the following. First, the service script (/etc/init.d/haproxy or equivalent) will verify that the configuration file parses correctly using "haproxy -c". After that it will try to start haproxy with this configuration file, using "-st" or "-sf". Then HAProxy tries to bind to all listening ports. If some fatal errors happen (eg: address not present on the system, permission denied), the process quits with an error. If a socket binding fails because a port is already in use, then the process will first send a SIGTTOU signal to all the pids specified in the "-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs all existing haproxy processes to temporarily stop listening to their ports so that the new process can try to bind again. During this time, the old process continues to process existing connections. If the binding still fails (because for example a port is shared with another daemon), then the new process sends a SIGTTIN signal to the old processes to instruct them to resume operations just as if nothing happened. The old processes will then restart listening to the ports and continue to accept connections. Not that this mechanism is system dependent and some operating systems may not support it in multi-process mode. If the new process manages to bind correctly to all ports, then it sends either the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case of "-sf") to all processes to notify them that it is now in charge of operations and that the old processes will have to leave, either immediately or once they have finished their job. It is important to note that during this timeframe, there are two small windows of a few milliseconds each where it is possible that a few connection failures will be noticed during high loads. Typically observed failure rates are around 1 failure during a reload operation every 10000 new connections per second, which means that a heavily loaded site running at 30000 new connections per second may see about 3 failed connection upon every reload. The two situations where this happens are : - if the new process fails to bind due to the presence of the old process, it will first have to go through the SIGTTOU+SIGTTIN sequence, which typically lasts about one millisecond for a few tens of frontends, and during which some ports will not be bound to the old process and not yet bound to the new one. HAProxy works around this on systems that support the SO_REUSEPORT socket options, as it allows the new process to bind without first asking the old one to unbind. Most BSD systems have been supporting this almost forever. Linux has been supporting this in version 2.0 and dropped it around 2.2, but some patches were floating around by then. It was reintroduced in kernel 3.9, so if you are observing a connection failure rate above the one mentioned above, please ensure that your kernel is 3.9 or newer, or that relevant patches were backported to your kernel (less likely). - when the old processes close the listening ports, the kernel may not always redistribute any pending connection that was remaining in the socket's backlog. Under high loads, a SYN packet may happen just before the socket is closed, and will lead to an RST packet being sent to the client. In some critical environments where even one drop is not acceptable, these ones are sometimes dealt with using firewall rules to block SYN packets during the reload, forcing the client to retransmit. This is totally system-dependent, as some systems might be able to visit other listening queues and avoid this RST. A second case concerns the ACK from the client on a local socket that was in SYN_RECV state just before the close. This ACK will lead to an RST packet while the haproxy process is still not aware of it. This one is harder to get rid of, though the firewall filtering rules mentioned above will work well if applied one second or so before restarting the process. For the vast majority of users, such drops will never ever happen since they don't have enough load to trigger the race conditions. And for most high traffic users, the failure rate is still fairly within the noise margin provided that at least SO_REUSEPORT is properly supported on their systems. 5. File-descriptor limitations ------------------------------ In order to ensure that all incoming connections will successfully be served, HAProxy computes at load time the total number of file descriptors that will be needed during the process's life. A regular Unix process is generally granted 1024 file descriptors by default, and a privileged process can raise this limit itself. This is one reason for starting HAProxy as root and letting it adjust the limit. The default limit of 1024 file descriptors roughly allow about 500 concurrent connections to be processed. The computation is based on the global maxconn parameter which limits the total number of connections per process, the number of listeners, the number of servers which have a health check enabled, the agent checks, the peers, the loggers and possibly a few other technical requirements. A simple rough estimate of this number consists in simply doubling the maxconn value and adding a few tens to get the approximate number of file descriptors needed. Originally HAProxy did not know how to compute this value, and it was necessary to pass the value using the "ulimit-n" setting in the global section. This explains why even today a lot of configurations are seen with this setting present. Unfortunately it was often miscalculated resulting in connection failures when approaching maxconn instead of throttling incoming connection while waiting for the needed resources. For this reason it is important to remove any vestigial "ulimit-n" setting that can remain from very old versions. Raising the number of file descriptors to accept even moderate loads is mandatory but comes with some OS-specific adjustments. First, the select() polling system is limited to 1024 file descriptors. In fact on Linux it used to be capable of handling more but since certain OS ship with excessively restrictive SELinux policies forbidding the use of select() with more than 1024 file descriptors, HAProxy now refuses to start in this case in order to avoid any issue at run time. On all supported operating systems, poll() is available and will not suffer from this limitation. It is automatically picked so there is nothing to do to get a working configuration. But poll's becomes very slow when the number of file descriptors increases. While HAProxy does its best to limit this performance impact (eg: via the use of the internal file descriptor cache and batched processing), a good rule of thumb is that using poll() with more than a thousand concurrent connections will use a lot of CPU. For Linux systems base on kernels 2.6 and above, the epoll() system call will be used. It's a much more scalable mechanism relying on callbacks in the kernel that guarantee a constant wake up time regardless of the number of registered monitored file descriptors. It is automatically used where detected, provided that HAProxy had been built for one of the Linux flavors. Its presence and support can be verified using "haproxy -vv". For BSD systems which support it, kqueue() is available as an alternative. It is much faster than poll() and even slightly faster than epoll() thanks to its batched handling of changes. At least FreeBSD and OpenBSD support it. Just like with Linux's epoll(), its support and availability are reported in the output of "haproxy -vv". Having a good poller is one thing, but it is mandatory that the process can reach the limits. When HAProxy starts, it immediately sets the new process's file descriptor limits and verifies if it succeeds. In case of failure, it reports it before forking so that the administrator can see the problem. As long as the process is started by as root, there should be no reason for this setting to fail. However, it can fail if the process is started by an unprivileged user. If there is a compelling reason for *not* starting haproxy as root (eg: started by end users, or by a per-application account), then the file descriptor limit can be raised by the system administrator for this specific user. The effectiveness of the setting can be verified by issuing "ulimit -n" from the user's command line. It should reflect the new limit. Warning: when an unprivileged user's limits are changed in this user's account, it is fairly common that these values are only considered when the user logs in and not at all in some scripts run at system boot time nor in crontabs. This is totally dependent on the operating system, keep in mind to check "ulimit -n" before starting haproxy when running this way. The general advice is never to start haproxy as an unprivileged user for production purposes. Another good reason is that it prevents haproxy from enabling some security protections. Once it is certain that the system will allow the haproxy process to use the requested number of file descriptors, two new system-specific limits may be encountered. The first one is the system-wide file descriptor limit, which is the total number of file descriptors opened on the system, covering all processes. When this limit is reached, accept() or socket() will typically return ENFILE. The second one is the per-process hard limit on the number of file descriptors, it prevents setrlimit() from being set higher. Both are very dependent on the operating system. On Linux, the system limit is set at boot based on the amount of memory. It can be changed with the "fs.file-max" sysctl. And the per-process hard limit is set to 1048576 by default, but it can be changed using the "fs.nr_open" sysctl. File descriptor limitations may be observed on a running process when they are set too low. The strace utility will report that accept() and socket() return "-1 EMFILE" when the process's limits have been reached. In this case, simply raising the "ulimit-n" value (or removing it) will solve the problem. If these system calls return "-1 ENFILE" then it means that the kernel's limits have been reached and that something must be done on a system-wide parameter. These trouble must absolutely be addressed, as they result in high CPU usage (when accept() fails) and failed connections that are generally visible to the user. One solution also consists in lowering the global maxconn value to enforce serialization, and possibly to disable HTTP keep-alive to force connections to be released and reused faster. 6. Memory management -------------------- HAProxy uses a simple and fast pool-based memory management. Since it relies on a small number of different object types, it's much more efficient to pick new objects from a pool which already contains objects of the appropriate size than to call malloc() for each different size. The pools are organized as a stack or LIFO, so that newly allocated objects are taken from recently released objects still hot in the CPU caches. Pools of similar sizes are merged together, in order to limit memory fragmentation. By default, since the focus is set on performance, each released object is put back into the pool it came from, and allocated objects are never freed since they are expected to be reused very soon. On the CLI, it is possible to check how memory is being used in pools thanks to the "show pools" command : > show pools Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED] - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED] - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED] - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED] - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED] - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED] Total: 11 pools, 26608 bytes allocated, 18544 used. The pool name is only indicative, it's the name of the first object type using this pool. The size in parenthesis is the object size for objects in this pool. Object sizes are always rounded up to the closest multiple of 16 bytes. The number of objects currently allocated and the equivalent number of bytes is reported so that it is easy to know which pool is responsible for the highest memory usage. The number of objects currently in use is reported as well in the "used" field. The difference between "allocated" and "used" corresponds to the objects that have been freed and are available for immediate use. It is possible to limit the amount of memory allocated per process using the "-m" command line option, followed by a number of megabytes. It covers all of the process's addressable space, so that includes memory used by some libraries as well as the stack, but it is a reliable limit when building a resource constrained system. It works the same way as "ulimit -v" on systems which have it, or "ulimit -d" for the other ones. If a memory allocation fails due to the memory limit being reached or because the system doesn't have any enough memory, then haproxy will first start to free all available objects from all pools before attempting to allocate memory again. This mechanism of releasing unused memory can be triggered by sending the signal SIGQUIT to the haproxy process. When doing so, the pools state prior to the flush will also be reported to stderr when the process runs in foreground. During a reload operation, the process switched to the graceful stop state also automatically performs some flushes after releasing any connection so that all possible memory is released to save it for the new process. 7. CPU usage ------------ HAProxy normally spends most of its time in the system and a smaller part in userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end connection setups and closes per second at 100% CPU on a single core. When one core is saturated, typical figures are : - 95% system, 5% user for long TCP connections or large HTTP objects - 85% system and 15% user for short TCP connections or small HTTP objects in close mode - 70% system and 30% user for small HTTP objects in keep-alive mode The amount of rules processing and regular expressions will increase the user land part. The presence of firewall rules, connection tracking, complex routing tables in the system will instead increase the system part. On most systems, the CPU time observed during network transfers can be cut in 4 parts : - the interrupt part, which concerns all the processing performed upon I/O receipt, before the target process is even known. Typically Rx packets are accounted for in interrupt. On some systems such as Linux where interrupt processing may be deferred to a dedicated thread, it can appear as softirq, and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of this load is generally defined by the hardware settings, though in the case of softirq it is often possible to remap the processing to another CPU. This interrupt part will often be perceived as parasitic since it's not associated with any process, but it actually is some processing being done to prepare the work for the process. - the system part, which concerns all the processing done using kernel code called from userland. System calls are accounted as system for example. All synchronously delivered Tx packets will be accounted for as system time. If some packets have to be deferred due to queues filling up, they may then be processed in interrupt context later (eg: upon receipt of an ACK opening a TCP window). - the user part, which exclusively runs application code in userland. HAProxy runs exclusively in this part, though it makes heavy use of system calls. Rules processing, regular expressions, compression, encryption all add to the user portion of CPU consumption. - the idle part, which is what the CPU does when there is nothing to do. For example HAProxy waits for an incoming connection, or waits for some data to leave, meaning the system is waiting for an ACK from the client to push these data. In practice regarding HAProxy's activity, it is in general reasonably accurate (but totally inexact) to consider that interrupt/softirq are caused by Rx processing in kernel drivers, that user-land is caused by layer 7 processing in HAProxy, and that system time is caused by network processing on the Tx path. Since HAProxy runs around an event loop, it waits for new events using poll() (or any alternative) and processes all these events as fast as possible before going back to poll() waiting for new events. It measures the time spent waiting in poll() compared to the time spent doing processing events. The ratio of polling time vs total time is called the "idle" time, it's the amount of time spent waiting for something to happen. This ratio is reported in the stats page on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means the load is extremely low. When it's close to 0%, it means that there is constantly some activity. While it cannot be very accurate on an overloaded system due to other processes possibly preempting the CPU from the haproxy process, it still provides a good estimate about how HAProxy considers it is working : if the load is low and the idle ratio is low as well, it may indicate that HAProxy has a lot of work to do, possibly due to very expensive rules that have to be processed. Conversely, if HAProxy indicates the idle is close to 100% while things are slow, it means that it cannot do anything to speed things up because it is already waiting for incoming data to process. In the example below, haproxy is completely idle : $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle Idle_pct: 100 When the idle ratio starts to become very low, it is important to tune the system and place processes and interrupts correctly to save the most possible CPU resources for all tasks. If a firewall is present, it may be worth trying to disable it or to tune it to ensure it is not responsible for a large part of the performance limitation. It's worth noting that unloading a stateful firewall generally reduces both the amount of interrupt/softirq and of system usage since such firewalls act both on the Rx and the Tx paths. On Linux, unloading the nf_conntrack and ip_conntrack modules will show whether there is anything to gain. If so, then the module runs with default settings and you'll have to figure how to tune it for better performance. In general this consists in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will disable the "pf" firewall and its stateful engine at the same time. If it is observed that a lot of time is spent in interrupt/softirq, it is important to ensure that they don't run on the same CPU. Most systems tend to pin the tasks on the CPU where they receive the network traffic because for certain workloads it improves things. But with heavily network-bound workloads it is the opposite as the haproxy process will have to fight against its kernel counterpart. Pinning haproxy to one CPU core and the interrupts to another one, all sharing the same L3 cache tends to sensibly increase network performance because in practice the amount of work for haproxy and the network stack are quite close, so they can almost fill an entire CPU each. On Linux this is done using taskset (for haproxy) or using cpu-map (from the haproxy config), and the interrupts are assigned under /proc/irq. Many network interfaces support multiple queues and multiple interrupts. In general it helps to spread them across a small number of CPU cores provided they all share the same L3 cache. Please always stop irq_balance which always does the worst possible thing on such workloads. For CPU-bound workloads consisting in a lot of SSL traffic or a lot of compression, it may be worth using multiple processes dedicated to certain tasks, though there is no universal rule here and experimentation will have to be performed. In order to increase the CPU capacity, it is possible to make HAProxy run as several processes, using the "nbproc" directive in the global section. There are some limitations though : - health checks are run per process, so the target servers will get as many checks as there are running processes ; - maxconn values and queues are per-process so the correct value must be set to avoid overloading the servers ; - outgoing connections should avoid using port ranges to avoid conflicts - stick-tables are per process and are not shared between processes ; - each peers section may only run on a single process at a time ; - the CLI operations will only act on a single process at a time. With this in mind, it appears that the easiest setup often consists in having one first layer running on multiple processes and in charge for the heavy processing, passing the traffic to a second layer running in a single process. This mechanism is suited to SSL and compression which are the two CPU-heavy features. Instances can easily be chained over UNIX sockets (which are cheaper than TCP sockets and which do not waste ports), and the proxy protocol which is useful to pass client information to the next stage. When doing so, it is generally a good idea to bind all the single-process tasks to process number 1 and extra tasks to next processes, as this will make it easier to generate similar configurations for different machines. On Linux versions 3.9 and above, running HAProxy in multi-process mode is much more efficient when each process uses a distinct listening socket on the same IP:port ; this will make the kernel evenly distribute the load across all processes instead of waking them all up. Please check the "process" option of the "bind" keyword lines in the configuration manual for more information. 8. Logging ---------- For logging, HAProxy always relies on a syslog server since it does not perform any file-system access. The standard way of using it is to send logs over UDP to the log server (by default on port 514). Very commonly this is configured to 127.0.0.1 where the local syslog daemon is running, but it's also used over the network to log to a central server. The central server provides additional benefits especially in active-active scenarios where it is desirable to keep the logs merged in arrival order. HAProxy may also make use of a UNIX socket to send its logs to the local syslog daemon, but it is not recommended at all, because if the syslog server is restarted while haproxy runs, the socket will be replaced and new logs will be lost. Since HAProxy will be isolated inside a chroot jail, it will not have the ability to reconnect to the new socket. It has also been observed in field that the log buffers in use on UNIX sockets are very small and lead to lost messages even at very light loads. But this can be fine for testing however. It is recommended to add the following directive to the "global" section to make HAProxy log to the local daemon using facility "local0" : log 127.0.0.1:514 local0 and then to add the following one to each "defaults" section or to each frontend and backend section : log global This way, all logs will be centralized through the global definition of where the log server is. Some syslog daemons do not listen to UDP traffic by default, so depending on the daemon being used, the syntax to enable this will vary : - on sysklogd, you need to pass argument "-r" on the daemon's command line so that it listens to a UDP socket for "remote" logs ; note that there is no way to limit it to address 127.0.0.1 so it will also receive logs from remote systems ; - on rsyslogd, the following lines must be added to the configuration file : $ModLoad imudp $UDPServerAddress * $UDPServerRun 514 - on syslog-ng, a new source can be created the following way, it then needs to be added as a valid source in one of the "log" directives : source s_udp { udp(ip(127.0.0.1) port(514)); }; Please consult your syslog daemon's manual for more information. If no logs are seen in the system's log files, please consider the following tests : - restart haproxy. Each frontend and backend logs one line indicating it's starting. If these logs are received, it means logs are working. - run "strace -tt -s100 -etrace=sendmsg -p " and perform some activity that you expect to be logged. You should see the log messages being sent using sendmsg() there. If they don't appear, restart using strace on top of haproxy. If you still see no logs, it definitely means that something is wrong in your configuration. - run tcpdump to watch for port 514, for example on the loopback interface if the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the packets are seen there, it's the proof they're sent then the syslogd daemon needs to be troubleshooted. While traffic logs are sent from the frontends (where the incoming connections are accepted), backends also need to be able to send logs in order to report a server state change consecutive to a health check. Please consult HAProxy's configuration manual for more information regarding all possible log settings. It is convenient to chose a facility that is not used by other daemons. HAProxy examples often suggest "local0" for traffic logs and "local1" for admin logs because they're never seen in field. A single facility would be enough as well. Having separate logs is convenient for log analysis, but it's also important to remember that logs may sometimes convey confidential information, and as such they must not be mixed with other logs that may accidentally be handed out to unauthorized people. For in-field troubleshooting without impacting the server's capacity too much, it is recommended to make use of the "halog" utility provided with HAProxy. This is sort of a grep-like utility designed to process HAProxy log files at a very fast data rate. Typical figures range between 1 and 2 GB of logs per second. It is capable of extracting only certain logs (eg: search for some classes of HTTP status codes, connection termination status, search by response time ranges, look for errors only), count lines, limit the output to a number of lines, and perform some more advanced statistics such as sorting servers by response time or error counts, sorting URLs by time or count, sorting client addresses by access count, and so on. It is pretty convenient to quickly spot anomalies such as a bot looping on the site, and block them. 9. Statistics and monitoring ---------------------------- It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket. 9.1. CSV format --------------- The statistics may be consulted either from the unix socket or from the HTTP page. Both means provide a CSV format whose fields follow. The first line begins with a sharp ('#') and has one word per comma-delimited field which represents the title of the column. All other lines starting at the second one use a classical CSV format using a comma as the delimiter, and the double quote ('"') as an optional text delimiter, but only if the enclosed text is ambiguous (if it contains a quote or a comma). The double-quote character ('"') in the text is doubled ('""'), which is the format that most tools recognize. Please do not insert any column before these ones in order not to break tools which use hard-coded column positions. In brackets after each field name are the types which may have a value for that field. The types are L (Listeners), F (Frontends), B (Backends), and S (Servers). 0. pxname [LFBS]: proxy name 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend, any name for server/listener) 2. qcur [..BS]: current queued requests. For the backend this reports the number queued without a server assigned. 3. qmax [..BS]: max value of qcur 4. scur [LFBS]: current sessions 5. smax [LFBS]: max sessions 6. slim [LFBS]: configured session limit 7. stot [LFBS]: cumulative number of sessions 8. bin [LFBS]: bytes in 9. bout [LFBS]: bytes out 10. dreq [LFB.]: requests denied because of security concerns. - For tcp this is because of a matched tcp-request content rule. - For http this is because of a matched http-request or tarpit rule. 11. dresp [LFBS]: responses denied because of security concerns. - For http this is because of a matched http-request rule, or "option checkcache". 12. ereq [LF..]: request errors. Some of the possible causes are: - early termination from the client, before the request has been sent. - read error from the client - client timeout - client closed connection - various bad requests from the client. - request was tarpitted. 13. econ [..BS]: number of requests that encountered an error trying to connect to a backend server. The backend stat is the sum of the stat for all servers of that backend, plus any connection errors not associated with a particular server (such as the backend having no active servers). 14. eresp [..BS]: response errors. srv_abrt will be counted here also. Some other errors are: - write error on the client socket (won't be counted for the server stat) - failure applying filters to the response. 15. wretr [..BS]: number of times a connection to a server was retried. 16. wredis [..BS]: number of times a request was redispatched to another server. The server value counts the number of times that server was switched away from. 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...) 18. weight [..BS]: total weight (backend), server weight (server) 19. act [..BS]: number of active servers (backend), server is active (server) 20. bck [..BS]: number of backup servers (backend), server is backup (server) 21. chkfail [...S]: number of failed checks. (Only counts checks failed when the server is up.) 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts transitions to the whole backend being down, rather than the sum of the counters for each server. 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition 24. downtime [..BS]: total downtime (in seconds). The value for the backend is the downtime for the whole backend, not the sum of the server downtime. 25. qlimit [...S]: configured maxqueue for the server, or nothing in the value is 0 (default, meaning no limit) 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...) 27. iid [LFBS]: unique proxy id 28. sid [L..S]: server id (unique inside a proxy) 29. throttle [...S]: current throttle percentage for the server, when slowstart is active, or no value if not in slowstart. 30. lbtot [..BS]: total number of times a server was selected, either for new sessions, or when re-dispatching. The server counter is the number of times that server was selected. 31. tracked [...S]: id of proxy/server if tracking is enabled. 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener) 33. rate [.FBS]: number of sessions per second over last elapsed second 34. rate_lim [.F..]: configured limit on new sessions per second 35. rate_max [.FBS]: max number of new sessions per second 36. check_status [...S]: status of last health check, one of: UNK -> unknown INI -> initializing SOCKERR -> socket error L4OK -> check passed on layer 4, no upper layers testing enabled L4TOUT -> layer 1-4 timeout L4CON -> layer 1-4 connection problem, for example "Connection refused" (tcp rst) or "No route to host" (icmp) L6OK -> check passed on layer 6 L6TOUT -> layer 6 (SSL) timeout L6RSP -> layer 6 invalid response - protocol error L7OK -> check passed on layer 7 L7OKC -> check conditionally passed on layer 7, for example 404 with disable-on-404 L7TOUT -> layer 7 (HTTP/SMTP) timeout L7RSP -> layer 7 invalid response - protocol error L7STS -> layer 7 response error, for example HTTP 5xx Notice: If a check is currently running, the last known status will be reported, prefixed with "* ". e. g. "* L7OK". 37. check_code [...S]: layer5-7 code, if available 38. check_duration [...S]: time in ms took to finish last health check 39. hrsp_1xx [.FBS]: http responses with 1xx code 40. hrsp_2xx [.FBS]: http responses with 2xx code 41. hrsp_3xx [.FBS]: http responses with 3xx code 42. hrsp_4xx [.FBS]: http responses with 4xx code 43. hrsp_5xx [.FBS]: http responses with 5xx code 44. hrsp_other [.FBS]: http responses with other codes (protocol error) 45. hanafail [...S]: failed health checks details 46. req_rate [.F..]: HTTP requests per second over last elapsed second 47. req_rate_max [.F..]: max number of HTTP requests per second observed 48. req_tot [.FB.]: total number of HTTP requests received 49. cli_abrt [..BS]: number of data transfers aborted by the client 50. srv_abrt [..BS]: number of data transfers aborted by the server (inc. in eresp) 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor (CPU/BW limit) 54. comp_rsp [.FB.]: number of HTTP responses that were compressed 55. lastsess [..BS]: number of seconds since last session assigned to server/backend 56. last_chk [...S]: last health check contents or textual error 57. last_agt [...S]: last agent check contents or textual error 58. qtime [..BS]: the average queue time in ms over the 1024 last requests 59. ctime [..BS]: the average connect time in ms over the 1024 last requests 60. rtime [..BS]: the average response time in ms over the 1024 last requests (0 for TCP) 61. ttime [..BS]: the average total session time in ms over the 1024 last requests 62. agent_status [...S]: status of last agent check, one of: UNK -> unknown INI -> initializing SOCKERR -> socket error L4OK -> check passed on layer 4, no upper layers testing enabled L4TOUT -> layer 1-4 timeout L4CON -> layer 1-4 connection problem, for example "Connection refused" (tcp rst) or "No route to host" (icmp) L7OK -> agent reported "up" L7STS -> agent reported "fail", "stop", or "down" 63. agent_code [...S]: numeric code reported by agent if any (unused for now) 64. agent_duration [...S]: time in ms taken to finish last check 65. check_desc [...S]: short human-readable description of check_status 66. agent_desc [...S]: short human-readable description of agent_status 67. check_rise [...S]: server's "rise" parameter used by checks 68. check_fall [...S]: server's "fall" parameter used by checks 69. check_health [...S]: server's health check value between 0 and rise+fall-1 70. agent_rise [...S]: agent's "rise" parameter, normally 1 71. agent_fall [...S]: agent's "fall" parameter, normally 1 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address. 74: cookie [..BS]: server's cookie value or backend's cookie name 75: mode [LFBS]: proxy mode (tcp, http, health, unknown) 76: algo [..B.]: load balancing algorithm 77: conn_rate [.F..]: number of connections over the last elapsed second 78: conn_rate_max [.F..]: highest known conn_rate 79: conn_tot [.F..]: cumulative number of connections 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats) 81: dcon [LF..]: requests denied by "tcp-request connection" rules 82: dses [LF..]: requests denied by "tcp-request session" rules 9.2. Typed output format ------------------------ Both "show info" and "show stat" support a mode where each output value comes with its type and sufficient information to know how the value is supposed to be aggregated between processes and how it evolves. In all cases, the output consists in having a single value per line with all the information split into fields delimited by colons (':'). The first column designates the object or metric being dumped. Its format is specific to the command producing this output and will not be described in this section. Usually it will consist in a series of identifiers and field names. The second column contains 3 characters respectively indicating the origin, the nature and the scope of the value being reported. The first character (the origin) indicates where the value was extracted from. Possible characters are : M The value is a metric. It is valid at one instant any may change depending on its nature . S The value is a status. It represents a discrete value which by definition cannot be aggregated. It may be the status of a server ("UP" or "DOWN"), the PID of the process, etc. K The value is a sorting key. It represents an identifier which may be used to group some values together because it is unique among its class. All internal identifiers are keys. Some names can be listed as keys if they are unique (eg: a frontend name is unique). In general keys come from the configuration, even though some of them may automatically be assigned. For most purposes keys may be considered as equivalent to configuration. C The value comes from the configuration. Certain configuration values make sense on the output, for example a concurrent connection limit or a cookie name. By definition these values are the same in all processes started from the same configuration file. P The value comes from the product itself. There are very few such values, most common use is to report the product name, version and release date. These elements are also the same between all processes. The second character (the nature) indicates the nature of the information carried by the field in order to let an aggregator decide on what operation to use to aggregate multiple values. Possible characters are : A The value represents an age since a last event. This is a bit different from the duration in that an age is automatically computed based on the current date. A typical example is how long ago did the last session happen on a server. Ages are generally aggregated by taking the minimum value and do not need to be stored. a The value represents an already averaged value. The average response times and server weights are of this nature. Averages can typically be averaged between processes. C The value represents a cumulative counter. Such measures perpetually increase until they wrap around. Some monitoring protocols need to tell the difference between a counter and a gauge to report a different type. In general counters may simply be summed since they represent events or volumes. Examples of metrics of this nature are connection counts or byte counts. D The value represents a duration for a status. There are a few usages of this, most of them include the time taken by the last health check and the time a server has spent down. Durations are generally not summed, most of the time the maximum will be retained to compute an SLA. G The value represents a gauge. It's a measure at one instant. The memory usage or the current number of active connections are of this nature. Metrics of this type are typically summed during aggregation. L The value represents a limit (generally a configured one). By nature, limits are harder to aggregate since they are specific to the point where they were retrieved. In certain situations they may be summed or be kept separate. M The value represents a maximum. In general it will apply to a gauge and keep the highest known value. An example of such a metric could be the maximum amount of concurrent connections that was encountered in the product's life time. To correctly aggregate maxima, you are supposed to output a range going from the maximum of all maxima and the sum of all of them. There is indeed no way to know if they were encountered simultaneously or not. m The value represents a minimum. In general it will apply to a gauge and keep the lowest known value. An example of such a metric could be the minimum amount of free memory pools that was encountered in the product's life time. To correctly aggregate minima, you are supposed to output a range going from the minimum of all minima and the sum of all of them. There is indeed no way to know if they were encountered simultaneously or not. N The value represents a name, so it is a string. It is used to report proxy names, server names and cookie names. Names have configuration or keys as their origin and are supposed to be the same among all processes. O The value represents a free text output. Outputs from various commands, returns from health checks, node descriptions are of such nature. R The value represents an event rate. It's a measure at one instant. It is quite similar to a gauge except that the recipient knows that this measure moves slowly and may decide not to keep all values. An example of such a metric is the measured amount of connections per second. Metrics of this type are typically summed during aggregation. T The value represents a date or time. A field emitting the current date would be of this type. The method to aggregate such information is left as an implementation choice. For now no field uses this type. The third character (the scope) indicates what extent the value reflects. Some elements may be per process while others may be per configuration or per system. The distinction is important to know whether or not a single value should be kept during aggregation or if values have to be aggregated. The following characters are currently supported : C The value is valid for a whole cluster of nodes, which is the set of nodes communicating over the peers protocol. An example could be the amount of entries present in a stick table that is replicated with other peers. At the moment no metric use this scope. P The value is valid only for the process reporting it. Most metrics use this scope. S The value is valid for the whole service, which is the set of processes started together from the same configuration file. All metrics originating from the configuration use this scope. Some other metrics may use it as well for some shared resources (eg: shared SSL cache statistics). s The value is valid for the whole system, such as the system's hostname, current date or resource usage. At the moment this scope is not used by any metric. Consumers of these information will generally have enough of these 3 characters to determine how to accurately report aggregated information across multiple processes. After this column, the third column indicates the type of the field, among "s32" (signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to know the type before parsing the value in order to properly read it. For example a string containing only digits is still a string an not an integer (eg: an error code extracted by a check). Then the fourth column is the value itself, encoded according to its type. Strings are dumped as-is immediately after the colon without any leading space. If a string contains a colon, it will appear normally. This means that the output should not be exclusively split around colons or some check outputs or server addresses might be truncated. 9.3. Unix Socket commands ------------------------- The stats socket is not enabled by default. In order to enable it, it is necessary to add one line in the global section of the haproxy configuration. A second line is recommended to set a larger timeout, always appreciated when issuing commands by hand : global stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m It is also possible to add multiple instances of the stats socket by repeating the line, and make them listen to a TCP port instead of a UNIX socket. This is never done by default because this is dangerous, but can be handy in some situations : global stats socket /var/run/haproxy.sock mode 600 level admin stats socket ipv4@192.168.0.1:9999 level admin stats timeout 2m To access the socket, an external utility such as "socat" is required. Socat is a swiss-army knife to connect anything to anything. We use it to connect terminals to the socket, or a couple of stdin/stdout pipes to it for scripts. The two main syntaxes we'll use are the following : # socat /var/run/haproxy.sock stdio # socat /var/run/haproxy.sock readline The first one is used with scripts. It is possible to send the output of a script to haproxy, and pass haproxy's output to another script. That's useful for retrieving counters or attack traces for example. The second one is only useful for issuing commands by hand. It has the benefit that the terminal is handled by the readline library which supports line editing and history, which is very convenient when issuing repeated commands (eg: watch a counter). The socket supports two operation modes : - interactive - non-interactive The non-interactive mode is the default when socat connects to the socket. In this mode, a single line may be sent. It is processed as a whole, responses are sent back, and the connection closes after the end of the response. This is the mode that scripts and monitoring tools use. It is possible to send multiple commands in this mode, they need to be delimited by a semi-colon (';'). For example : # echo "show info;show stat;show table" | socat /var/run/haproxy stdio If a command needs to use a semi-colon or a backslash (eg: in a value), it must be preceded by a backslash ('\'). The interactive mode displays a prompt ('>') and waits for commands to be entered on the line, then processes them, and displays the prompt again to wait for a new command. This mode is entered via the "prompt" command which must be sent on the first line in non-interactive mode. The mode is a flip switch, if "prompt" is sent in interactive mode, it is disabled and the connection closes after processing the last command of the same line. For this reason, when debugging by hand, it's quite common to start with the "prompt" command : # socat /var/run/haproxy readline prompt > show info ... > Since multiple commands may be issued at once, haproxy uses the empty line as a delimiter to mark an end of output for each command, and takes care of ensuring that no command can emit an empty line on output. A script can thus easily parse the output even when multiple commands were pipelined on a single line. It is important to understand that when multiple haproxy processes are started on the same sockets, any process may pick up the request and will output its own stats. The list of commands currently supported on the stats socket is provided below. If an unknown command is sent, haproxy displays the usage message which reminds all supported commands. Some commands support a more complex syntax, generally it will explain what part of the command is invalid when this happens. Some commands require a higher level of privilege to work. If you do not have enough privilege, you will get an error "Permission denied". Please check the "level" option of the "bind" keyword lines in the configuration manual for more information. add acl Add an entry into the acl . is the # or the returned by "show acl". This command does not verify if the entry already exists. This command cannot be used if the reference is a file also used with a map. In this case, you must use the command "add map" in place of "add acl". add map Add an entry into the map to associate the value to the key . This command does not verify if the entry already exists. It is mainly used to fill a map after a clear operation. Note that if the reference is a file and is shared with a map, this map will contain also a new pattern entry. clear counters Clear the max values of the statistics counters in each proxy (frontend & backend) and in each server. The accumulated counters are not affected. The internal activity counters reported by "show activity" are also reset. This can be used to get clean counters after an incident, without having to restart nor to clear traffic counters. This command is restricted and can only be issued on sockets configured for levels "operator" or "admin". clear counters all Clear all statistics counters in each proxy (frontend & backend) and in each server. This has the same effect as restarting. This command is restricted and can only be issued on sockets configured for level "admin". clear acl Remove all entries from the acl . is the # or the returned by "show acl". Note that if the reference is a file and is shared with a map, this map will be also cleared. clear map Remove all entries from the map . is the # or the returned by "show map". Note that if the reference is a file and is shared with a acl, this acl will be also cleared. clear table [ data. ] | [ key ] Remove entries from the stick-table
. This is typically used to unblock some users complaining they have been abusively denied access to a service, but this can also be used to clear some stickiness entries matching a server that is going to be replaced (see "show table" below for details). Note that sometimes, removal of an entry will be refused because it is currently tracked by a session. Retrying a few seconds later after the session ends is usual enough. In the case where no options arguments are given all entries will be removed. When the "data." form is used entries matching a filter applied using the stored data (see "stick-table" in section 4.2) are removed. A stored data type must be specified in , and this data type must be stored in the table otherwise an error is reported. The data is compared according to with the 64-bit integer . Operators are the same as with the ACLs : - eq : match entries whose data is equal to this value - ne : match entries whose data is not equal to this value - le : match entries whose data is less than or equal to this value - ge : match entries whose data is greater than or equal to this value - lt : match entries whose data is less than this value - gt : match entries whose data is greater than this value When the key form is used the entry is removed. The key must be of the same type as the table, which currently is limited to IPv4, IPv6, integer and string. Example : $ echo "show table http_proxy" | socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:2 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \ bytes_out_rate(60000)=187 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1 $ echo "show table http_proxy" | socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:1 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1 $ echo "show table http_proxy" | socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:1 del acl [|#] Delete all the acl entries from the acl corresponding to the key . is the # or the returned by "show acl". If the is used, this command delete only the listed reference. The reference can be found with listing the content of the acl. Note that if the reference is a file and is shared with a map, the entry will be also deleted in the map. del map [|#] Delete all the map entries from the map corresponding to the key . is the # or the returned by "show map". If the is used, this command delete only the listed reference. The reference can be found with listing the content of the map. Note that if the reference is a file and is shared with a acl, the entry will be also deleted in the map. disable agent / Mark the auxiliary agent check as temporarily stopped. In the case where an agent check is being run as a auxiliary check, due to the agent-check parameter of a server directive, new checks are only initialized when the agent is in the enabled. Thus, disable agent will prevent any new agent checks from begin initiated until the agent re-enabled using enable agent. When an agent is disabled the processing of an auxiliary agent check that was initiated while the agent was set as enabled is as follows: All results that would alter the weight, specifically "drain" or a weight returned by the agent, are ignored. The processing of agent check is otherwise unchanged. The motivation for this feature is to allow the weight changing effects of the agent checks to be paused to allow the weight of a server to be configured using set weight without being overridden by the agent. This command is restricted and can only be issued on sockets configured for level "admin". disable dynamic-cookie backend Disable the generation of dynamic cookies fot the backend disable frontend Mark the frontend as temporarily stopped. This corresponds to the mode which is used during a soft restart : the frontend releases the port but can be enabled again if needed. This should be used with care as some non-Linux OSes are unable to enable it back. This is intended to be used in environments where stopping a proxy is not even imaginable but a misconfigured proxy must be fixed. That way it's possible to release the port and bind it into another process to restore operations. The frontend will appear with status "STOP" on the stats page. The frontend may be specified either by its name or by its numeric ID, prefixed with a sharp ('#'). This command is restricted and can only be issued on sockets configured for level "admin". disable health / Mark the primary health check as temporarily stopped. This will disable sending of health checks, and the last health check result will be ignored. The server will be in unchecked state and considered UP unless an auxiliary agent check forces it down. This command is restricted and can only be issued on sockets configured for level "admin". disable server / Mark the server DOWN for maintenance. In this mode, no more checks will be performed on the server until it leaves maintenance. If the server is tracked by other servers, those servers will be set to DOWN during the maintenance. In the statistics page, a server DOWN for maintenance will appear with a "MAINT" status, its tracking servers with the "MAINT(via)" one. Both the backend and the server may be specified either by their name or by their numeric ID, prefixed with a sharp ('#'). This command is restricted and can only be issued on sockets configured for level "admin". enable agent / Resume auxiliary agent check that was temporarily stopped. See "disable agent" for details of the effect of temporarily starting and stopping an auxiliary agent. This command is restricted and can only be issued on sockets configured for level "admin". enable dynamic-cookie backend Enable the generation of dynamic cookies for the backend . A secret key must also be provided. enable frontend Resume a frontend which was temporarily stopped. It is possible that some of the listening ports won't be able to bind anymore (eg: if another process took them since the 'disable frontend' operation). If this happens, an error is displayed. Some operating systems might not be able to resume a frontend which was disabled. The frontend may be specified either by its name or by its numeric ID, prefixed with a sharp ('#'). This command is restricted and can only be issued on sockets configured for level "admin". enable health / Resume a primary health check that was temporarily stopped. This will enable sending of health checks again. Please see "disable health" for details. This command is restricted and can only be issued on sockets configured for level "admin". enable server / If the server was previously marked as DOWN for maintenance, this marks the server UP and checks are re-enabled. Both the backend and the server may be specified either by their name or by their numeric ID, prefixed with a sharp ('#'). This command is restricted and can only be issued on sockets configured for level "admin". get map get acl Lookup the value in the map or in the ACL . or are the # or the returned by "show map" or "show acl". This command returns all the matching patterns associated with this map. This is useful for debugging maps and ACLs. The output format is composed by one line par matching type. Each line is composed by space-delimited series of words. The first two words are: : The match method applied. It can be "found", "bool", "int", "ip", "bin", "len", "str", "beg", "sub", "dir", "dom", "end" or "reg". : The result. Can be "match" or "no-match". The following words are returned only if the pattern matches an entry. : "tree" or "list". The internal lookup algorithm. : "case-insensitive" or "case-sensitive". The interpretation of the case. : match="". Return the matched pattern. It is useful with regular expressions. The two last word are used to show the returned value and its type. With the "acl" case, the pattern doesn't exist. return=nothing: No return because there are no "map". return="": The value returned in the string format. return=cannot-display: The value cannot be converted as string. type="": The type of the returned sample. get weight / Report the current weight and the initial weight of server in backend or an error if either doesn't exist. The initial weight is the one that appears in the configuration file. Both are normally equal unless the current weight has been changed. Both the backend and the server may be specified either by their name or by their numeric ID, prefixed with a sharp ('#'). help Print the list of known keywords and their basic usage. The same help screen is also displayed for unknown commands. prompt Toggle the prompt at the beginning of the line and enter or leave interactive mode. In interactive mode, the connection is not closed after a command completes. Instead, the prompt will appear again, indicating the user that the interpreter is waiting for a new command. The prompt consists in a right angle bracket followed by a space "> ". This mode is particularly convenient when one wants to periodically check information such as stats or errors. It is also a good idea to enter interactive mode before issuing a "help" command. quit Close the connection when in interactive mode. set dynamic-cookie-key backend Modify the secret key used to generate the dynamic persistent cookies. This will break the existing sessions. set map [|#] Modify the value corresponding to each key in a map . is the # or returned by "show map". If the is used in place of , only the entry pointed by is changed. The new value is . set maxconn frontend Dynamically change the specified frontend's maxconn setting. Any positive value is allowed including zero, but setting values larger than the global maxconn does not make much sense. If the limit is increased and connections were pending, they will immediately be accepted. If it is lowered to a value below the current number of connections, new connections acceptation will be delayed until the threshold is reached. The frontend might be specified by either its name or its numeric ID prefixed with a sharp ('#'). set maxconn server Dynamically change the specified server's maxconn setting. Any positive value is allowed including zero, but setting values larger than the global maxconn does not make much sense. set maxconn global Dynamically change the global maxconn setting within the range defined by the initial global maxconn setting. If it is increased and connections were pending, they will immediately be accepted. If it is lowered to a value below the current number of connections, new connections acceptation will be delayed until the threshold is reached. A value of zero restores the initial setting. set rate-limit connections global Change the process-wide connection rate limit, which is set by the global 'maxconnrate' setting. A value of zero disables the limitation. This limit applies to all frontends and the change has an immediate effect. The value is passed in number of connections per second. set rate-limit http-compression global Change the maximum input compression rate, which is set by the global 'maxcomprate' setting. A value of zero disables the limitation. The value is passed in number of kilobytes per second. The value is available in the "show info" on the line "CompressBpsRateLim" in bytes. set rate-limit sessions global Change the process-wide session rate limit, which is set by the global 'maxsessrate' setting. A value of zero disables the limitation. This limit applies to all frontends and the change has an immediate effect. The value is passed in number of sessions per second. set rate-limit ssl-sessions global Change the process-wide SSL session rate limit, which is set by the global 'maxsslrate' setting. A value of zero disables the limitation. This limit applies to all frontends and the change has an immediate effect. The value is passed in number of sessions per second sent to the SSL stack. It applies before the handshake in order to protect the stack against handshake abuses. set server / addr [port ] Replace the current IP address of a server by the one provided. Optionally, the port can be changed using the 'port' parameter. Note that changing the port also support switching from/to port mapping (notation with +X or -Y), only if a port is configured for the health check. set server / agent [ up | down ] Force a server's agent to a new state. This can be useful to immediately switch a server's state regardless of some slow agent checks for example. Note that the change is propagated to tracking servers if any. set server / agent-addr Change addr for servers agent checks. Allows to migrate agent-checks to another address at runtime. You can specify both IP and hostname, it will be resolved. set server / agent-send Change agent string sent to agent check target. Allows to update string while changing server address to keep those two matching. set server / health [ up | stopping | down ] Force a server's health to a new state. This can be useful to immediately switch a server's state regardless of some slow health checks for example. Note that the change is propagated to tracking servers if any. set server / check-port Change the port used for health checking to set server / state [ ready | drain | maint ] Force a server's administrative state to a new state. This can be useful to disable load balancing and/or any traffic to a server. Setting the state to "ready" puts the server in normal mode, and the command is the equivalent of the "enable server" command. Setting the state to "maint" disables any traffic to the server as well as any health checks. This is the equivalent of the "disable server" command. Setting the mode to "drain" only removes the server from load balancing but still allows it to be checked and to accept new persistent connections. Changes are propagated to tracking servers if any. set server / weight [%] Change a server's weight to the value passed in argument. This is the exact equivalent of the "set weight" command below. set server / fqdn Change a server's FQDN to the value passed in argument. This requires the internal run-time DNS resolver to be configured and enabled for this server. set severity-output [ none | number | string ] Change the severity output format of the stats socket connected to for the duration of the current session. set ssl ocsp-response This command is used to update an OCSP Response for a certificate (see "crt" on "bind" lines). Same controls are performed as during the initial loading of the response. The must be passed as a base64 encoded string of the DER encoded response from the OCSP server. This command is not supported with BoringSSL. Example: openssl ocsp -issuer issuer.pem -cert server.pem \ -host ocsp.issuer.com:80 -respout resp.der echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \ socat stdio /var/run/haproxy.stat set ssl tls-key Set the next TLS key for the listener to . This key becomes the ultimate key, while the penultimate one is used for encryption (others just decrypt). The oldest TLS key present is overwritten. is either a numeric # or returned by "show tls-keys". is a base64 encoded 48 bit TLS ticket key (ex. openssl rand -base64 48). set table
key [data. ]* Create or update a stick-table entry in the table. If the key is not present, an entry is inserted. See stick-table in section 4.2 to find all possible values for . The most likely use consists in dynamically entering entries for source IP addresses, with a flag in gpc0 to dynamically block an IP address or affect its quality of service. It is possible to pass multiple data_types in a single call. set timeout cli Change the CLI interface timeout for current connection. This can be useful during long debugging sessions where the user needs to constantly inspect some indicators without being disconnected. The delay is passed in seconds. set weight / [%] Change a server's weight to the value passed in argument. If the value ends with the '%' sign, then the new weight will be relative to the initially configured weight. Absolute weights are permitted between 0 and 256. Relative weights must be positive with the resulting absolute weight is capped at 256. Servers which are part of a farm running a static load-balancing algorithm have stricter limitations because the weight cannot change once set. Thus for these servers, the only accepted values are 0 and 100% (or 0 and the initial weight). Changes take effect immediately, though certain LB algorithms require a certain amount of requests to consider changes. A typical usage of this command is to disable a server during an update by setting its weight to zero, then to enable it again after the update by setting it back to 100%. This command is restricted and can only be issued on sockets configured for level "admin". Both the backend and the server may be specified either by their name or by their numeric ID, prefixed with a sharp ('#'). show acl [] Dump info about acl converters. Without argument, the list of all available acls is returned. If a is specified, its contents are dumped. if the # or . The dump format is the same than the map even for the sample value. The data returned are not a list of available ACL, but are the list of all patterns composing any ACL. Many of these patterns can be shared with maps. show backend Dump the list of backends available in the running process show cli sockets List CLI sockets. The output format is composed of 3 fields separated by spaces. The first field is the socket address, it can be a unix socket, a ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump. The second field describe the level of the socket: 'admin', 'user' or 'operator'. The last field list the processes on which the socket is bound, separated by commas, it can be numbers or 'all'. Example : $ echo 'show cli sockets' | socat stdio /tmp/sock1 # socket lvl processes /tmp/sock1 admin all 127.0.0.1:9999 user 2,3,4 127.0.0.2:9969 user 2 [::1]:9999 operator 2 show cache List the configured caches and the objects stored in each cache tree. $ echo 'show cache' | socat stdio /tmp/sock1 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918) 1 2 3 4 1. pointer to the cache structure 2. cache name 3. pointer to the mmap area (shctx) 4. number of blocks available for reuse in the shctx 0x7f6ac6c5b4cc hash:286881868 size:39114 (39 blocks), refcount:9, expire:237 1 2 3 4 5 6 1. pointer to the cache entry 2. first 32 bits of the hash 3. size of the object in bytes 4. number of blocks used for the object 5. number of transactions using the entry 6. expiration time, can be negative if already expired show env [] Dump one or all environment variables known by the process. Without any argument, all variables are dumped. With an argument, only the specified variable is dumped if it exists. Otherwise "Variable not found" is emitted. Variables are dumped in the same format as they are stored or returned by the "env" utility, that is, "=". This can be handy when debugging certain configuration files making heavy use of environment variables to ensure that they contain the expected values. This command is restricted and can only be issued on sockets configured for levels "operator" or "admin". show errors [|] [request|response] Dump last known request and response errors collected by frontends and backends. If is specified, the limit the dump to errors concerning either frontend or backend whose ID is . Proxy ID "-1" will cause all instances to be dumped. If a proxy name is specified instead, its ID will be used as the filter. If "request" or "response" is added after the proxy name or ID, only request or response errors will be dumped. This command is restricted and can only be issued on sockets configured for levels "operator" or "admin". The errors which may be collected are the last request and response errors caused by protocol violations, often due to invalid characters in header names. The report precisely indicates what exact character violated the protocol. Other important information such as the exact date the error was detected, frontend and backend names, the server name (when known), the internal session ID and the source address which has initiated the session are reported too. All characters are returned, and non-printable characters are encoded. The most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one letter following a backslash. The backslash itself is encoded as '\\' to avoid confusion. Other non-printable characters are encoded '\xNN' where NN is the two-digits hexadecimal representation of the character's ASCII code. Lines are prefixed with the position of their first character, starting at 0 for the beginning of the buffer. At most one input line is printed per line, and large lines will be broken into multiple consecutive output lines so that the output never goes beyond 79 characters wide. It is easy to detect if a line was broken, because it will not end with '\n' and the next line's offset will be followed by a '+' sign, indicating it is a continuation of previous line. Example : $ echo "show errors -1 response" | socat stdio /tmp/sock1 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1) response length 213 bytes, error at position 23: 00000 HTTP/1.0 200 OK\r\n 00017 header/bizarre:blah\r\n 00038 Location: blah\r\n 00054 Long-line: this is a very long line which should b 00104+ e broken into multiple lines on the output buffer, 00154+ otherwise it would be too large to print in a ter 00204+ minal\r\n 00211 \r\n In the example above, we see that the backend "http-in" which has internal ID 2 has blocked an invalid response from its server s2 which has internal ID 1. The request was on session 54 initiated by source 127.0.0.1 and received by frontend fe-eth0 whose ID is 1. The total response length was 213 bytes when the error was detected, and the error was at byte 23. This is the slash ('/') in header name "header/bizarre", which is not a valid HTTP character for a header name. show fd [] Dump the list of either all open file descriptors or just the one number if specified. This is only aimed at developers who need to observe internal states in order to debug complex issues such as abnormal CPU usages. One fd is reported per lines, and for each of them, its state in the poller using upper case letters for enabled flags and lower case for disabled flags, using "P" for "polled", "R" for "ready", "A" for "active", the events status using "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and "I" for "input", a few other flags like "N" for "new" (just added into the fd cache), "U" for "updated" (received an update in the fd cache), "L" for "linger_risk", "C" for "cloned", then the cached entry position, the pointer to the internal owner, the pointer to the I/O callback and its name when known. When the owner is a connection, the connection flags, and the target are reported (frontend, proxy or server). When the owner is a listener, the listener's state and its frontend are reported. There is no point in using this command without a good knowledge of the internals. It's worth noting that the output format may evolve over time so this output must not be parsed by tools designed to be durable. show activity Reports some counters about internal events that will help developers and more generally people who know haproxy well enough to narrow down the causes of reports of abnormal behaviours. A typical example would be a properly running process never sleeping and eating 100% of the CPU. The output fields will be made of one line per metric, and per-thread counters on the same line. These counters are 32-bit and will wrap during the process' life, which is not a problem since calls to this command will typically be performed twice. The fields are purposely not documented so that their exact meaning is verified in the code where the counters are fed. These values are also reset by the "clear counters" command. show info [typed|json] Dump info about haproxy status on current process. If "typed" is passed as an optional argument, field numbers, names and types are emitted as well so that external monitoring products can easily retrieve, possibly aggregate, then report information found in fields they don't know. Each field is dumped on its own line. If "json" is passed as an optional argument then information provided by "typed" output is provided in JSON format as a list of JSON objects. By default, the format contains only two columns delimited by a colon (':'). The left one is the field name and the right one is the value. It is very important to note that in typed output format, the dump for a single object is contiguous so that there is no need for a consumer to store everything at once. When using the typed output format, each line is made of 4 columns delimited by colons (':'). The first column is a dot-delimited series of 3 elements. The first element is the numeric position of the field in the list (starting at zero). This position shall not change over time, but holes are to be expected, depending on build options or if some fields are deleted in the future. The second element is the field name as it appears in the default "show info" output. The third element is the relative process number starting at 1. The rest of the line starting after the first colon follows the "typed output format" described in the section above. In short, the second column (after the first ':') indicates the origin, nature and scope of the variable. The third column indicates the type of the field, among "s32", "s64", "u32", "u64" and "str". Then the fourth column is the value itself, which the consumer knows how to parse thanks to column 3 and how to process thanks to column 2. Thus the overall line format in typed mode is : ..::: Example : > show info Name: HAProxy Version: 1.7-dev1-de52ea-146 Release_date: 2016/03/11 Nbproc: 1 Process_num: 1 Pid: 28105 Uptime: 0d 0h00m04s Uptime_sec: 4 Memmax_MB: 0 PoolAlloc_MB: 0 PoolUsed_MB: 0 PoolFailed: 0 (...) > show info typed 0.Name.1:POS:str:HAProxy 1.Version.1:POS:str:1.7-dev1-de52ea-146 2.Release_date.1:POS:str:2016/03/11 3.Nbproc.1:CGS:u32:1 4.Process_num.1:KGP:u32:1 5.Pid.1:SGP:u32:28105 6.Uptime.1:MDP:str:0d 0h00m08s 7.Uptime_sec.1:MDP:u32:8 8.Memmax_MB.1:CLP:u32:0 9.PoolAlloc_MB.1:MGP:u32:0 10.PoolUsed_MB.1:MGP:u32:0 11.PoolFailed.1:MCP:u32:0 (...) In the typed format, the presence of the process ID at the end of the first column makes it very easy to visually aggregate outputs from multiple processes. Example : $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \ echo show info typed | socat /var/run/haproxy.sock2 ) | \ sort -t . -k 1,1n -k 2,2 -k 3,3n 0.Name.1:POS:str:HAProxy 0.Name.2:POS:str:HAProxy 1.Version.1:POS:str:1.7-dev1-868ab3-148 1.Version.2:POS:str:1.7-dev1-868ab3-148 2.Release_date.1:POS:str:2016/03/11 2.Release_date.2:POS:str:2016/03/11 3.Nbproc.1:CGS:u32:2 3.Nbproc.2:CGS:u32:2 4.Process_num.1:KGP:u32:1 4.Process_num.2:KGP:u32:2 5.Pid.1:SGP:u32:30120 5.Pid.2:SGP:u32:30121 6.Uptime.1:MDP:str:0d 0h01m28s 6.Uptime.2:MDP:str:0d 0h01m28s (...) The format of JSON output is described in a schema which may be output using "show schema json". The JSON output contains no extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. Example : $ echo "show info json" | socat /var/run/haproxy.sock stdio | \ python -m json.tool The JSON output contains no extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. Example : $ echo "show info json" | socat /var/run/haproxy.sock stdio | \ python -m json.tool show map [] Dump info about map converters. Without argument, the list of all available maps is returned. If a is specified, its contents are dumped. is the # or . The first column is a unique identifier. It can be used as reference for the operation "del map" and "set map". The second column is the pattern and the third column is the sample if available. The data returned are not directly a list of available maps, but are the list of all patterns composing any map. Many of these patterns can be shared with ACL. show pools Dump the status of internal memory pools. This is useful to track memory usage when suspecting a memory leak for example. It does exactly the same as the SIGQUIT when running in foreground except that it does not flush the pools. show resolvers [] Dump statistics for the given resolvers section, or all resolvers sections if no section is supplied. For each name server, the following counters are reported: sent: number of DNS requests sent to this server valid: number of DNS valid responses received from this server update: number of DNS responses used to update the server's IP address cname: number of CNAME responses cname_error: CNAME errors encountered with this server any_err: number of empty response (IE: server does not support ANY type) nx: non existent domain response received from this server timeout: how many time this server did not answer in time refused: number of requests refused by this server other: any other DNS errors invalid: invalid DNS response (from a protocol point of view) too_big: too big response outdated: number of response arrived too late (after an other name server) show servers state [] Dump the state of the servers found in the running configuration. A backend name or identifier may be provided to limit the output to this backend only. The dump has the following format: - first line contains the format version (1 in this specification); - second line contains the column headers, prefixed by a sharp ('#'); - third line and next ones contain data; - each line starting by a sharp ('#') is considered as a comment. Since multiple versions of the output may co-exist, below is the list of fields and their order per file format version : 1: be_id: Backend unique id. be_name: Backend label. srv_id: Server unique id (in the backend). srv_name: Server label. srv_addr: Server IP address. srv_op_state: Server operational state (UP/DOWN/...). 0 = SRV_ST_STOPPED The server is down. 1 = SRV_ST_STARTING The server is warming up (up but throttled). 2 = SRV_ST_RUNNING The server is fully up. 3 = SRV_ST_STOPPING The server is up but soft-stopping (eg: 404). srv_admin_state: Server administrative state (MAINT/DRAIN/...). The state is actually a mask of values : 0x01 = SRV_ADMF_FMAINT The server was explicitly forced into maintenance. 0x02 = SRV_ADMF_IMAINT The server has inherited the maintenance status from a tracked server. 0x04 = SRV_ADMF_CMAINT The server is in maintenance because of the configuration. 0x08 = SRV_ADMF_FDRAIN The server was explicitly forced into drain state. 0x10 = SRV_ADMF_IDRAIN The server has inherited the drain status from a tracked server. 0x20 = SRV_ADMF_RMAINT The server is in maintenance because of an IP address resolution failure. 0x40 = SRV_ADMF_HMAINT The server FQDN was set from stats socket. srv_uweight: User visible server's weight. srv_iweight: Server's initial weight. srv_time_since_last_change: Time since last operational change. srv_check_status: Last health check status. srv_check_result: Last check result (FAILED/PASSED/...). 0 = CHK_RES_UNKNOWN Initialized to this by default. 1 = CHK_RES_NEUTRAL Valid check but no status information. 2 = CHK_RES_FAILED Check failed. 3 = CHK_RES_PASSED Check succeeded and server is fully up again. 4 = CHK_RES_CONDPASS Check reports the server doesn't want new sessions. srv_check_health: Checks rise / fall current counter. srv_check_state: State of the check (ENABLED/PAUSED/...). The state is actually a mask of values : 0x01 = CHK_ST_INPROGRESS A check is currently running. 0x02 = CHK_ST_CONFIGURED This check is configured and may be enabled. 0x04 = CHK_ST_ENABLED This check is currently administratively enabled. 0x08 = CHK_ST_PAUSED Checks are paused because of maintenance (health only). srv_agent_state: State of the agent check (ENABLED/PAUSED/...). This state uses the same mask values as "srv_check_state", adding this specific one : 0x10 = CHK_ST_AGENT Check is an agent check (otherwise it's a health check). bk_f_forced_id: Flag to know if the backend ID is forced by configuration. srv_f_forced_id: Flag to know if the server's ID is forced by configuration. srv_fqdn: Server FQDN. srv_port: Server port. srvrecord: DNS SRV record associated to this SRV. show sess Dump all known sessions. Avoid doing this on slow connections as this can be huge. This command is restricted and can only be issued on sockets configured for levels "operator" or "admin". Note that on machines with quickly recycled connections, it is possible that this output reports less entries than really exist because it will dump all existing sessions up to the last one that was created before the command was entered; those which die in the mean time will not appear. show sess Display a lot of internal information about the specified session identifier. This identifier is the first field at the beginning of the lines in the dumps of "show sess" (it corresponds to the session pointer). Those information are useless to most users but may be used by haproxy developers to troubleshoot a complex bug. The output format is intentionally not documented so that it can freely evolve depending on demands. You may find a description of all fields returned in src/dumpstats.c The special id "all" dumps the states of all sessions, which must be avoided as much as possible as it is highly CPU intensive and can take a lot of time. show stat [{|} ] [typed|json] Dump statistics using the CSV format; using the extended typed output format described in the section above if "typed" is passed after the other arguments; or in JSON if "json" is passed after the other arguments . By passing , and , it is possible to dump only selected items : - is a proxy ID, -1 to dump everything. Alternatively, a proxy name may be specified. In this case, this proxy's ID will be used as the ID selector. - selects the type of dumpable objects : 1 for frontends, 2 for backends, 4 for servers, -1 for everything. These values can be ORed, for example: 1 + 2 = 3 -> frontend + backend. 1 + 2 + 4 = 7 -> frontend + backend + server. - is a server ID, -1 to dump everything from the selected proxy. Example : $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1 >>> Name: HAProxy Version: 1.4-dev2-49 Release_date: 2009/09/23 Nbproc: 1 Process_num: 1 (...) # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...) stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...) stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...) (...) www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...) $ In this example, two commands have been issued at once. That way it's easy to find which process the stats apply to in multi-process mode. This is not needed in the typed output format as the process number is reported on each line. Notice the empty line after the information output which marks the end of the first block. A similar empty line appears at the end of the second block (stats) so that the reader knows the output has not been truncated. When "typed" is specified, the output format is more suitable to monitoring tools because it provides numeric positions and indicates the type of each output field. Each value stands on its own line with process number, element number, nature, origin and scope. This same format is available via the HTTP stats by passing ";typed" after the URI. It is very important to note that in typed output format, the dump for a single object is contiguous so that there is no need for a consumer to store everything at once. When using the typed output format, each line is made of 4 columns delimited by colons (':'). The first column is a dot-delimited series of 5 elements. The first element is a letter indicating the type of the object being described. At the moment the following object types are known : 'F' for a frontend, 'B' for a backend, 'L' for a listener, and 'S' for a server. The second element The second element is a positive integer representing the unique identifier of the proxy the object belongs to. It is equivalent to the "iid" column of the CSV output and matches the value in front of the optional "id" directive found in the frontend or backend section. The third element is a positive integer containing the unique object identifier inside the proxy, and corresponds to the "sid" column of the CSV output. ID 0 is reported when dumping a frontend or a backend. For a listener or a server, this corresponds to their respective ID inside the proxy. The fourth element is the numeric position of the field in the list (starting at zero). This position shall not change over time, but holes are to be expected, depending on build options or if some fields are deleted in the future. The fifth element is the field name as it appears in the CSV output. The sixth element is a positive integer and is the relative process number starting at 1. The rest of the line starting after the first colon follows the "typed output format" described in the section above. In short, the second column (after the first ':') indicates the origin, nature and scope of the variable. The third column indicates the type of the field, among "s32", "s64", "u32", "u64" and "str". Then the fourth column is the value itself, which the consumer knows how to parse thanks to column 3 and how to process thanks to column 2. Thus the overall line format in typed mode is : .....::: Here's an example of typed output format : $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1 F.2.0.0.pxname.1:MGP:str:private-frontend F.2.0.1.svname.1:MGP:str:FRONTEND F.2.0.8.bin.1:MGP:u64:0 F.2.0.9.bout.1:MGP:u64:0 F.2.0.40.hrsp_2xx.1:MGP:u64:0 L.2.1.0.pxname.1:MGP:str:private-frontend L.2.1.1.svname.1:MGP:str:sock-1 L.2.1.17.status.1:MGP:str:OPEN L.2.1.73.addr.1:MGP:str:0.0.0.0:8001 S.3.13.60.rtime.1:MCP:u32:0 S.3.13.61.ttime.1:MCP:u32:0 S.3.13.62.agent_status.1:MGP:str:L4TOUT S.3.13.64.agent_duration.1:MGP:u64:2001 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout S.3.13.67.check_rise.1:MCP:u32:2 S.3.13.68.check_fall.1:MCP:u32:3 S.3.13.69.check_health.1:SGP:u32:0 S.3.13.70.agent_rise.1:MaP:u32:1 S.3.13.71.agent_fall.1:SGP:u32:1 S.3.13.72.agent_health.1:SGP:u32:1 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888 S.3.13.75.mode.1:MAP:str:http B.3.0.0.pxname.1:MGP:str:private-backend B.3.0.1.svname.1:MGP:str:BACKEND B.3.0.2.qcur.1:MGP:u32:0 B.3.0.3.qmax.1:MGP:u32:0 B.3.0.4.scur.1:MGP:u32:0 B.3.0.5.smax.1:MGP:u32:0 B.3.0.6.slim.1:MGP:u32:1000 B.3.0.55.lastsess.1:MMP:s32:-1 (...) In the typed format, the presence of the process ID at the end of the first column makes it very easy to visually aggregate outputs from multiple processes, as show in the example below where each line appears for each process : $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \ echo show stat typed | socat /var/run/haproxy.sock2 - ) | \ sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n B.3.0.0.pxname.1:MGP:str:private-backend B.3.0.0.pxname.2:MGP:str:private-backend B.3.0.1.svname.1:MGP:str:BACKEND B.3.0.1.svname.2:MGP:str:BACKEND B.3.0.2.qcur.1:MGP:u32:0 B.3.0.2.qcur.2:MGP:u32:0 B.3.0.3.qmax.1:MGP:u32:0 B.3.0.3.qmax.2:MGP:u32:0 B.3.0.4.scur.1:MGP:u32:0 B.3.0.4.scur.2:MGP:u32:0 B.3.0.5.smax.1:MGP:u32:0 B.3.0.5.smax.2:MGP:u32:0 B.3.0.6.slim.1:MGP:u32:1000 B.3.0.6.slim.2:MGP:u32:1000 (...) The format of JSON output is described in a schema which may be output using "show schema json". The JSON output contains no extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. Example : $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \ python -m json.tool The JSON output contains no extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. Example : $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \ python -m json.tool show table Dump general information on all known stick-tables. Their name is returned (the name of the proxy which holds them), their type (currently zero, always IP), their size in maximum possible number of entries, and the number of entries currently in use. Example : $ echo "show table" | socat stdio /tmp/sock1 >>> # table: front_pub, type: ip, size:204800, used:171454 >>> # table: back_rdp, type: ip, size:204800, used:0 show table [ data. ] | [ key ] Dump contents of stick-table . In this mode, a first line of generic information about the table is reported as with "show table", then all entries are dumped. Since this can be quite heavy, it is possible to specify a filter in order to specify what entries to display. When the "data." form is used the filter applies to the stored data (see "stick-table" in section 4.2). A stored data type must be specified in , and this data type must be stored in the table otherwise an error is reported. The data is compared according to with the 64-bit integer . Operators are the same as with the ACLs : - eq : match entries whose data is equal to this value - ne : match entries whose data is not equal to this value - le : match entries whose data is less than or equal to this value - ge : match entries whose data is greater than or equal to this value - lt : match entries whose data is less than this value - gt : match entries whose data is greater than this value When the key form is used the entry is shown. The key must be of the same type as the table, which currently is limited to IPv4, IPv6, integer, and string. Example : $ echo "show table http_proxy" | socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:2 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \ bytes_out_rate(60000)=187 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:2 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 $ echo "show table http_proxy data.conn_rate gt 5" | \ socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:2 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 $ echo "show table http_proxy key 127.0.0.2" | \ socat stdio /tmp/sock1 >>> # table: http_proxy, type: ip, size:204800, used:2 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ bytes_out_rate(60000)=191 When the data criterion applies to a dynamic value dependent on time such as a bytes rate, the value is dynamically computed during the evaluation of the entry in order to decide whether it has to be dumped or not. This means that such a filter could match for some time then not match anymore because as time goes, the average event rate drops. It is possible to use this to extract lists of IP addresses abusing the service, in order to monitor them or even blacklist them in a firewall. Example : $ echo "show table http_proxy data.gpc0 gt 0" \ | socat stdio /tmp/sock1 \ | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt ( or | awk '/key/{ print a[split($2,a,"=")]; }' ) show tls-keys [id|*] Dump all loaded TLS ticket keys references. The TLS ticket key reference ID and the file from which the keys have been loaded is shown. Both of those can be used to update the TLS keys using "set ssl tls-key". If an ID is specified as parameter, it will dump the tickets, using * it will dump every keys from every references. show schema json Dump the schema used for the output of "show info json" and "show stat json". The contains no extra whitespace in order to reduce the volume of output. For human consumption passing the output through a pretty printer may be helpful. Example : $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \ python -m json.tool The schema follows "JSON Schema" (json-schema.org) and accordingly verifiers may be used to verify the output of "show info json" and "show stat json" against the schema. shutdown frontend Completely delete the specified frontend. All the ports it was bound to will be released. It will not be possible to enable the frontend anymore after this operation. This is intended to be used in environments where stopping a proxy is not even imaginable but a misconfigured proxy must be fixed. That way it's possible to release the port and bind it into another process to restore operations. The frontend will not appear at all on the stats page once it is terminated. The frontend may be specified either by its name or by its numeric ID, prefixed with a sharp ('#'). This command is restricted and can only be issued on sockets configured for level "admin". shutdown session Immediately terminate the session matching the specified session identifier. This identifier is the first field at the beginning of the lines in the dumps of "show sess" (it corresponds to the session pointer). This can be used to terminate a long-running session without waiting for a timeout or when an endless transfer is ongoing. Such terminated sessions are reported with a 'K' flag in the logs. shutdown sessions server / Immediately terminate all the sessions attached to the specified server. This can be used to terminate long-running sessions after a server is put into maintenance mode, for instance. Such terminated sessions are reported with a 'K' flag in the logs. 10. Tricks for easier configuration management ---------------------------------------------- It is very common that two HAProxy nodes constituting a cluster share exactly the same configuration modulo a few addresses. Instead of having to maintain a duplicate configuration for each node, which will inevitably diverge, it is possible to include environment variables in the configuration. Thus multiple configuration may share the exact same file with only a few different system wide environment variables. This started in version 1.5 where only addresses were allowed to include environment variables, and 1.6 goes further by supporting environment variables everywhere. The syntax is the same as in the UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening curly brace ('{'), then the variable name followed by the closing brace ('}'). Except for addresses, environment variables are only interpreted in arguments surrounded with double quotes (this was necessary not to break existing setups using regular expressions involving the dollar symbol). Environment variables also make it convenient to write configurations which are expected to work on various sites where only the address changes. It can also permit to remove passwords from some configs. Example below where the the file "site1.env" file is sourced by the init script upon startup : $ cat site1.env LISTEN=192.168.1.1 CACHE_PFX=192.168.11 SERVER_PFX=192.168.22 LOGGER=192.168.33.1 STATSLP=admin:pa$$w0rd ABUSERS=/etc/haproxy/abuse.lst TIMEOUT=10s $ cat haproxy.cfg global log "${LOGGER}:514" local0 defaults mode http timeout client "${TIMEOUT}" timeout server "${TIMEOUT}" timeout connect 5s frontend public bind "${LISTEN}:80" http-request reject if { src -f "${ABUSERS}" } stats uri /stats stats auth "${STATSLP}" use_backend cache if { path_end .jpg .css .ico } default_backend server backend cache server cache1 "${CACHE_PFX}.1:18080" check server cache2 "${CACHE_PFX}.2:18080" check backend server server cache1 "${SERVER_PFX}.1:8080" check server cache2 "${SERVER_PFX}.2:8080" check 11. Well-known traps to avoid ----------------------------- Once in a while, someone reports that after a system reboot, the haproxy service wasn't started, and that once they start it by hand it works. Most often, these people are running a clustered IP address mechanism such as keepalived, to assign the service IP address to the master node only, and while it used to work when they used to bind haproxy to address 0.0.0.0, it stopped working after they bound it to the virtual IP address. What happens here is that when the service starts, the virtual IP address is not yet owned by the local node, so when HAProxy wants to bind to it, the system rejects this because it is not a local IP address. The fix doesn't consist in delaying the haproxy service startup (since it wouldn't stand a restart), but instead to properly configure the system to allow binding to non-local addresses. This is easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This is also needed in order to transparently intercept the IP traffic that passes through HAProxy for a specific target address. Multi-process configurations involving source port ranges may apparently seem to work but they will cause some random failures under high loads because more than one process may try to use the same source port to connect to the same server, which is not possible. The system will report an error and a retry will happen, picking another port. A high value in the "retries" parameter may hide the effect to a certain extent but this also comes with increased CPU usage and processing time. Logs will also report a certain number of retries. For this reason, port ranges should be avoided in multi-process configurations. Since HAProxy uses SO_REUSEPORT and supports having multiple independent processes bound to the same IP:port, during troubleshooting it can happen that an old process was not stopped before a new one was started. This provides absurd test results which tend to indicate that any change to the configuration is ignored. The reason is that in fact even the new process is restarted with a new configuration, the old one also gets some incoming connections and processes them, returning unexpected results. When in doubt, just stop the new process and try again. If it still works, it very likely means that an old process remains alive and has to be stopped. Linux's "netstat -lntp" is of good help here. When adding entries to an ACL from the command line (eg: when blacklisting a source address), it is important to keep in mind that these entries are not synchronized to the file and that if someone reloads the configuration, these updates will be lost. While this is often the desired effect (for blacklisting) it may not necessarily match expectations when the change was made as a fix for a problem. See the "add acl" action of the CLI interface. 12. Debugging and performance issues ------------------------------------ When HAProxy is started with the "-d" option, it will stay in the foreground and will print one line per event, such as an incoming connection, the end of a connection, and for each request or response header line seen. This debug output is emitted before the contents are processed, so they don't consider the local modifications. The main use is to show the request and response without having to run a network sniffer. The output is less readable when multiple connections are handled in parallel, though the "debug2ansi" and "debug2html" scripts found in the examples/ directory definitely help here by coloring the output. If a request or response is rejected because HAProxy finds it is malformed, the best thing to do is to connect to the CLI and issue "show errors", which will report the last captured faulty request and response for each frontend and backend, with all the necessary information to indicate precisely the first character of the input stream that was rejected. This is sometimes needed to prove to customers or to developers that a bug is present in their code. In this case it is often possible to relax the checks (but still keep the captures) using "option accept-invalid-http-request" or its equivalent for responses coming from the server "option accept-invalid-http-response". Please see the configuration manual for more details. Example : > show errors Total events captured on [13/Oct/2015:13:43:47.169] : 1 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request backend (#-1), server (#-1), event #0 src 127.0.0.1:51981, session #0, session flags 0x00000080 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000 HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00808002, out 0 bytes, total 31 bytes pending 31 bytes, wrapping at 8040, error at position 13: 00000 GET /invalid request HTTP/1.1\r\n The output of "show info" on the CLI provides a number of useful information regarding the maximum connection rate ever reached, maximum SSL key rate ever reached, and in general all information which can help to explain temporary issues regarding CPU or memory usage. Example : > show info Name: HAProxy Version: 1.6-dev7-e32d18-17 Release_date: 2015/10/12 Nbproc: 1 Process_num: 1 Pid: 7949 Uptime: 0d 0h02m39s Uptime_sec: 159 Memmax_MB: 0 Ulimit-n: 120032 Maxsock: 120032 Maxconn: 60000 Hard_maxconn: 60000 CurrConns: 0 CumConns: 3 CumReq: 3 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 0 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 0 ConnRateLimit: 0 MaxConnRate: 1 SessRate: 0 SessRateLimit: 0 MaxSessRate: 1 SslRate: 0 SslRateLimit: 0 MaxSslRate: 0 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 0 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 5 Run_queue: 1 Idle_pct: 100 node: wtap description: When an issue seems to randomly appear on a new version of HAProxy (eg: every second request is aborted, occasional crash, etc), it is worth trying to enable memory poisoning so that each call to malloc() is immediately followed by the filling of the memory area with a configurable byte. By default this byte is 0x50 (ASCII for 'P'), but any other byte can be used, including zero (which will have the same effect as a calloc() and which may make issues disappear). Memory poisoning is enabled on the command line using the "-dM" option. It slightly hurts performance and is not recommended for use in production. If an issue happens all the time with it or never happens when poisoning uses byte zero, it clearly means you've found a bug and you definitely need to report it. Otherwise if there's no clear change, the problem it is not related. When debugging some latency issues, it is important to use both strace and tcpdump on the local machine, and another tcpdump on the remote system. The reason for this is that there are delays everywhere in the processing chain and it is important to know which one is causing latency to know where to act. In practice, the local tcpdump will indicate when the input data come in. Strace will indicate when haproxy receives these data (using recv/recvfrom). Warning, openssl uses read()/write() syscalls instead of recv()/send(). Strace will also show when haproxy sends the data, and tcpdump will show when the system sends these data to the interface. Then the external tcpdump will show when the data sent are really received (since the local one only shows when the packets are queued). The benefit of sniffing on the local system is that strace and tcpdump will use the same reference clock. Strace should be used with "-tts200" to get complete timestamps and report large enough chunks of data to read them. Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence numbers and complete timestamps. In practice, received data are almost always immediately received by haproxy (unless the machine has a saturated CPU or these data are invalid and not delivered). If these data are received but not sent, it generally is because the output buffer is saturated (ie: recipient doesn't consume the data fast enough). This can be confirmed by seeing that the polling doesn't notify of the ability to write on the output file descriptor for some time (it's often easier to spot in the strace output when the data finally leave and then roll back to see when the write event was notified). It generally matches an ACK received from the recipient, and detected by tcpdump. Once the data are sent, they may spend some time in the system doing nothing. Here again, the TCP congestion window may be limited and not allow these data to leave, waiting for an ACK to open the window. If the traffic is idle and the data take 40 ms or 200 ms to leave, it's a different issue (which is not an issue), it's the fact that the Nagle algorithm prevents empty packets from leaving immediately, in hope that they will be merged with subsequent data. HAProxy automatically disables Nagle in pure TCP mode and in tunnels. However it definitely remains enabled when forwarding an HTTP body (and this contributes to the performance improvement there by reducing the number of packets). Some HTTP non-compliant applications may be sensitive to the latency when delivering incomplete HTTP response messages. In this case you will have to enable "option http-no-delay" to disable Nagle in order to work around their design, keeping in mind that any other proxy in the chain may similarly be impacted. If tcpdump reports that data leave immediately but the other end doesn't see them quickly, it can mean there is a congested WAN link, a congested LAN with flow control enabled and preventing the data from leaving, or more commonly that HAProxy is in fact running in a virtual machine and that for whatever reason the hypervisor has decided that the data didn't need to be sent immediately. In virtualized environments, latency issues are almost always caused by the virtualization layer, so in order to save time, it's worth first comparing tcpdump in the VM and on the external components. Any difference has to be credited to the hypervisor and its accompanying drivers. When some TCP SACK segments are seen in tcpdump traces (using -vv), it always means that the side sending them has got the proof of a lost packet. While not seeing them doesn't mean there are no losses, seeing them definitely means the network is lossy. Losses are normal on a network, but at a rate where SACKs are not noticeable at the naked eye. If they appear a lot in the traces, it is worth investigating exactly what happens and where the packets are lost. HTTP doesn't cope well with TCP losses, which introduce huge latencies. The "netstat -i" command will report statistics per interface. An interface where the Rx-Ovr counter grows indicates that the system doesn't have enough resources to receive all incoming packets and that they're lost before being processed by the network driver. Rx-Drp indicates that some received packets were lost in the network stack because the application doesn't process them fast enough. This can happen during some attacks as well. Tx-Drp means that the output queues were full and packets had to be dropped. When using TCP it should be very rare, but will possibly indicate a saturated outgoing link. 13. Security considerations --------------------------- HAProxy is designed to run with very limited privileges. The standard way to use it is to isolate it into a chroot jail and to drop its privileges to a non-root user without any permissions inside this jail so that if any future vulnerability were to be discovered, its compromise would not affect the rest of the system. In order to perform a chroot, it first needs to be started as a root user. It is pointless to build hand-made chroots to start the process there, these ones are painful to build, are never properly maintained and always contain way more bugs than the main file-system. And in case of compromise, the intruder can use the purposely built file-system. Unfortunately many administrators confuse "start as root" and "run as root", resulting in the uid change to be done prior to starting haproxy, and reducing the effective security restrictions. HAProxy will need to be started as root in order to : - adjust the file descriptor limits - bind to privileged port numbers - bind to a specific network interface - transparently listen to a foreign address - isolate itself inside the chroot jail - drop to another non-privileged UID HAProxy may require to be run as root in order to : - bind to an interface for outgoing connections - bind to privileged source ports for outgoing connections - transparently bind to a foreign address for outgoing connections Most users will never need the "run as root" case. But the "start as root" covers most usages. A safe configuration will have : - a chroot statement pointing to an empty location without any access permissions. This can be prepared this way on the UNIX command line : # mkdir /var/empty && chmod 0 /var/empty || echo "Failed" and referenced like this in the HAProxy configuration's global section : chroot /var/empty - both a uid/user and gid/group statements in the global section : user haproxy group haproxy - a stats socket whose mode, uid and gid are set to match the user and/or group allowed to access the CLI so that nobody may access it : stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600 I am well aware of the popular opinion that such subjects are too abstruse to be understood by practical mechanics¡ªan assumption that is founded mainly in the fact that the subject of heat and motion are not generally studied, and have been too recently demonstrated in a scientific way to command confidence and attention; but the subject is really no more difficult to understand in an elementary sense than that of the relation between movement and force illustrated in the "mechanical powers" of school-books, which no apprentice ever did or ever will understand, except by first studying the principles of force and motion, independent of mechanical agents, such as screws, levers, wedges, and so on. A new design should be based upon one of two suppositions¡ªeither that existing mechanism is imperfect in its construction, or that it lacks functions which a new design may supply; and if those who spend their time in making plans for novel machinery would stop to consider this from the beginning, it would save no little of the time wasted in what may be called scheming without a purpose. A man, coming silently from some concealment, in a dory, undetected in their busy absorption, held something menacingly businesslike and sending sun glints from its blue steel. Its hollow nose covered both at the range he had. ¡°Well, Sky Patrol¡ªand Ground Crew,¡± he hailed them. ¡°We are going to see some excitement at last!¡± "You're right, Si," shouted the Lieutenant and Shorty. "Hip, hip, hooray for the Army o' the Cumberland and old Pap Thomas!" "Glad ain't no name for it," said Levi. "Did you say you'd got the boys in there? Here, you men, bring me two or three of those cracker-boxes." "The very same company," gasped the woman. And the other asked: "And the raid'll be made ter-morrer?" "My¡ª" But before they had time to answer, something burst from between the stalls and ran down the darkling slope, brandishing a knife. It was Mexico Bill, running amok, as he had sometimes run before, but on less crowded occasions. The women sent up an ear-splitting yell, and made a fresh onslaught on the hedge. Someone grabbed the half-breed from behind, but his knife flashed, and the next moment he was free, dashing through the gorse towards his victims. Such a discovery could not long remain a secret;¡ªthe tale reached the ears of young De Boteler, and, already prepossessed in his favour, it was but a natural consequence that Calverley should rise from being first an assistant, to be the steward, the page, and, at length, the esquire to the heir to the barony of Sudley. But the progress of his fortunes did but add to the malevolence of the detractor and the tale-bearer; theft, sacrilege, and even murder were hinted at as probable causes for a youth, who evidently did not belong to the vulgar, being thus a friendless outcast. But the most charitable surmise was, that he was the offspring of the unhallowed love of some dame or damsel who had reared him in privacy, and had destined him for the church; and that either upon the death of his protectress, or through some fault, he had been expelled from his home. Calverley had a distant authoritative manner towards his equals and inferiors, which, despite every effort, checked inquisitiveness; and all the information he ever gave was, that he was the son of a respectable artizan of the city of London, whom his father's death had left friendless. Whether this statement was correct or not, could never be discovered. Calverley was never known to allude to aught that happened in the years previous to his becoming an inmate of the castle: what little he had said was merely in reply to direct questions. It would seem, then, that he stood alone in the world, and such a situation is by no means enviable; and although duplicity, selfishness and tyranny, formed the principal traits in his character; and though independently of tyranny and selfishness, his mind instinctively shrunk from any contact, save that of necessity, with those beneath him, yet had he gazed upon the growing beauty of Margaret till a love pure and deep¡ªa love in which was concentrated all the slumbering affections, had risen and expanded in his breast, until it had, as it were, become a part of his being. HoMEAÒ»¼¶ÈÕ±¾l00Ãâ·Ñ¿´ ENTER NUMBET 0018weez.com.cn
aeqv.com.cn
www.bianyimin.com.cn
scdehui.com.cn
www.taboohunter.com.cn
iwru.com.cn
www.gzgjzg.com.cn
www.costory.com.cn
xiureng.com.cn
www.trbus.com.cn
长篇失控的淫乱小说 欧美美女游戏节目 使尽操成人网 大鸡巴爱丝袜 成人电影免播放器的 一级色图欧美 人体艺术大胆下体图 女人下体凹凸 欧美骚逼25 少妇穴穴20p 老女人l 日本少妇做爱图片25p 女学生性交口交肛交 就爱操逼录音 胖女同 诱惑写真那个网站好 丰腴熟女乱伦 哪里能看到苍井空网络硬盘 女教师乱淫动态图片 熟妇掰逼图 性爱之人兽交 中国最火簧片 WWW.A1J3.COM WWW.HZGRYY.COM WWW.JPTUBES.COM WWW.XZYWHY.COM WWW.LYJJBJ.COM WWW.U2CHE.COM WWW.CCC195.COM WWW.ZISHG.COM WWW.VB04.COM WWW.88SCSC.COM WWW.CQGGZY.COM WWW.YNKQN.COM WWW.HAOTE.COM WWW.FXE9.COM WWW.DDD91.COM WWW.ZQTYQC.COM WWW.YUEJLWO.COM WWW.007CB.COM WWW.XMCGHH.COM WWW.AV977.COM WWW.6789DA.COM WWW.QULA7.COM WWW.HHH738.COM WWW.QB5200.COM WWW.0755MSX.NET WWW.BKHCG.COM WWW.REN999.COM WWW.AV577.COM WWW.BX857.COM WWW.QUXUNW.COM WWW.V2D5.COM WWW.500173.COM WWW.771588.COM WWW.MAV7676.COM WWW.6655.COM WWW.YOKOO.COM WWW.G8GW.COM WWW.CCC294.COM WWW.868RRC.COM WWW.XIQUREN.COM WWW.XAZHKJ.COM HEZE.DZWWW.COM WWW.UB73.COM WWW.8220365.COM WWW.AVTT2014.COM WWW.33TVTV.COM WWW.SHHEZWZ.COM WWW.IFP7.COM WWW.SZZBAF.COM WWW.1122NJ.COM WWW.ENET.COM.CN WWW.WWW.91DIZHI.SPACE WWW.13YM.COM WWW.BBB528.COM WWW.Y5QD.COM WWW.CCC903.COM WWW.OUOULU.COM WWW.YESHEMAO.NET WWW.JLZDYY.COM WWW.612SE.COM WWW.SHTJZK.COM WWW.CHUNMEN.COM WWW.A0663.COM WWW.019GEGE.COM WWW.CZJXDQ.COM WWW.GAO540.COM WWW.JLMMBB.COM WWW.HGDVD.COM WWW.9LALA.COM WWW.CC354.COM WWW.97WEN.CN WWW.OUYUJX.COM WWW.DAJIE.COM AWWW.777ZYZ.COM WWW.CCC700.COM WWW.612621.COM WWW.NI3456.COM 另类变态图片调教 放尿系列 糖糖幼幼免费网站 艹幼处在线 在线免费观看视频偷拍 窝久久草 999热这里只9999p9有精品视频 AV搞视频 王宪三级片 亚洲成人A片毛片 熟女乱伦20p上一篇下一篇 成人激情午夜网 色七七影视 插插日本骚女人 妈妈人妻受孕 第四色先锋视频 国产美女自慰视频在线观看 青楼社区的最新网址是 色欲影视2p x小色哥 色色肉肉伦乱图 少女和少男做爱黄色网站 黑崎礼子在线 美国二级伦理宅宅网 橹射 露外阴的伦理电影 裸体学生妹 童话村av 亚洲欧美幼齿无码 超碰在线大片 迷奸漂亮女邻居 小泽玛利亚与人妖图片 淑女爆菊网 古典武侠妹妹和朋友 后入90后少妇18p 超碰sm免费公开成人视频 da炮机av无码 你懂的AV资源网 制服丝袜最新在线视频 大棒插穴乱伦小说 狼国48Q 男女操B图 wwwAV4455com 天上人间宝宝福利吧 寂寞人妻居家自拍色图 性爱综合AV 草群裙社区在线视频 6080三级片mp4 成人哥哥干哥哥 街夜色亚洲视屏 白白发布 2016AV撸撸射在线视频 强奸幼女开苞小说 谢文的人体艺术 泷泽萝拉AV在线wwwlulukan1com 噜噜噜偷拍自拍 295cccom 搜索www妈妈与儿子乱伦大杂烩 性奴骚奶子 美淫小 屁眼调教 看岛国大片 偷拍亚洲美女性爱视频 偷偷摸狠狠干 狼友基地在线 高中处女开苞落红伦理聚合免插件在线 wwwfff2345 caopporn超碰 2017天天撸 搞搞电影网成人视频 国产自拍国产久久自拍视频 欧洲成人AV片 diao青青草 爱爱综合x0 狠狠舔干 曰本理伦图片 hentaixxx少女 淫淫网淫妻交换 人曾交长片 后门插逼逼动态 骗朋友出来轮奸她操死她 手机美利坚中文字幕 在线Av东方伊甸园 韩国r在线网站 男女野战图片 西瓜操 dilidili艳母 欧美成人在线免费视频 熟女乱亚洲影院 wwwpp398comVR 皇色精彩视频久草在线 小姐被黑人轮流干 天天更新在线视频 影音先锋ye321最新地址 人体艺术日本 国产女神自慰在线 淫妻交换性爱技巧校园春色hhxxoo1com 91驾校在线视频 影音先锋网站亚洲 不需下载在线观看操逼短片 猛男干娘女图片 找老熟妇做爱 543cccn 偷拍偷拍少妇25p 真人上传黄色视频免费在线观看 岛国片公媳乱伦 亚州黄色小说 成人电影偷拍无码 成人啪啪啪看骚妇的小逼插的水汪汪高潮连连视频 wwwtomitaocom 淫色直播的有哪些 人兽性爱欧美三级片 男人第四色网 久久在线经典视频 wwwaa847com下载 1111kf李宗瑞 ww26xecom 赞助商影片分类偷拍视频230自拍视频240国产视频19日韩视频 东京热MV 撸一撸亚洲色图 最新的黄色网站 775jj 百性阁mecom sss911 日本性感女护士舔男孩大鸡巴全部视频大全免费观看 影音先锋看激情电影 亚洲天堂av在线直播 www567net最新网站 各种国产AV有什么免费网址 wwwddd20com 爆乳国产 亚洲另类一国产aⅴ在线视频 特菲娜样子 爱的色放图 女人逼逼 91超在线观看视频播放 玩幼女b黄色电影淫放 哦哦叉叉 www1111avcowang 快播里面能看黄色图片不 少妇性爱电影 黑白中文母22p bb啪啪网站视频 超碰成人公开视频超碰免费131spcom 成人看片自慰免费视频在线观看视频 迅雷色色强暴小说 欧美性爱色域网 日韩av手机在线 色色人阁www63cocom av久久在线观看 欧美图片偷拍图片区手机在线播放 5080午夜电影 亚洲av无码久久在线 wwwseqing爆 公交车上干美女 人妻熟女激情自拍 俺去也插插插插 seri123一样的网站 淫女偷拍 免费三级金梅瓶 河南工业大学钟月双 美女屁股wwwpp0022com 影音先锋主播自慰 4hu46 强奸幼女妹妹小说 女人阴道穴片 av美女天堂下载 淫色人妻哥哥操 亚洲啊T天堂 狠狠爱在线牛人视频 紧急通知小姨子 田野色在线视频 xia12345magnet 天天色播 av毛片成人在线观看网站 激情啊嗯啊嗯啊啊啊嗯嗯啊 欧美大香蕉毛片 天天啪啪 色站成人美眉红楼 露脸良家人妻熟女 BNSPS298 www123chaopengcom 校园春色处女女大学生 淫荡母乳人妻 99pp黄色网站 男女上床私照 淫色网极品美穴 有木有AV软件 古墓丽影h版免费观看 黄色网址导行 办公室女秘书伦理片 日本超级AV在线视频 在哪里可以免费观看无码黄片 www1122vgcm 成人色漫 在线手机播放器 我爱操操电影免费ti789com 三给片区电影在线观看 wwgaosecom 麻生希第一部快播 meinvz1ynet 幼幼被破处 搜索美女三级黄片口活 撸一撸日日爱狠狠爱 男女大尺度啪啪图片 万达影院 97超碰色视频在线观看 宜春院首页怡红院 jlzz4欧美 www58hhhhcom 夜夜干夜夜5岁到13岁 1024jd在线看免费视频 人气女优小说 亚渊成人影院 日本激情点的床上男女 欧洲黄绝 欧美人与兽肏屄电影 淫乱派对资源 www2333ca 叼嗨视频直播真人版 免播放器成人熟女乱伦电影 成人A片小说 色色撸手机在线观看 美女哈鞭视频 上黑丝美女 无码噜噜噜AV在线观看 插日本美女粉嫩小嫩穴 免费视性爱频在线观看网站1 日本2017最新H动漫 丝袜日本美女小说 美国最新黄网址大全 蝌蚪窝久久视频 日本女模做爱视频下载 狗鸡巴插骚穴小说 香港a片毛片hciyycom 413121神马电影 jjady3infoa121html 国产视频妈妈在儿子面前自慰 黄鳝门影音先锋 表姐穴穴湿润 偷拍人妻影音 叫做sh什么的电影 草了同事老婆 超碰视频A片在线视频wwwvb111com 偷拍自拍撸撸她 lunliwang 狗干MM porn老女人 达恩电影网你懂的网站 欧美高跟骚女 男尼所巨炮无遮图mman189com 爆操情人 qq公众号色色的 青青草是华人绿色18 阿女AV 新惰色站 PU510COM 插日本少妇20p 国产谢妹妹影院wwwqqqq95com 在线访问升级中 东方在线校园春色 日韩插插插 phoenixmarie小男人 狼友a∨在线视频xw970com 亚洲高清可乐操第一站 制度丝袜国产手机在线 天堂妹2017 夜夜撸在线视频暗暗撸在线视频加多撸在线视频天天撸在线视频 私拍性爱视频 美女激情超碰 bu444com 迅雷色色种子 健身房女教练av 26UUU天天曰久久射MP4 亚洲涩图AV WWW210BECOM A极超碰 sunpornecom avttktv caov2 女同爱爱mp4 亚洲无码超碰在线播放视频2017年版中文字幕中文字幕中文字幕人妻淫乱 人人日动漫 哥哥色哥哥射哥哥干哥哥撸 男女操免费视频 想屄图 奇米影视777撸 nipingdebi 日本极品大胆私阴艺术 美女激情裸聊自拍 欧美色尼玛肛交图库 苍井空插穴图片hha6 taiwansexvideo 日本av美女裸体 色18美女游客 熟女bb被操 爱爱在线成人视频 美国长吊爆肏嫩逼 人体美鲍视频 淫乱性交图片论坛 抽插片 丝袜美女撸 真是嫩啊 肉穴被插 日本美女人体图 做爱的黄色诗 孕妇 torrent百度云 黄色网站2016自拍视频 偷拍色老大导航400色导航 嫩屄人体艺术摄影 调教骚穴 正在播放和朝鲜老妈操 岳母的肉洞 88ri88ri图片88i88 大鸡巴插狗b 操骚熟娘们 成人在线走光 嫂子 阴唇 色bt导航 少妇群交色无码 操老汉影视 操衅淫荡 成人虐待系列网站 美女裸体艺术图片15p 成人综合论坛2014 幼女做爱av 幼女激情做爱 zumeiav oumeidngfu 色妺妹快播电影 三级完整影 亚洲90后色图 迷奸小妹影音先锋 日本女性裸体色图 sha人妻civou 车模打炮17p奔雷 美女淫乱合成图片 美女人体 东北操逼电影 楚留香色网 现在有什么说新的中国美少女组合 影视先锋看动漫 山村乱伦小说远山的呼唤 日本美女美鲍人体图片 无毛幼女潮吹 车上草妈 美女被干黄色无遮挡图片 吸吮龟头 艳色荡母有声小说 百人体图片下载 WWW_X8S2_COM 美女搞鸡激情 淫女影音先锋 日本母乳喂养宠物视频 性欲强的母子淫乱 110139 白白色趁人视频 亚洲五月色人阁 日本人妖性交网 和张妈做爱 吉吉影音av激情电影 后进式猛擦美女p 日本大胆妹妹做爱 苍井空电影种子 下载 振动器黄色网站 thisav新网 女孩放尿 女人被狠操动态图 嫩白的嫂嫂 高中美女做爱图 先锋影音 伦理 影音先锋美女躶体乱伦小说 xex8cc 黑人学生内射美女老师 女优嫩逼图 工藤美纱qvod在线 WWW_979AV_COM 苍井空の玩具rmvb 日本美女性交图淫香淫色 强奸援交女艳照门 舔她的嫩心 强奸乱伦3d大奶网 色顶综合论坛 屄屄的粉红肉 ujizz姘撹夫鎴 帼鑱h仭 欧美老头做爱 丝袜熟女gif 曹查理 三级 狠狠射狠狠操色妈妈色姐姐 日本女人被干 人体外排人体艺术黄色片 寂寞少妇被插的一浪一浪 色站图片看不了 美女在酒店内射 奇米网下载av的 av爆乳公车ed2k 曰本全捰人体写真百度图片搜索 免费涩情小说 伴娘满足摄影师 同时为伴郎和摄影师服务 口交showtime 肏妈啦 田韩a片 kaobi动作片 日本成人强奸乱伦电影 刘晓庆的屄 人兽坏弟弟 成人图片成人视频 日韩户外大胆人体艺术 内涵鸡巴 人体美女阴 小穴穴进进出出图片 舔范冰冰玉足 菊色宫嘻嘻色儿女 妈妈的嫩穴洞 大胭人体艺术 百性阁撸撸侠 bbbbb666 妈妈和哥哥性交 黑石塔地图 周淑仪 wherewereyou 昵称网 臀肥骚穴图片 搜查宫先锋影音 快播毛屄屄 大鸡巴轮奸淫荡女儿 小说区淫妻交换小说史 操姐姐的小屄 西西热艺术 女主播在主播室被肏 激情明星合成论坛哥哥妹妹 欧美最大乳房人体艺术 如何操逼才舒服 性爱小说网视频 熟女骚屄姿势视频 91快播电影国外处女 台北聚色网 偷拍自拍论坛 u性爱网站 马六人体美鲍 张雨欣人体 我和2个女同事做爱 老鸡巴同志小说 熟女的角色扮演性爱快播 WWWKANDIUCOM 男男无忌 少妇偷情亚洲色图 武汉17中操操视频 在线看大奶人妻 老荡妇高跟丝袜足交 爱北京熟女 美女美学穴p WWW51MM520COM 男强奸女做 双飞小骚逼 张柏芝婐照吃鸡绝版 WWWUU11COM 我和表姐偷情 公媳乱伦影音先锋电影 泰国妹舔好 五月天四情 赵世熙年龄 欧美老太女郎丰满人体艺术 淫色影香 e416355f00025f2c 父女两性插入乱轮 新女体洗澡 WWW69OOOCOM 蒲天杯钢琴大赛 成人电泓网 金正恩操逼 性爱色站 乱论中文影院幼女 苍井空qvod电影在线 亚洲无毛穴 女明星的外阴 欧洲伦理小说 qvod亚洲东京热 omeichengrenwang 爆操舅母 黄色我的老师 动漫美女穿黑丝袜能看见洞 俄罗斯学院色色撸撸 黑人肛交亚裔女 性生活duppid1duppid1 不愿意露脸14p sjp成人动漫电影 林雨欣小雄性事全集mp3 色女图区亚洲色图操逼 香港龙虎豹五月 欧美性爱淫色 肥乳肥逼 瑟瑟性交图片 美同十次啦 裸身美女dongtai 无水印超大胆图片 骚妇性艺术 亚洲欧美卡通动漫偷拍自拍 爱玩老爸大鸡巴 女性bt图片搜索 爆操少妇骚贱逼 岳母的红裤头乱伦小说 在线观看色色影院无需播放器 淫妻交换色小说 美女小穴19p 一个样先锋影院 骚逼图欧美 男人社区 男人尻屄鸡巴拔不出来照片 熟妇操p百性阁 狠狠操幼女 张筱雨私穴 日本十次啦 长谷川凉子 我爱看片台湾永久app 亚洲成人女子偷拍图片 操老网友自拍 欧美日韩熟女变态 WWWSESEOCOM 五月成人小妹妹被射电影网 去哪里找李宗瑞视频 插逼微视频 好色小姑 丰满熟女迷恋 韩子萱抠逼 最大胆美女阴道艺术图片展 成人性爱电影母狗 大鸡吧干小姑娘 依依淫色网 黑寡妇黄色小说 幼幼潮吹 深圳龙岗鸡婆电召 快射电影 白色天使电影 下载 欧美激情校园春色www34qfcom 中文无码字幕qovd搜查官 老奶奶射精 花瓣床上美熟 中年夫妻作爱射精动态图片 小说乡村留守女人滥情 欧洲女亚洲夫妻炮 小女孩阴部视频自慰视频 av妈咪 t偷拍小电影 百合真人在线视频 天天影视jiatingluanlun 狼人专业维修 欧美男女性抽插动图片 亚洲性l大爷视频av 德国熟女大妈 东北乱交 水野朝阳丝袜凉鞋诱惑先锋影音 东莞小姐全集magnet 理伦乱伦网站 军哥哥操我 巨乳人妻催眠 xxxmobimediaweibocn WWW916RRRCOMCN wwwtr6688net 日本色倩女星波多野结衣 奇奇热奇奇色妻子撸 涩涩淫淫 91porm自拍我爱我妻 wwwhbmaocom av专卖店微信 欲表姐一家 撸色阁 丁香五月香 强暴小说成人动漫 撸撸啪啪啪撸nanrenfulicom 韩国女主播青草超碰 巨乳美女自拍自慰 操逼哥哥操小妹妹 无敌先锋mp4 wwwsaozivipsom 强奸乱干 影院先锋h卡通 先锋资源武藤クレア 大香蕉yinminwang 中国一线女星性交图 动漫另类亚洲色图 口交技巧mmissno1com 中国激情大片免费 最新巨乳波霸pppp38com 妹妹援妓git 日本大胸熟女妈妈在线视频 WWW123hp下载 乱伦美少妇 18岁人妻少妇口爆吞精 掰穴写真 亚洲色图丝袜美腿丝袜美腿偷拍自拍 激情网站五月色 青青草AV在线视频观免www388crwcom 幼幼av无需下免费看 骚熟丝足微博 www88qvqv 妖女内射17p wwwkkyuxguin 淫荡丝袜老熟女 超骚97超碰在线视频 一路路向西2在线完整版 内衣妹妹扣逼 腌也撸日日撸 人与兽的性事黄色网站 ggbb日本一级黄色操逼 亚洲色图幼y 长谷真理香bt资源 欧美女人与狗ZXXX 国模私拍gogo人体艺术 香港妹超碰 免费空姐撸管视频网站 操新疆女孩 东热在线视频女子大生 sesxxxxx 亚洲男人第四色婷婷 黄页网站成年人香港赌场黄色电影 妖怪黄色小说 av在线人妖和美女 武侠古典在线理论三级欧美激情 男屁眼被曰小说 淫斗罗 超碰人妻人人碰5533tcom 先锋影音官方下载 色色男奇米ckplayer 进逼眼图片图片大全 欧美av夜夜干夜夜 梦到舔小女孩的逼 日本肉肉美女阴毛左山 WWW_ADY9_NET 全祼大胆下体图片 嘻嘻女大人体艺术 sejiejie导航 韩国女厕所偷拍影音先锋 免费有声小说网站 猎国 有声小说 樱井莉亚电影美愚 樱井莉亚search酒吧 樱井莉亚松岛 小泽玛利亚btdiz 求无毒h网 求可以看的h网 在线观看的h网 手机快播能看的h网 可以看图片的h网 h网视频 www黄色小说com 开心五月激 东京热系列图片 五月 酒色网电影小说 下载看黄片 黄色小说在线观看 婷婷五月色桃色激情 爱川美里菜 织田真子 双叶美佳 性感内裤 一起做爱 ass69 哥哥色高清 蝴蝶谷影视 喜爱色社区 性乐汇综合 一本道AV 一道色导航 重武器女孩 女色无罪成人 台灣佬娛樂網 我爱我色成人 酒色鬼伦理资源 90后av(荐) 买春堂G谁有E谁有E 910668快播电影 xxxymovies 熟女乱伦网qvod电影 哥也爱 色一把 深爱基情网 天天基金每日净值表 无码支付在线播放 午夜AV在线观看 超碰在线关晓彤0 滨崎里绪女同全集 无需播放器的av无码电影 水梅公开超碰在线 外国色污视频 500性福利看片 聊斋艳谭17影院 资源网丝袜 亚州视频二区在线视频 我爱五月色 暴风影音怎么下载A片 日本地铁av电影 青草2018CK在线观看 日韩高清无码午夜 magnet 蒂亚AV资源 午夜男日B视频 snis623 在线播放 放课后无码观看 皇家Lu23 男人天堂2018亚洲男人天堂大香蕉 22 6ppav 邪恶天堂第99 狂欢a片 WWW6666SQCOM 三kkkk xhatmer 18 F2DZY 男人天堂在线福利2019 亚洲日韩 国产自拍 在线视频 情侣自拍内射 迅雷下载黄色视频 magnet 强奸舒不舒服 秋霞电影社长夫人 全球热门视频 magnet 强奸之夜视屏 性交内涵视频 秀玲叔嫂 在线福利视频广州富姐 榆次炮友 御姐成人纵欲视频 日本视频在校妾 玉桃园毛片 日韩主播高清福利 日本人妻资源下载地址 SNIS-430 大香蕉澳门皇冠国产自拍成人 韩国成年在线视频 性爱自拍直播视频 外国福利在线 快狐成年app 鸡鸡插嫩屄的影院 宅男影院xo 人人色人人干 五月婷婷心爱 四级剧情美国 magnet 华裔张丽精选11 magnet 滨崎真绪 西瓜影音 色大姐五月天丁香 苹果手机a片 怎么查询最新得AV番号 亚洲色综合伊人色 7080wcom手机伦理 米卡本子 www wf96 com 曰本性交派对 国产性高潮自拍 mp4 saobo下载 一本道手机在线秒播福利 免费成人激情视频 www4hu998cam 狼友 国庆 福利 提示:点开黑屏或白屏缓冲五秒 [红包] 福利免费视频 [红包] ht 张伯芝自拍在线视频 村上丽奈三级视频 京香在线视频p 葡京夜夜夜 77yy伦理 猛干黑丝袜老师 在线观看 日本一级大黄毛片 美足福利小视频在线影院 少妇美女主播应狼友要求半夜路边勾引 欧美色一色大香蕉GV 日本做爱无码动画 mp4 高树零磁力链接 巨屌性交 九九爱爱视频6re 微拍福利247 软妹在线福利 国产 同事 校园 在线 李宗瑞奇奥网33 一本道大香蕉伊人线av 95福利视频 日本色一情 依依亚洲图片去哪里 国超福利视频免费 百度云黄片 《鬼父》全集在线观看无码 爽歪歪色视频 姿势强化操视频 最新艺术片快播 5侧所性视频 阿V小视频 超碰国产思瑞 德田重男和儿媳妇嘴对嘴喝酒 淫妻自拍艳照 我的世界中国版宣传片 av免费在线手机 与和尚交缠的色却之夜 在线 av川村真矢在线影片 怎样免费看欧美性爱视频真人秀 美艳娇妻肖云韵 免费福利成人 77色女 何殷纯个人信息 ganmimi 大哥综合站 最新地址 韩国车模番号 wwwxiai09cm 牛牛碰人人础免费视频 av手机日韩在线 午夜av影院免费播放版 丘咲エミリ nnuu66日本系列 免费任你日 日本处女-视频@Here 偶偶福利私密视频 18x同学伪娘到厕所里射精动漫 操碰福利视频 star534 ftp 亚洲日韩激情在线 二本道AVDVD 在线av日韩经典 亚洲图片京东热av 剥皮人魔BT迅雷 91男人天堂91福利社 水中 五月丁香 水岛津实蓝衣地铁快播 影院在线协和 色老板在线影院观看2017 成人视频5 yingshi 399 空姐GIF av色中 免费av电影微网站在线观看 亚须希磁力 下载 动感小站福利小视频 菲菲影院 厨房塞蔬菜番号 大黄片福利 东方在线aav视频 充气娃娃A片在线观看 番号鹌 初美沙希初裸写真在线 大胸美女和黑人爱爱 日本专区无码视频3166 rbd的系列人妻在线播放 48号缚师 神马电影69小情侣 2369小电影 苍井空毛片免费 myloved视频 男欢女爱视频录像 大肚人妻孕交视频 nannuzuoaihuangsedaqquan 亚洲女同视频 k视频手机在线 无码乱操 66BAB视频在线 d群交 番号 推荐 富姐血柠檬宾馆调教视频 亚州黄色无码视频 真实迷奸大学校花一线天嫩逼 www609ee 大香蕉伊人视频免费整 欧美高清h 日本高清aaDVD 色久久久久 第四色网男人香蕉 谷露影院在线国产 caoporen公开视频在线播放 91情侣理任在线 九州av–男人的天堂! 丁香五月天小说网 伊人成人电影色大哥 WWW,A片 AV小视频在线播放 青草a免费线观 日韩高清无码在线视频 ftxx00 sayaka fukuhara 4480青苹果影院免费4460 西田麻衣高清无码视频 长沙丝足调教 xingchashipin 国产老熟女大尺度自拍 偷拍自拍在线看100p www37ibcon 福利电影tcn 欧美高跟鞋射视频在线 馒头B紧身裤视频热舞 深夜福利无码小电影 sao360 日本av视频欧美性爱视频 下载 成人久久午夜电影 草莓论坛 se cop在线视频 成人影片不需下载gav 星野遥电影手机在线观看 丁香婷婷五月天小说 破初系列在线观看网站 悠悠资源色 日b视频过程狠狠色哥网站 26uuu最新亚洲欧美在线 南日p无码午夜影院 东方影库av无码在线播放 换妻一族电影 幼女AV 媚药女同按摩师在线观看 zzjiyou 韩妞在酒吧被黑鬼下药 泷泽萝拉在线播放教师 69堂在线看草莓 美丽坚共和国 草b大片免费的 森下真衣 视频在线观看 草莓视频在线看绝斯斯 日本69式视频有码 ADy无码 你懂的直播免费 日本无码丝袜 佐々木爱美 磁力 口交内射视频 校园 都市 欧美 自拍 磁力网 武侠古典之萧历 letvclient://msiteAction?actionType=9&amp;pid=&amp;vid=21286812&amp;cid=30&amp;zid=0&amp;ver av毛片在线观看直播 白洁 高杰 欧美性爱大白屁股娘们性爱视频 免費高清視頻一色佬 影音先锋阿姨不约 最新av电影在线 有声性说 涩66 日本吹潮在线观看 含羞草大人影院 黑鹰坠落h版5060 青娱乐色琪琪 色色鸟亚洲 极速在线 欧美 亚洲 偷拍 王梦溪 迅雷下载 和99re一网的网站 操空姐 avzon kjfuli福利视频 经典 小鲜肉苏州94 香椎 jav anal hd ady手机 5533992c0m 日本重口网站在线 小视屏福利网站 性爱动漫福利 小老弟视频精品 国产自拍 小妹自拍自慰视频网站 学生妹被艹视频 先锋eeuss 小清新影院性高清视频 校园另类自拍欧美 邪恶里番肉番 馨雨女神调教视频 筱慧视频在线网址 泄欲哥导航网址 中国清纯大学生默默 www84gncom 女耻物 明光大尺度歌舞团 2018在线看的视频你懂得 免费无毒福利 狠狠的艹免费视频 国产网红福利 欧洲老妈A片 www,俄罗斯,幼色 老外影院黄色 bl插插电影 微熟女在线播放 兴奋生中出巨乳交配 橘优花 www8733cc 插菊花综合网人妻 4k 在线福利影院 莉莉影院俄罗斯少妇露脸 午夜群交视频 千百噜噜噜影片 美女强奸啪啪啪视频 任性操 做爱视频教师 日本美女重口味啪啪啪做爱视频 想要零用钱妹妹 素股 日本强奸乱伦在线观看网站 淫乱视频操屄 欧美 日本 国产 导航 激情黄片超爽 1小沢在线播放 哥去射偷拍自拍在线观看 最新97视频网站 94套图吧 阿，我要被你操，被我舔 深圳同居换夫 日本在线高清m949dtv 少妇爱爱 大香焦网视频免费视频,i xp123亚洲影院 熟女 博彩 wwwmvm888co 福利宅男影院免费视频福利在线看 黄片影院。 uuu778 mp4 综合网人 亚洲无码中文字幕成人动漫 大箱焦成人网 &gt;&gt;宅*男*影*院&lt;&lt; 西瓜影音 王思懿金瓶在线看片 在哪里可以看h动漫短视频 印度av视频网 drp无码影院 国语自拍对白在线 操逼比赛 苍井空无码av种子磁力链接迅雷 操逼故事在线视屏 韩国成人主播 双飞 不用下载安装就能看的吃男人鸡巴视频 北京熟女取精2女上位 变态调教性奴视频 avop-360 欧美三级伦理大尺度 马牛叉电影 搜人体苍井空50分钟无码 男人装 明日花 大香蕉AV在线播放 丝袜av排行榜前10名 无码黄电影在线 影音先锋AV有码丝袜美腿 2018最好的塔巴夫影视 苍老师在线55集 亚洲制服无码欧美 大贯杏里AV百度云 怡春院偷拍首页综合网 www 702qq cnm 被窝午夜手机自拍福利视频 Thailand年轻的制服高中女孩淫乱生活传闻风波 玩弄淫穴 无套爆操 爆精内射 附高清 vv影厍 操逼小视频在哪下载 tayelu免费视频 播播影院女性向 操后妈6o分钟 成人在线动漫 黄色a片在线免费观看 我妻如妓 秘社mm视频 纯洁 亚洲 国产网友自拍偷拍视频 连袜裤javlibrary 万色吧影视 1啊无套清晰 日本日日夜夜bb 黑木一香 magnet i日本人69种视频 日本aⅴ视频天堂肮脏医生 熟女大阴户视频 爱区 看片岛 vr捆绑美女 92午夜免费200部 深夜做爱视频在线观看 国产成人福利 magnet 澳门色片 av女捜査官yingyuan jb影院下载入口 日本漫画之工fanmu angelbaby19部 在线播放 变态搞基网站 先锋熟女少妇 vip7116韩国电影 黄色电影院六度电影老女人 欧美同性视频vibes 欧美在线天堂视频一本道 伦理 大片 高跟丝袜女视频 后插资源 波多野结衣末剪版在线观看 苍井空 在线播放 8p 肏女人 日本姊妹同 福利fl218 久久爱免费福利在线 男女性爱傻拍拍视频高清 伊人大香蕉在线视频网 综合色爱视频 肉色丝袜爱福利在线观看 偷拍自拍第八十五 美女真播母乳真播 西瓜影音 曰本无码在线 345bkcom 手机亚洲mm88cc 一本道dpp 美国理论午夜十二点 后入式xoxo免费影院 日本av女优视屏 黑裤袜系列在线 斑斑马电影街 欧美图区 国内自拍 强奸乱伦 我爱你AV52 亚洲av剧情 立花琉莉在线网站 Caoporn网站 午夜福利757在线视1000 ssni048在线 男人的肉棒插进了女人的小穴里不用给钱的视频 琪琪在线狠狠射 九哥小浪窑网 2018天天干夜夜啪天天射天天日影院久久99 麻美由真 bt字幕 日日久久天天 k卫生间男女ⅹⅹoo视频 CD性爱视频 天天操天天玩 调教堕落中日韩美女 激情戏新视觉 尻屄怎么读 边玉洁年轻时的照片 老外干十岁女孩磁力链接 黑丝少妇影音先锋 a片一线看 宅樱三级片 欧美女孩w18 捆绑女奴 很黄很色的动画片在线观看 wwwqiuxia66路cam ipz483百度网盘很黄很色的动画片在线观看 丝雅电影网伦理片在线看 美国巨乳波霸护士 美国邪恶毛片 色色琪电影 少妇操视频在线观看 美国十次大公鸡 毛片观看福利影院合集 美女被xxoo奶摸 伦理片eeuss2θ12手机板 噜噜色噜噜吧琪琪网 人妻凌辱参观日百度云 气质美女经理酒醉被上司带宾馆各种玩操自拍流出 极品巨乳美女高潮视频 精品成人A片电 染上春色在线播放 激情视频无码丁香五月 激情影琓 自由 黃色片干姐姐3131 麻美由真 恶父 白妇少洁的小说 福利视频午夜小说自拍 卡在墙洞男孩汉化版百度云 百姓阁不夜城最新地址 日本少妇Her a,ji ,huang,pian 秋弄韩 百合润美 magnet 欧洲av成人在线 蛋壳姬 岛国种子搬运工官网 吻逼抠逼视频 任你操视频这里只精品 无码专区 大xj香蕉49 精品国产自在线拍400部 黑人巨大战原纱央莉mp4 欧美熟妇系列1032 女主女王sm视频免费专区 横山美雪视频在线观看 六间房黄带 国产av自拍在线 www@pziyuan@com 口活最好的番号排名 哪个网站能看小泽玛利亚的视频 下载 得弟日001 二奶夺位 下载 www3133dcon 日本′电影强暴女搜查迅雷哥 亚州高清无吗不卡视屏 狼好色日韩高清视频 高清国产牛牛碰视频 凹凸视频分类在线a一 一级毛扁L 高老庄成人在线 看片 酒店偷拍 日本AV群交游戏 男的精喷如泉的一部av 手机看片秒拍修复 bbb977改成什么了 男人天堂噜 AV日产 1电影天堂EEUSS 萝莉小电影种子 ftp MSWD-10023 ftp snis911下载 射大嫂在线播放 浅香结菜 国美女主播福利视频 韩国vip主播无码 国产主播自拍磁力链接bt种子下载 国产在线短篇 拿男女做实验的电影 国内酒店女生骚逼视频 国外无码ay免费视频 江疏影 口 交 视频 欧美Av色中色拉风影院 自拍porn87高清日本mp4 国产农村夫妻啪 国内自拍网盘 色B心 自拍偷拍激情国产 97高清影院 亚 无码性视频 动态做爱LOⅤE 长筒靴做爱 magnet 美国成人午夜片 铃原爱蜜莉视频在线播放 欧美激mp4 日韩自拍在线高清视频 yyyfuli 最美肛交无码下载 下载 视频一AAVV、com 操同事的小女友爱剪辑 wohejiaoshimamadeseqinggushi 77mp4 sixt 偷拍视频毛片日本 国内自拍女厕小便 珑泽萝拉无码av下载 杨幂 醉酒 1024你懂的 污漫韩国漫画57 艾迪av无码 亚洲第一成人福利网 shkd-744在线 豹纹内裤影院 黑人熟妇视频在线观看 爱爱无码视屏 黄色视屏免费看在线 卧室五六式老太太每个人的阴道毛视频 花蜜性爱小说 heniaoxiaoshuo 外国美女性感穴图 骚屄性交还录像基地 插动漫美女动态 97sese图片区 我让老婆找大鸡巴把她日到高潮 草裙裸体艺术照片 我和门口的阿姨做爱小说 日本女人与男人性交图片 男子艺术图片 偷拍自拍激情小说迅雷下载 老人与小孩大性交 快来干我小说 人体艺术下阴 孕妇母乳先锋 欧美色图一巨乳 中国山西忻州岢岚美女脱了衣服内裤大奶屁股露出来上床b的图片 口述6p 看真人操屄视频 日本女老师写真 男人日女人下面的视频 酒色网婷婷五月天 良家肥臀 俄罗斯色妞 成人大尺度gif 极品美女影音做爱 姐弟性交大赛无码 口交动大鸡吧 熟色201412 91热色色色 亚洲获奖a片 好想看看美女的写穴视频 舒淇人艺术 美女无衣自拍图片 刘嘉玲大胆人体写真图 gege揉搓 欧美金发熟妇图 超级mv性交大战电影 青沼知朝番号 欧洲色图色福利 继母阴道淫荡风骚 撸死你资源站兽交 250ppcum 成年美女图 女女调教电影 9797滛滛网 妹妹操哥哥日 营野沙莉亚 操淫屄网 婶婶的小屄 abcdduppid1 44tttt 恋古成人图片论坛 成人激悁大?6?4 操逼张慧敏逼照 强奸乱伦飞 夫妻性爱录音 操逼操电影网 性爱动态图片15p 美国玩中国女人的图片 WWW_5555K_COM 免费论理片 色幼女性交导航 色播五月天亚洲图片资料 321操逼网 人体裸体艺术美 乡村色妹性交视频网 曰本性爱第一页 丰满少妇给我足交 外国投拍美女上厕所视频 性爱图片做爱电视 影音先锋母亲性交 中出し 东めぐみ 147美女人体艺术波 女子逼毛 umei 丝足 � 日本美女性交欧美色图偷偷橹 日本黄色成人很黄的 亚洲图片小说网 幼幼百度云种子 WWW_YULE_COM 姐妹色 人与兽黄片视频一级片 超碰caoporn成人 那个软件能看白片 va999资源网 动漫黄图小说 中国人体莎莉 女人淫秽乱伦 俄罗斯明星人体艺术 谁有熟女网站 射极品空姐超碰 日韩柔体性爱影片 wwwre999com 李宗瑞不雅视频文件影音先锋 欧美 亚洲 娜 WWW_4HU50_COM 色小说乱伦故事 chengrendianyingxengai 百度美穴 WWW_168_INFO 春暖花开影视性吧 泰岚性感 xiao77论坛新年 极美印象苍井空奇色 天使妹妹淫荡网 婷婷五月色中阁 师姐淫屄 qiangjiancaobiwang 海岛人淫乱 日本人t体yishu 大牛村群 美女艺术照 欧美骚妇和狗交配 色人谷美国发布站 农场性爱在线视频 内射美女小嫩逼 欧美入逼色图 赤裸天使讨论区 欧美大但pp图片 97sese 97ai 97gan 美女人一人体图片 明星漏逼逼 偷拍自拍区操老肥熟 偷情人妻骚母狗淫语对白说要被狠狠的操 情趣连体 超级诱惑 tube24韩国 亚洲淫淫碰 肛交567视频在线成人 女同学肛交小穴肛门草 美女色图qvod 肥奶大逼 俺去乱了 西西大胆人体艺术黑木耳 经典av片名字 成人图片 鲍鱼 快播怎么看无码电影 欧美淫秽影片 傅 贞怡 人体艺术 色 中 色亚洲色图人与狗 花开半夏演员表 中国净水器供应商 重庆赵红霞真实照片 辛亥革命电影 温州第十二中学 淫香淫色天天色 黑人和韩国女人做爱 激情少妇乱伦 3d玉铺团 偷情老婆小说 办公室操我 骨感小骚货视频 哇嘎影视四色 苍井空和黑人拍的av是9那部 俄罗斯裸体体图 我想操比找美女 欧美性爱人与猪 美女裸体漏阴图片 插入小妈小穴 西西处女人体艺体欣赏 色图五六月天 免费先锋成人网 处女搞屄 当母音屯 qiangjianluanlunying1 免费黄色图片网站 逼邪恶漫画 新乱伦影音先锋 凉拌木耳女生每次最多吃多少 狠狠地mv撸五月天 女子身体艳照 电视剧激情图 国产妇女做爱视频 搡大白逼 性交乱伦p 嫩逼的女郎做爱图片 26eeee 色444电影 三级片一thunderftp WWW_888XXHH_COM 牛牛视频播放器护士 治阳瘘早泄的中药泡酒 熟女人妻裸体 长谷川惠美全裸图片 夫妻性爱偷拍照片 美轮美做求歌名 WWW_678BS_COM 日逼的事情 骚狼操淫妇电影 日本av女优馒头逼图片 北条麻妃快播影片 2014最新乱伦强奸电影 幼香幼色论坛 女人与大黄狗做爱 淫乱的少妇 欧美黄色片午夜剧场a片 大色窝色狼网夜夜激情 看淫荡老婆群交兽交 国产熟女在线视频 欧美母猪和人性交 偷拍自拍武士色 小妹屄图 色站导航欧美色图 爱爱射欧美图片 美妖操逼 巴士超短摩擦色网 女人与公狗高清rm yazhouxingaitaiwanlao 在线自拍亚洲欧美 山斗香港三级 欧美日韩偷拍 高树玛利亚bt 抽插熟女撸 WWW38XTCOM 大黑油葫芦交配 伦理熟女人妻 色桥人体艺术抓美网抓拍美女美丽 最逼的中国演员 伦乱透屄故事 中学屄吧 野外大胆裸照 国外天体海滩视频 日韩影片北原多香子 那晚小姨子把我狂吸 人体彩绘图片大全图 美女裸体劈腿照片大阴唇 女人掰逼图百度 直走丈夫买烟妻子被李书记蹂躏 亚洲陈丽佳人体艺术 美女草比视频 屄的写真 聚色冈样 日本66人体艺术摄影 亚洲美女白虎色图 WWWOTALUMODECOM 性花宫黄色成人激情 亚洲最大成人操穴电影网 宬人电影在线播 春满四合院视频 韩国美少女主播荷恩 操白百何的骚逼 欧洲色图激情网 va看逼 强奸滛乱 曰日撸小说 波多野结衣作品那一部好看 成人片撸一撸 淫淫巨乳合网 heiguigan 色妹妹a 性感女妺妹 5x在线视频 xxoo青涩综合网 俺去也影音先锋 乱伦奸幼性交小说阅读 6699色图 一夲道京东热全部电影 女人被强奸时的反映 迢碰av 极品美女骚屄 俄罗斯乱伦抢 成人性爱图片网址 永旺cgv影院 裸图b美女 撸撸成人动漫云播放器 戴文青木写真 日本人妻斩先锋影音 看女人色图bbb 女优希崎杰西卡黑色丝袜人体艺术 老妇少男忘年乱小说 鸡巴尻小屄 uu小孩逼11p yy6029新视觉影院官 爱人体静雨 ok5858com tangfangdadan人体艺术 近亲相奸お母さんと爱欲性交 少女性奴母狗血泪斑斑调教 幼女小嫩逼10p 岳母乱伦淫荡肉戏故事 媳妇叫公公摸奶 ppp36官方网站 嫩馒头鲍色图 WWWHDCCBKRCOM cccc80 内射小妹妹图片15p 美女大学教师视频 狗茎欧美女人 WWW55HHCOM 男人阴茎实图高清 5月激情乱伦 新福利影院宅男 老妇的肉逼 强奸乱伦少女破处 学生性爱视频 欧美少女做爱美图 藏姬阁第一福利老司机 欧美电影家庭手机 三邦美女视频在线观看 se鲍鱼 福利嫂超碰 厕所阴唇 邯郸性趣天堂 色尼姑免费原官方网站gya1024cn 被催眠的校花小柔 av导航全球 老熟妇好爽 日本近亲乱伦中文字幕mp4 色播五月天永久网站 扒开屄操 swww99aaww 黄视频体验区 自拍偷拍性爱视频www16efcom 欧美男人与狗兽交 飞华两性试衣间 爸爸女儿乱伦妈妈儿子做爱 亚洲男同性恋做爱图片 视频裸聊女同骚逼 群交谷城 日本女优恋足视频 JJ射精正面图片 乱仑吃屄 亚洲熟女图片wwwgzyunhecom 成人色片电视频 成人电影哥哥碰 最新人妻av在线网站 伦理微电影magnet 巨乳爆乳大奶子插肉穴小说 和巨人美女做爱 123gbgb男人看的地址 强奸乱伦校园都市激情 快来干我老婆 性交短篇小说 AVMO欧美 s级女优网友自拍 手指插花核小说 成人做爱乱伦人体艺术 绿母漫画真人版 青青3p 色欲母子性爱小说 家庭伦理小说网址 奇米影视手机版百度 夜撸va 成人小说快播电影 美女干逼人体艺术 meinvluotitu 日本三线片线观看 逼里逼里香哥哥 jizzjizz丝足 爱爱激情五月 香港三级片系列激凸男女1高清在线观看四虎影库 欧美搞B片 哎到花心了啊 美女大胆露穴 谭晶h文 野花撸进不去怎么办 那里可以免费观看三级做爱视频 操台湾妹子综合网站 动漫人物h小说合集 另类女同群交小说 爱爱扣插舔 大奶妹官网 为什么收不到韩国女主播青草的视频 淫妻乱伦龙腾小说 播播影院色播五月先锋资源 苍井空潦草av片 久久鲁免费观看到底 三级片三级片的坏的视频播放 巨乳女教师magnet 美女黑木茸百度图片 老女人的淫 风俗娘湿 亚洲少妇xxoo 搜色电影影音先锋 插妹妹日妹妹dj520netwww78p78info 光棍电影网限制 用力插妈妈的骚逼啊啊 日本捆绑性爱 四虎影库A片 免费网男人草女人 在找黄片人与兽 可免费观看激情全过程的APP 纯妹妹极品穴穴享用整晚 侏儒a片 风流少妇人体艺术图片 快播日本乱伦 天浴电影黄色 风骚日本老师下载 柚子塞屄色图 sm妻子捆绑兽交小说 乱伦小说意淫强奸校园春色古典武侠淫妻交换 我的女朋友是40岁的人妻 www84ytom 日本春药按摩 女人玉门高清大图百度 AV电景色网 免费在线欧美AVwwwc5508com 微博btdownloadbaiducom 公公大鸡巴太大了视频 少妇五月天综合网 巨乳丝袜制服中文剧情字幕 谁知道手机能看的片 人性交配色色 快播欧洲成人套图网 保定色尼姑色和尚 西欧拳交 haosex a片99kk511cnarrseseappsitewww90abc95w4cn 亚洲级欲 日本妈妈v淫 少女b图片亚洲色图 xiongmeideseqingyouse 自拍偷拍彭娜 亚洲色图暮色吧 成人一级黄片网站 elluxurycom 爱碰网官网 操丝袜姐姐小说系列 性交姿势wwwcnkangcomwww7788xsnet sss电脑版 苍井空和男佳也那部 裸体模特在雪中 被大吊干是什么感觉 大嫂激情图片 菲菲肉穴 变态另类6eeeeecom 色斗鱼wwwsedouyucom AV网站www520vodcomwwwxx557com 肏了小骚屄 xf77piay 说几个看幼幼片的网站 哥哥干网友骚货 苍井空超过激交大乱 淫淫网刘晓庆裸图 苍井空的私处照 酒色五月丁香 仓井空老师剧照 教师无码潮吹 张柏芝的桃源洞真嫩 kk99se访问升级 偷怕自拍撸 矢吹春奈 强奸处女口述 5566影音先锋干大姑 强奸伦理小说视频 国产自拍剧情在线 求推荐可以看高清色图的网站 998av 成人动漫阿凡达 寂寞妈妈免费视频 奇米影视色资源 堀井美月av 欧美人人免费视频在线观看 美国a片啪啪 WWV111WeCOm 小穴潮吹什虐阴么 五色房间播播影院 wwwqingjunlu3tv Aa片影院播放器 男人的av影库 一级片视频 小妹子论坛资源分享 色图片图片区 跟女朋友啪啪直播贴吧 第九月激情网yuyongniancom 美女馒头缝 H幼幼萝莉图片 成人无毒网站 阴布图片大全 空姐被黑人插 高跟美女 黄色电影姐姐妹妹是拉拉 亚洲免费快播网站 黄色电影A片毛片 51cC0m 欧美双插电影 欧美男女av影片 4438五月天 朋友淫荡的妈妈我可以操你妈妈吗 色玖玖AVcom 双性黄色人电影 老外的大肉捧系列小说 免费的黄网站网 老妇成人图片大全 黄色图片观看 995cc 人体美女床上诱惑 36d色图百度 透逼自拍 无码有码在线 丝袜15p 欧美激情狐狸精 成人电影AAA990 天天啪久久wwwgeerlscom 哈尔滨色 全程露脸52岁视频拍摄 wwwqyuletvagmailcom 激情校园春色老师 亚洲色图韩国明星潜规则p 捆绑强奸娇喘视频 西门庆导航网址 青青草针对华在线视 久久热时间停止器视频 淫荡熟妇16P 亚洲视频在线成人 搜索wwwweibocomu5405295901 好了av第色 爰看AV电影 啪啪啪萝莉被操 亚洲露b人体艺术 东京热一本道金8天国 久久热我爱看a片 自拍偷拍网址wwwhhhh15com 微拍福利wwwmiwpscom 免费看天天A片美女图片免费 狠狠爱十狠狠干十狠狠撸日 青青草手机在线vip免费 桐岛惠理香 美女红色曰逼图片单曲 放个国产韩国日本美国欧州男舔逼逼女舔大鸡巴把鸡巴扎入逼里猛抽猛扎放个最黄 ckplayer在线亚洲 肉文ed2k WWW460TVCoM 9911影院 在线电影选项院 日妹妹干妹妹影院百度 wwwbb698com wwwnv81com新网址 青之驱魔师cosplay 大爷操作影院388sesecom 幼女ayi 操嫩人 人妖操人妖电影 大鸡巴干性感美女 操妹妹插妹妹AV 东北夫妻4P艳照 国产精品偷拍自拍明星系列在线 在线播放国产偷拍 姑坏姑集百万潮流 抽插淫荡少妇小清15p 和公公尻穴 一本道av在线三级 母狗奴隶 黄色网站草莓影院 裸体姿势艺术诱惑 操久久 幼女动漫下载迅雷下载 成人电影之手机偷拍 嫩穴妻 al7788 鸡巴视频国产在线 yc15电影com 阿v对白 妈妈的丝袜20P 毛片基地SM另类 xxxx欧美制服 swwwpu730comhtmindexhtm 大吊性交 大尺度激情性爱美女 图片专区欧美色图 强奸骚货女朋友 影音先锋咪咪色 熟女人妻校园春色淫色人妻 www48xycom 皇色王朝2356 辣文娇妻系列小说 激情婷婷第五月婷婷强奸乱伦 泡幼女网 就爱就爱娱乐网 5577k 亚洲色网偷拍图片 播乐子超在线 10万部爽片等你欣赏 看美女裸体视烦 武则天H版 性色影视 天天拍拍国美在线视频 www48kcon 碗君西西人体大胆 搜索去色色吧 淫荡阿姨在线 87bbb最新网 妻子与大学生黑子 怡红院首页一百度 黄色乱伦电影av 香港性禁片 ae517magnet 青青草的快播 狠狠操骚b 射你阴b 女孩发骚自慰 搜索www4444com 19sei最新地地址 AV在线国语对白52kukucomwwwaa0ecom 欧美人人妻激情小说与视 5XX 女人自慰囗交动物乱伦 性感美人妻电影 久草在线站街女 色色色色色干干妹妹 www路360ppcom 99pp黄色网站 国产AV偷拍视频 40岁少妇露脸久久草热在线 小衣与狗 91pro内部地址 男女性交猛抽动 调教男啊啊啊 空姐被强奸的日本AV电影 国产自拍操逼试视频 给表姐调教成性奴 哥哥色欧美激情 大香焦爱爱网 wwgaosecom 先锋影音av资源 亲脱摸扣操 av色色com 少妇性爱故事论坛 姐弟乱伦网站 AVPP6SCOM 久草在线妹妹干 114张悠雨魅惑图片色色看看色色看看主 男女性爱三级毛片 开心酒色吧 网友自拍偷拍国产在线第一页 酒店调教性奴母狗亚洲图片 Av电视在线看 Sm小说虐丁妈妈 黄色视觉黄色小说 swwwbu370comhtmindexhtm 在线娇喘 亚州强爆乱伦视频 wwwjjj2345co 像淫荡游戏这种动画有哪些 日本午夜户外直播 杏吧有你大香蕉 pcn中文网 日本女人色中色35wwwwcom 色系大片观看方法 超碰巨乳视频在线 成人套图49p 鸡巴25p WWW884aacon下载 伦理片小向美奈子 HAOXAVCOMmagnet www57bbee 大香蕉久草免费的成人视频 大乳乳乱伦 撸醒睡熟儿子乱伦视频在线 尤18TV 伦理片琪琪第8页 黄色真人网址下载 成人动漫父女乱伦 美女丝袜激情图 色婷婷伦理网站 修真抽插穴 火影忍者谢哥博客 豆豆色穴成人网 狠狠干妹妹图片 最骚美女最骚美女最骚美女最骚美女最骚美女最骚美女 WWW92ddcccom 沈阳母子 手机看av片成人电影 高清av在线无码 牛牛碰电影 姐姐高潮出水10p 淫哥哥下载 神马性爱交易 av天天堂在线观看日本 群交乱伦姐姐 冲田杏梨人体777e丫图 水仙二嫂视频福利在线 国产原创自拍996 ee655con 同性恋舔我小穴 超碰图片wwwxmxh188com 黄图的网站是什么 三妹影院 青青草制服性爱片 天海翼 亚洲 图片 色五小姐在线视频 仓井空全集百度影音 WWW_RP_INFO 人体艺术巨乳巨炮 我爱亚洲人体爱人体 劲爆乱伦故事 一女二男性爱图 日本美写真视频 处女中出 偷拍自拍美女操逼在线播放 欧美极品嫩穴 fc2日本成人视频最新 卫校女同学自慰图 台湾美腿模特视频全集 三浦惠里子作品封面 母子淫乱日逼一神爱爱小说网 藏獒吃伟哥强奸女人 草裙社区蜜祧成人快播 人体狠狠鲁 最美女屄图 狠狠挺近花心射 撸波波明星美图 一窃开一脏器 少妇裸体性交写真图片 哥哥去在线图片欧美 性爱派对图片 什么样的女人逼最紧 性吧春暖花开性爱自拍 日本漂亮女人三级照片 邻家小妹的大奶子 色妹妹来也青草影视 色迷淫色 9115视频在线资源sss 经典あ级制あ丝袜另类专区巨あ波霸清纯あ美 黄色1级piang 女优性交免费电影 77sssee 人妻少妇被 人体艺术 筱 美女裸照操逼照 为什么我喜欢让情人操我 夫妻性交真像视频 最大胆美女人体艺术 欧美色图种子百度网盘 a片强奸系列快播片 高树玛丽亚 ed2k 强奸作爱图片 site 免费在线观看日本人av 黑丝肉丝双飞乐 色史中色 超性感丝袜美女内衣视频 se情wangzhan 日本鬼子抢奸中国妇女 大香蕉狠很撸 oumejiqing 虐足酷刑 少女拳交视频 和少妇 仓井空magnet 汤加丽美女 孕妇系列 乱伦 肥女胖女3p 色图 色百合电影导航 苍井空无码先锋下载 苍井空裸体无码写真 日本少妇11p图片 日本妹妹性爱射精图 操逼视频先操逼在玩逼 少年与老太太性爱日本说中文 媚药先锋 性门照 快播肏淫妇 好看图片小说 制服 日本 幼幼 高清 丝袜肛门视频 最新尿尿偷拍 公公插媳妇逼 看80后操bthunderftp thisav新网 性交组图 成人色短小说 口述母子伦理感觉 操肿了少妇小穴 aikojiaocom 一 级黄色图片 骚妹妹爱爱图网 reitiyshu人体艺术rtys日本美女大胆人体艺术激情人体艺术 下一章激情内射视频 大鸡吧小臭臭操操操小穴 强奸苍井空 毕夏漏点 肏胖骚屄 巨乳护士 绣 亚洲大肚男激情自拍 欧美大黑棍pk亚洲美女 快播 h片段 老人头人体艺术图片 撸撸ase 操了大姨姐的骚穴 711c人体大胆 父女叉屄 老外女与动物 韩国女主播阿狸快播 做爱网子 白皙亚裔女大战老外 套图超市有声小说 35p激情熟女 淫叫骚妇 中年家庭妇女色相 汤加丽人体艺术专辑 mhaodizhi4ifno 意淫miki 苍井空百度网盘种子 美女双穴被奸 写真视频快播 自拍偷拍 露脸 夫妻 少妇艺术祼体照片 丰满裸体大奶 无码av社区网 女人正面裸体艺术 十九女孩庄媛 网友自拍偷拍天天 阿门阿前 可爱多儿童摄影 我比想象中爱你 南京江宁房产 家具风水 漫步者1000tc 想着胜利前进 东南亚女孩艺术人体 苍井空人体全集 siwameijiao中文字幕 熟女淫乱色图 五月天激情鲁大妈色播 www520色偷偷撸 波霸彭丹露奶图 金发美女小嫩屄 美人阁第四色 熟女合成48p 原千惠透明装 女儿操父亲大鸡巴小说 暴力操女 剧情新片换妻影音先锋播放 先锋影院 av 不撸 女阴部无遮挡 爆操宾馆大龄女 聊斋乱伦性事 妈妈被老外狂干 操大黑逼逼 撸撸小说母狗 金梅瓶电视剧5集在线wwwqitetecom 贵女淫乱 WWWEYHJCOMM 7788sesewang 缔D杏那 黄昏操逼在线播放 都市激情撸情 avbbcom 瑟瑟色综合 日本妹妹综合 十八岁女孩人体艺术 www79vvvcom 日本美女下体艺术摄影图 我和少女做爱视频 成人电影一级a片 大黄瓜vs小菊花 黄色录像电影片段 老富婆操比性爱 幼女性交站 yazhouav亚洲av 美女美穴图片30p 熟女精品写真 成年dianin 茂如森林的阴毛图 夫妻性爱自拍16p 青岛小护士视频 明星不于明加的不雅照 影院上映最新电影 摔跤吧爸爸吉吉影院 岳母女婿生子 太阳的后裔苏格影院 阿姨吸我鸡巴舔我屁股吃我大便 天海翼作品被截图 丰满人妖的性爱 淫乱图片草裙网 女乱15p 杉田瞳i～淫若妻妊妇 插插综合图看黄色一级图片 纸做的爱的小贺图片 照片自拍小说图片电影 透视旗袍人体艺术图片 五月天色姐妹 我和处女小护士操逼 撸图屋聚色阁 罗体美妞图 快播亚洲色图自拍偷拍 淫荡女老师教室淫穴被大鸡巴插 熟女人妻阿姨在线电影 姐夫和小姨子激情视频 红衣美女舞劈腿 成人电影av天堂网 杀戮都市吧国产各种门 解放的潘多拉 禽兽父亲绑起自己女儿猛操 欧毛毛女三级黄色片图 尻嫩屄图 日本幼女美鲍 乳大黑女人 香港掰穴 阴毛多的大胆人体 美女行爱视频 男吊写真 国产农村野台子脱衣舞 美女fs2you 疯狂肏屄视频 WWW098HHCOM 操屄小说视频 西西88人体艺术 我看午夜人体艺术 日本美少女大胆人体艺术图片 2014最新偷拍自拍 撸撸b图 女人体艺术专题博客 好看的亚洲伦理无插件 撸炮动漫 大香蕉在线观看少妇www22oxoxcom yy6029淫影院 有没有哥哥妹妹乱伦的小说 轻吻也飘然在线福利 cos先锋影音 mcc色区催眠 街射ed2k 农村大胸阿姨做爱 爱搞逼图片 5x社区一样的网站 春暖花开操逼吧 日本男女裸体性交照片 姐姐帮哥哥推精油 黄图m226wwcomnxgxcom 与淫荡女医生做爱 美屄magnet 天天鲁大香蕉网 白丝护士爱爱 足交久久帆布鞋 2233d最新2233d地址 色无极亚洲dddd88com751vvcom 美国九九九色成人网成人网站 亚洲幼女口交电影 青青草www99kk496cn 色图人妻斩 三级黄图片欣赏 wwwjizzluolicom 恐怖十八禁ftp 偷拍女士推油 南美熟妇视频 郭碧婷一级片 东京热哥哥妹妹 肏妹玩肏我呀 淫淫hhhh 720lu自拍www820qqcc 97色色强奸电子书 生孩子视频色色 两只角头上插着叶子 kangavhaole001com 欧美母子乱伦影视先锋资源 美女与动物乱伦 骚妇狠狠操插 欧美色图凌辱 wwwva6nt8wcn 姐姐精品播放器 日本成人黄色六级片 成人动漫淫乱乱伦强奸 吉吉影音乱伦小说 干亚洲女孩 东南亚老母猪15p 久久欧美视频aa0ecom 日本邪恶电影在线观看网 wwwseqingwangzcom av天堂松岛枫 欧美图片偷拍图片区勃起 骚逼逼撸 50岁女快播电影 久草热这里只有精品网 货车司机轮奸 影音先锋色谔谔 性趣阁论坛地址一 陈人日韩短片在线免费观看 伦理片伦理聚合网站 色人谷色小姐 毛毛生活片 偷拍自拍亚洲女同 亚欧淫色 jipinseyinyuan 操美女NB 内射轮奸男女小说 一楼一凤影音 老狼网最新网站 求最新簧片软件 欧美胖妞性爱电影 美国x级片女优 亚洲小说另类人妖 5252avavw我爱好色 亚洲色图偷拍自拍wwwssyy555com 亚图片综合网 大爷操影院院18k 男人插美女护士私处视频 林志玲假期去基地色图 13岁少女阴帝的图片 黄色录像同性恋口交 亚洲小说图片偷拍电影下载区 日本无毛少女逼图片 被黑人轮奸好爽 小美女的毛真多东方av 性侵人妻小说 操60老太逼对白 日本空姐艺术丝袜图片 95后大鸡巴 星戒有声小说 亮剑有声小说 有声小说小仙儿 寻春色 春色括号 春色龙 樱井莉亚跳舞 小泽玛利亚女王 小泽玛利亚msn 小泽玛利亚现今 小泽玛利亚luozhao 跪求一个h网 www.唯品网 www酷狗音乐mp3 www传奇时间 se开心五月天 开心五月天最新地址 东京热真实 看黄片怎么找 空调看黄片 织田真子 五月激情网 人兽综合 色男色女 视频下载 四房播播 我爱手淫 av幼稚园 插妹妹高清 大M成人网 骚女窝影片 色狐狸小说 四门成人网 我色淫我乐 黄网二十一区 妈妈儿子乱伦 色导航百度骚 色色3A丝袜 第四色在线电影 淫荡小妹激情网 在线视频返回顶部 撸一撸 俺去也 天天看 777米奇 色论坛 急先锋 瑟瑟爱 白虎活络膏 色妊阁影音先锋 弟必撸 弟弟干 蝴蝶谷成人 色色南 热のAV 澳门葡京城AV 大尺度视频试看5分钟 自拍直播网av 在线亚洲中文呦呦 www路avav008路com 吉泽明步女教师在线DVD 3d人妖ladybayshd 阴模人人看人草 波多野结衣丈夫去世 wwwaaak7 超级草碰碰人人 噜噜色噜噜巴噜噜网 莉莉影院正在播 女大学生 武汉情侣自拍 ed2k 韩国孕交av 五月丁香深爱基地 av美国绿色导航 伊凡综合成人 成人猫咪993ii 做出综合网 4438人成 2018秋霞理论电网在线视频 超碰无码97国产人妻手机在线 红怡影院 adyhh 日本哺乳av在钱视频 手机AV迅雷 激情四射啪啪的视频网站 欧美群交一级毛片 亚洲毛片av手机看片 青青草93观看视频视频播放 邪恶少漫画acg邪恶帝 情人添逼逼视频让我高、潮 小媳妇寂寞在家自慰水真多 强奸之夜视屏 茄子自拍在线视频 人妻操逼免费视频 青青草影院兔看 秋霞高清无码在放 青青草视频网站 迅雷下载 秋霞电影手机版八妻子影院 情趣黑丝高跟美骚妇装看病勾引药店医师 病床上激烈ML爆操 无套暴力抽插爆精内射 强奸强奸 樱木あゆ美 吉吉影音 日本三级2017大尺度 日本三级强奸 日本三级无码动画下载 日本女优深田奈奈人体 日本三p视频在线 日本性爱啪啪视频 日韩先锋伦理人妻 羽田桃子 白丝视频 hd porn 深喉电影在线 青春草原视频免费观看 无需播放器视频国产 asiafox电击 在线 一本道东京热大香蕉aⅴ 埃及艳后 av版影音先锋 Cccc74 大香蕉X影院 VOSS-069 16974视频观看 手机国产AV 阿v影音在线观看 微拍福利Av 色播153 色噜噜插 人妻3pmp4 free 波多野结衣video 我要看小日本儿看三级片小日本人太三级片。 白浆40p 全裸美女秀磁力链接 亚洲爱液视频在线 千花集林志玲 马匹窝在线视频 曼丁哥视频可以播放 女主播啪啪种子 下载 纹舞兰在线 吽哆啪啪 色午夜福利影院 本庄优花在线观看手机 直播毛片无码啪啪 91视频青青草影院 小俊资源网 色爱天天插 成人电影迅雷下载 91久了re9热在线观看 lu559 亚洲日韩无码av影音先锋 斯卡拉琪琪 欧美 啄木鸟 在线 日本很黄很慌bb很色的视频 内地伦理片无码视频 日本黄页大全视频 找个小姐日着玩视频 395UaGG 二线女明星不雅视频 欧美磁性链接 2018每日在线女优AV视频 92后女性食奶视频 色色色av激情视频欧美 夜夜插gif wwweee184 超级大奶头黄片 av, com 蝌蚪窝米奇网 色嘛嘛 现在的小视频网址 变态人妖性交视频 京香julia艺术照 潜入女捜査官手机在线 东方成人lu片 wwwe8817comwww7caopcom 1269av在线视频 在线AV色色 男人天堂2018天堂网西瓜影音 伊人成年小说综合网 97夏同学旧网 欧洲色b 无码AV网站大全 日本伦理无码高清mp4迅雷下载 A片毛片基地 一边看书一边自慰番号 鲁鸡把播放器 sebi97 av番号 mp4 58看片 97色色酒吧电影院 亚洲日韩激情在线 龚玥菲新潘金莲斑马影视 色东方 MIDE–500 亚洲欧美av在线观看 av天5 中文字幕人妻出轨av番号 87国产一区 牛bai 聊斋仙桃影视 xo影院在线观看免费观看 丁香花婷婷 色妞abs130 红怡院成人 强奸美女的视频app 宅男视频3p3 天堂岛av大全 吉泽明步986先锋 SSNI-413 ed2k 秋霞170 大奶av音像先锋 大片播放器 X 影片名:网红美女演绎学生看到老师穿着高跟丝袜很性感就尾随跟到家里和老 福利社男人把鸡插入美女尿道视频 疯狂缠绵桃色视频 丁香视频资源站 大波妹 yinyin福利自拍 大香蕉兽交人在线视频 大沢萌夜总会 大奶少妇喂 手淫69视频 情侣偷拍自拍在线 微拍福利99 大胆西西人体44rt ,net 一个护士的性爱录音 国产亚洲精品自拍偷拍野战视频 亚洲图片 自拍 俄罗斯t极度另类 流浪汉强x番号 欧美妻片 漫画区成人福利 超碰在线视频自拍偷拍国产 成人av所有网址 动力电影 西瓜影音 SNIS-300 800AV亚洲 丁香五月在线观看线 日本一本道av京东热高清 欧洲美女视频 足控磁力 mp4 ckplayer菠萝影院 欧美性爱bb视频 a免费高清不卡视频 youbb线视频 ses涩涩影院 XOXO在线 bibibi视频女主播 鸡巴插入欧美美女的b舒服 日本护士x x x o o o 露出视频福利 小黄瓜免费的福利视频 韩国伦理片s歩兵 毛多水多的女优 bt美国福利影吧 52我爱干免费看 四库影视 免费视频 成人影片不需下载gav 破初系列在线观看网站 三级黄线下载 白丝网站你懂的 自拍秦先生 男用j插女b 武汉第七中学门国产自拍在线 蒂亚 中文字幕在线观看 久热 中文字幕母亲和孩子 恋夜女主播福利合集 周末同床 ddy2499 江疏影不良视频magnet 威尼斯人大屁股做爱无码视频 迅雷资源链接 在线看成人小视频 在线看片网站国产 一个女人有男朋友为了生活和老板上床电影 社区性爱视频 最新欧美Av在线 欧美曰本一本道免费视频在线观看 日韩人妻免费高清视频 中国一级A片、/ 欧美无码av先锋影音 美人蒲 电影强奸乱伦 蚂蚁窝一个释放蝌蚪 密桃影 任你操任你曰爽爽 无码在线高速 av俘虏 五月丁678 任你躁视频搬运工 精品 鸭子澳门网址 日本最长69式在线 我妻如妓我如奴四哥 红豆导航精品在线 xx676 日本avv喷潮 九卅影城tv777 日本黄色影院 日本我不卡av 性污秽小视频 新忍尿大会。part2 性自虐视频网站 性感女神级美女主播诱惑要被迷死了 性感学生妹小视频 现在哪里还有小视频网站 泄欲哥网 小苹果性交影院 柳州莫菁 福利 温碧霞b cup 操逼直播视频直播 艹逼色中色 少妇掰B自慰21p ady伦理影音先锋 厕奴调教番号 京香junlia作品动态图 操白嫩女友露脸图 欧美日一本道 - 百度 - 百度 - 百度 - 百度 百度天堂acg silk024在线观西瓜看 美乳少妇动态视频 谷露琪琪五月丁香 freex彩漫 爱爱呻吟视频在线 办公室女神小雅漫画在线阅读 男人吸奶视频在线 91影院色激情 一级厕所偷拍视频 网红美女磁力链接 黑大屁股男女性交视频 日本女优中文字幕 午夜激情XXOO 电影院里摸她屄 美女视频做爱 t亚洲黄片 人人摸人人搞人人操 大机巴搞影院 仔仔网福利视频在线观看 女主丰满 国产 风骚中年妇女 台湾野外伦理 甘榴影院 国产福利直播在线手机视频 人妖做爱a片在线看 天堂鲁丝袜 jav在线播放无需下载播放器 亚洲午马天堂 亚洲新一木道, av网址色福利大全 成人网站来一波 四叶AV影院 男人同性视频在线观看 - 百度 老司机电影天堂看懂的 美殴色图 无码a试看 闪一下成人视频网站 华裔女与黑人群交 国产重口味自拍 波多野结衣3d无码作品番号及封面 国产丝袜高跟恋足调教视频 北京妞性爱视频 播放器大全女女女女女 国产视频母子做爱 被抄的影院 国内自拍在线吧 草逼 不堪凌辱的小林 波多野结衣被内射的 百度云 国产小青蛙搭讪 西瓜。 XXⅩ日本学生观看播放 湿妹伦理片 一本道素人三级 松下沙荣子BF-557 设为首页加入收藏图片区88 看米奇资源站 一本道最新色视频 少妇骚逼天天日 777影院 色老影院 水岛津实 媚药发狂种子 www875bbcon jjkkrrrr 冲田杏梨小时工 东方超碰进入 兔费在线直播欧美性爱电影 国产一本道久在道在线播放 淫妻的变化 色和尚色琪琪在线 1024手机基地看电影旧版恐怖片 人人澡操碰碰中文字 haodiaose37qao视频 协和影院影音先锋 暴风福利在线播放 real睿宝内部V8视频种子 PPPD-642 骑马乳交插乳抽插 JULIA 最后是厉害的 超碰人妻福利视频在线观看 本庄优花磁力 播播撸 t人体写真APP s:∥aui:a26067:C0m 暴风影音在线视频秋霞 sm 亚洲 欧美 少妇 浴室迷情苍井空视频大全 嗯嗯 宝贝在线观看视频 美女裸体无阴毛艺术 日本老黄片电影阿香 爱看福利群 动画无码种子 magnet 鸡吧操屄视频 午夜十二点福利导航 日本老女人丝祙 色色屋影院 动漫操操操人人操hhenhen 日本性交视濒 国产av 丝袜美腿 久久reav在线观看 亚洲阿v天堂2017手机avtt 4438x4最新视频网 四房播人肉 樱花族论坛在线视频 甘婷婷种子磁力搜索 97资源 magnet 日本加嘞比性交视频 3手机激情在线成人影院 类似仙桃福利视频的网站 198黄色网站 女教师监禁奴隶免费 rio在线手机视频免费视频 黄片啊啊嗯激烈 O福利导航 qiuxia手机影院同性恋 午夜欧美成人影dvd片 噜噜吧老司机免费影院 第九影院福利 在线观看 莉哥不雅视频1分钟完整版 奇优影院 WebCache 视频区在线av 欧美爱爱a∨视频 国外在线nobotv 国产熟女超碰在线视频 哺乳期的淫荡少妇 本田莉子视频 国产性虑自拍 国模抠B自拍 藏经阁色 亚洲无码性爱视频在线 让老公插自己小姊妹 japanesevoices 1819 福利gif magnet 2泰國超正女學生自拍影片流出 无毛伦理 欧美萝莉自慰视频 av视频 日本av 福利电影青春草 caoporn 国产自拍 偷拍自拍自拍视频嫖妓 48号缚师绑美女 高跟爆乳在线 天堂网_avmp4 国产自拍 小辣椒 苍井空在线教师2015 免费无码伦理片havtv 小草草大黄瓜在线观看 勇者传说ol加速器 伦伦影院手机在线看 看黄色视频, 谷露邪恶影院体验区 国产自拍HD高清 97影院霞秋在线 我的班主任是个超级大色鬼里番 青娱乐吧· 肏多毛的老女人 一本道av 免费频 内裤飞走了百度云 重生fast girls btav 七色成人在线 成人先锋爱爱视频 97paoa 毛片儿激情淫乱视频 超爽福利的秒拍视频 222kkk 乐色视频国产自拍 久草视频前田由美中出 河台性游戏 丁香色区大香蕉视频 欧美tv快来射电影 CD性爱视频 成人小视频日本 韩国舔屄 5017神级无码番号推荐 大桥未久快播视频在线 俄罗斯波神在线播放 人本黄片在线观看 玫瑰花大眼睛模特裸体视频 美国老太大阴部性成熟 舌恋丝 乱伦视频在线免费看 毛色性一级片 毛片试看3o秒的视频 裸体影城 美女仓井啪啪无翼乌 美女操操 撸撸炮撸一炮在线影院 极品魔鬼身材女神被满身毛 清纯唯美五月香蕉 青青草视频苍井空 精品幼女在线视频 寂寞丝妇 经典伦理片abc 人与畜生配交电影 强奸乱伦图片 激情视频体验版女自慰 朴妮麦福利视频优酷网 会员 28:37 大神夜店搭讪极品外围女带回高级寓所阳台一直干到莎发720P高清无 金瓶梅在线第五季 精品一级国产黄片 伦理片明末 一级韩国毛片性一交图片 真实强奸ed2k 下载 800av播放器 美乳女神思瑞 我要涩涩 李美淑左爱 99精品任你干 日本黄色视频裸照 午夜影院视费x看1000 深喉吞精在线 超碰人人很很操图片 性51免费视频在线观看 被窝网理论在线福利视频 午夜图库三级 Lingleizhuanqv 亚洲欧洲日美有码无码有声小说 宝宝新福利影院 91爱丝小仙女酒店前台 我要打飞 机com2019 waifu哔咔官网 57av08 sigua 888com 欧州av与亚州av 亚洲图日韩专区 好声音 好好的日免费视频 下载 黄色影院福利区免费无需下载在线试看。 黑人来中国跟美妞说鸡巴超级大骚货主动约炮鸡巴大也是很好的一件事啊- 黄色彩大美女的逼逼 户外女主播勾引外卖小哥打野炮 欧美图片综合自拍19p 黄片毛片淫乱篇快速影院 日本高清无码视频自拍 大香蕉在线导航视频 插少妇綾合网 wwwady51 日本井川由依影视 汤姆影院AVt0n 大香蕉狼狼日伊人 伦理电影网无码 a片毛片香港澳门 色妞老秃驴 欧美 日韩 巨乳人妻 精品国产自在线拍fennencaonu 一本道久在线名模 翔田千里 艳情 在线播放 捆绑调教 拳交番号大全 国产啪啪啪视频网友自拍 58影视网在线 神码伦理在线 加勒比系列丝袜中出 男女做爱视频黄片 mp4 单身男福利剧场 夫面前侵犯柚木 seyutv 么什网战看成人免费 99thz桃色论坛 百威快播成人电网 日本AV群交游戏 成年人免费黄色网站 kedouzipaiwang sesihu 大香蕉伊人综合网色屋 先锋影在线看片 caopo国产成人 99thz在线论坛 澳门三级操碰在线视频 id034一之濑桃 猛男操美女视频 91国产乱伦剧情006-“儿子我受不了”母子乱伦之卡拉OK內射骚屄妈妈高清无水印版 欧美幼老色与狗色 五感图片在线 桃隐福利社区官方论坛入口 91福利站在线 3d口球视频 国模蒂蒂在线 韩国美女自慰视频在线看 久久 国产制服丝袜福利视频 韩国午夜爱阴 韩国美女vip视频合集在线观看 国产自拍无极制 国产自拍啪啪啪在线 国产自拍、欧美 男女鸡鸡插阴道里面的视频 男女交配视频真人 爱爱午夜老司机福利影院 日本黄片视频软件 美女,主播真心漂亮椅子上紫薇逼逼无毛 u15 番号 丝袜长腿秘书性爱 国产自拍视频同事 日本爱爱成人视频网站 肥熟大妞 丝袜黑色脚交图片福利 一日本老太视频播放 川相美月 无码 9191手机国产在线播放 qiangjianshaonushipin 五十路无码合集thunder 马凡舒裸体视频迅雷下载下载 Caoii∪1024 porn三级 magnet 国产自拍yi 自拍5xsq 里番douluo123 青草草社区 网友自拍 国庆 厕所视频 黄片A片视频 日本激情性短视频观看 - 百度 做素股 yezubuluo18,com 珑泽萝拉无码av下载 玉色天堂 快手AV网址 加勒比 东京热 Japanese av在线 色中色址导航 aa2424凤凰影院 wwwxx男人天堂 爱paipai 在线成人AV magnet 日韩无码茌线 成人综合娱乐在线视频 mav磁力链接 爱人体福利导航 欧美 色 图 中国人的最爱苍井空520 欧美高中情侣 视频 黄色艳舞视频 成人视频黄色网站 大香蕉高清在线播放 欧美畜生伦理 Japanese av jav hd 国产精品大香蕉在线视频 乡本佳人视频 菅野松雪磁力种子下载 日本女优视频 ed2k 玛格丽特公主的艳照 乱伦骚逼12p 那里能看苍井空的 打屁股2小游戏3kk 拍和日本人做爱的电影 偷挶? 大白逼黄片 吃林心如身体 人体艺体大写真 hd色妹妹影院第三页 人体艺术网站有那几个 色战 黑丝袜电影院 日本美女全祼体图片免费 鬼地狱ⅱ长泽雪野 胖子大阴唇也很胖 操美女妹妹小说 淫妻女友系列小说 黄朝激情 撸撸色色屄 户外掰开美女人体艺术 肛交合集观看 御姐很哀伤无码照片 逍遥仙境论坛最新地址 丝交p 我操邻居小媳妇 明日香捆绑 撸女热 能看的欧美图片 闫凤娇大尺度艺术 波多野结衣14影音先锋 自拍在线网 伊沢淳子先锋影音