Asynchronous I/O Subsystem Synchronous I/O occurs while you wait. Applications processing cannot continue until the I/O operation is complete. In contrast, asynchronous I/O operations run in the background and do not block user applications. This improves performance, because I/O operations and applications processing can run simultaneously. Using asynchronous I/O will usually improve your I/O throughput, especially when you are storing data in raw logical volumes (as opposed to Journaled file systems). The actual performance, however, depends on how many server processes are running that will handle the I/O requests. Many applications, such as databases and file servers, take advantage of the ability to overlap processing and I/O. These asynchronous I/O operations use various kinds of devices and files. Additionally, multiple asynchronous I/O operations can run at the same time on one or more devices or files. Each asynchronous I/O request has a corresponding control block in the application's address space. When an asynchronous I/O request is made, a handle is established in the control block. This handle is used to retrieve the status and the return values of the request. Applications use the aio_read and aio_write subroutines to perform the I/O. Control returns to the application from the subroutine, as soon as the request has been queued. The application can then continue processing while the disk operation is being performed. A kernel process (kproc), called a server, is in charge of each request from the time it is taken off the queue until it completes. The number of servers limits the number of disk I/O operations that can be in progress in the system simultaneously. The default values are minservers=1 and maxservers=10. In systems that seldom run applications that use asynchronous I/O, this is usually adequate. For environments with many disk drives and key applications that use asynchronous I/O, the default is far too low. The result of a deficiency of servers is that disk I/O seems much slower than it should be. Not only do requests spend inordinate lengths of time in the queue, but the low ratio of servers to disk drives means that the seek-optimization algorithms have too few requests to work with for each drive. Note: Asynchronous I/O will not work if the control block or buffer is created using mmap (mapping segments). In AIX 5.2 there are two Asynchronous I/O Subsystems. The original AIX AIO, now called LEGACY AIO, has the same function names as the posix compliant POSIX AIO. The major differences between the two involve different parameter passing. Both subsytems are defined in the /usr/include/sys/aio.h file. The _AIO_AIX_SOURCE macro is used to distinguish between the two versions. Note: The _AIO_AIX_SOURCE macro used in the /usr/include/sys/aio.h file must be defined when using this file to compile an aio application with the LEGACY AIO function definitions. The default compile using the aio.h file is for an application with the new POSIX AIO definitions. To use the LEGACY AIO function defintions do the following in the source file: #define _AIO_AIX_SOURCE #include <sys/aio.h> or when compiling on the command line, type the following: xlc ... -D_AIO_AIX_SOURCE ... classic_aio_program.c For each aio function there is a legacy and a posix definition. LEGACY AIO has an additional aio_nwait function, which although not a part of posix definitions has been included in POSIX AIO to help those who want to port from LEGACY to POSIX definitions. POSIX AIO has an additional aio_fsync function, which is not included in LEGACY AIO. For a list of these functions, see Asynchronous I/O Subroutines. How Do I Know if I Need to Use AIO? Using the vmstat command with an interval and count value, you can determine if the CPU is idle waiting for disk I/O. The wa column details the percentage of time the CPU was idle with pending local disk I/O. If there is at least one outstanding I/O to a local disk when the wait process is running, the time is classified as waiting for I/O. Unless asynchronous I/O is being used by the process, an I/O request to disk causes the calling process to block (or sleep) until the request has been completed. Once a process's I/O request completes, it is placed on the run queue. A wa value consistently over 25 percent may indicate that the disk subsystem is not balanced properly, or it may be the result of a disk-intensive workload. Note: AIO will not relieve an overly busy disk drive. Using the iostat command with an interval and count value, you can determine if any disks are overly busy. Monitor the %tm_act column for each disk drive on the system. On some systems, a %tm_act of 35.0 or higher for one disk can cause noticeably slower performance. The relief for this case could be to move data from more busy to less busy disks, but simply having AIO will not relieve an overly busy disk problem. SMP Systems For SMP systems, the us, sy, id and wa columns are only averages over all processors. But keep in mind that the I/O wait statistic per processor is not really a processor-specific statistic; it is a global statistic. An I/O wait is distinguished from idle time only by the state of a pending I/O. If there is any pending disk I/O, and the processor is not busy, then it is an I/O wait time. Disk I/O is not tracked by processors, so when there is any I/O wait, all processors get charged (assuming they are all equally idle). How Many AIO Servers Am I Currently Using? To determine you how many Posix AIO Servers (aios) are currently running, type the following on the command line: pstat -a | grep posix_aioserver | wc -l Note: You must run this command as the root user. To determine you how many Legacy AIO Servers (aios) are currently running, type the following on the command line: pstat -a | egrep ' aioserver' | wc -l   Note: You must run this command as the root user. If the disk drives that are being accessed asynchronously are using either the Journaled File System (JFS) or the Enhanced Journaled File System (JFS2), all I/O will be routed through the aios kprocs. If the disk drives that are being accessed asynchronously are using a form of raw logical volume management, then the disk I/O is not routed through the aios kprocs. In that case the number of servers running is not relevant. However, if you want to confirm that an application that uses raw logic volumes is taking advantage of AIO, you can disable the fast path option via SMIT. When this option is disabled, even raw I/O will be forced through the aios kprocs. At that point, the pstat command listed in preceding discussion will work. You would not want to run the system with this option disabled for any length of time. This is simply a suggestion to confirm that the application is working with AIO and raw logical volumes. At releases earlier than AIX 4.3, the fast path is enabled by default and cannot be disabled. How Many AIO Servers Do I Need? Here are some suggested rules of thumb for determining what value to set maximum number of servers to: 1.        The first rule of thumb suggests that you limit the maximum number of servers to a number equal to ten times the number of disks that are to be used concurrently, but not more than 80. The minimum number of servers should be set to half of this maximum number. 2.        Another rule of thumb is to set the maximum number of servers to 80 and leave the minimum number of servers set to the default of 1 and reboot. Monitor the number of additional servers started throughout the course of normal workload. After a 24-hour period of normal activity, set the maximum number of servers to the number of currently running aios + 10, and set the minimum number of servers to the number of currently running aios - 10. In some environments you may see more than 80 aios KPROCs running. If so, consider the third rule of thumb. 3.        A third suggestion is to take statistics using vmstat -s before any high I/O activity begins, and again at the end. Check the field iodone. From this you can determine how many physical I/Os are being handled in a given wall clock period. Then increase the maximum number of servers and see if you can get more iodones in the same time period. Prerequisites To make use of asynchronous I/O the following fileset must be installed: bos.rte.aio To determine if this fileset is installed, use: lslpp -l bos.rte.aio You must also make the aio0 or posix_aio0 device available using SMIT. smit chgaio smit chgposixaio STATE to be configured at system restart available or smit aio smit posixaio Configure aio now Functions of Asynchronous I/O Functions provided by the asynchronous I/O facilities are: •        Large File-Enabled Asynchronous I/O •        Nonblocking I/O •        Notification of I/O completion •        Cancellation of I/O requests Large File-Enabled Asynchronous I/O The fundamental data structure associated with all asynchronous I/O operations is struct aiocb. Within this structure is the aio_offset field which is used to specify the offset for an I/O operation. Due to the signed 32-bit definition of aio_offset, the default asynchronous I/O interfaces are limited to an offset of 2G minus 1. To overcome this limitation, a new aio control block with a signed 64-bit offset field and a new set of asynchronous I/O interfaces has been defined. These 64–bit definitions end with "64". The large offset-enabled asynchronous I/O interfaces are available under the _LARGE_FILES compilation environment and under the _LARGE_FILE_API programming environment. For further information, see Writing Programs That Access Large Files in AIX 5L Version 5.3 General Programming Concepts: Writing and Debugging Programs. Under the _LARGE_FILES compilation environment, asynchronous I/O applications written to the default interfaces see the following redefinitions: Item        Redefined To Be        Header File struct aiocb        struct aiocb64        sys/aio.h aio_read()        aio_read64()        sys/aio.h aio_write()        aio_write64()        sys/aio.h aio_cancel()        aio_cancel64()        sys/aio.h aio_suspend()        aio_suspend64()        sys/aio.h aio_listio()        aio_listio64()        sys/aio.h aio_return()        aio_return64()        sys/aio.h aio_error()        aio_error64()        sys/aio.h For information on using the _LARGE_FILES environment, see Porting Applications to the Large File Environment in AIX 5L Version 5.3 General Programming Concepts: Writing and Debugging Programs In the _LARGE_FILE_API environment, the 64-bit API interfaces are visible. This environment requires recoding of applications to the new 64-bit API name. For further information on using the _LARGE_FILE_API environment, see Using the 64-Bit File System Subroutines in AIX 5L Version 5.3 General Programming Concepts: Writing and Debugging Programs Nonblocking I/O After issuing an I/O request, the user application can proceed without being blocked while the I/O operation is in progress. The I/O operation occurs while the application is running. Specifically, when the application issues an I/O request, the request is queued. The application can then resume running before the I/O operation is initiated. To manage asynchronous I/O, each asynchronous I/O request has a corresponding control block in the application's address space. This control block contains the control and status information for the request. It can be used again when the I/O operation is completed. Notification of I/O Completion After issuing an asynchronous I/O request, the user application can determine when and how the I/O operation is completed. This information is provided in three ways: •        The application can poll the status of the I/O operation. •        The system can asynchronously notify the application when the I/O operation is done. •        The application can block until the I/O operation is complete. Polling the Status of the I/O Operation The application can periodically poll the status of the I/O operation. The status of each I/O operation is provided in the application's address space in the control block associated with each request. Portable applications can retrieve the status by using the aio_error subroutine.The aio_suspend subroutine suspends the calling process until one or more asynchronous I/O requests are completed. Asynchronously Notifying the Application When the I/O Operation Completes Asynchronously notifying the I/O completion is done by signals. Specifically, an application may request that a SIGIO signal be delivered when the I/O operation is complete. To do this, the application sets a flag in the control block at the time it issues the I/O request. If several requests have been issued, the application can poll the status of the requests to determine which have actually completed. Blocking the Application until the I/O Operation Is Complete The third way to determine whether an I/O operation is complete is to let the calling process become blocked and wait until at least one of the I/O requests it is waiting for is complete. This is similar to synchronous style I/O. It is useful for applications that, after performing some processing, need to wait for I/O completion before proceeding. Cancellation of I/O Requests I/O requests can be canceled if they are cancelable. Cancellation is not guaranteed and may succeed or not depending upon the state of the individual request. If a request is in the queue and the I/O operations have not yet started, the request is cancellable. Typically, a request is no longer cancelable when the actual I/O operation has begun. Asynchronous I/O Subroutines Note:The 64-bit APIs are as follows: The following subroutines are provided for performing asynchronous I/O: Subroutine        Purpose aio_cancel or aio_cancel64 Cancels one or more outstanding asynchronous I/O requests. aio_error or aio_error64 Retrieves the error status of an asynchronous I/O request. aio_fsync Synchronizes asynchronous files. lio_listio or lio_listio64 Initiates a list of asynchronous I/O requests with a single call. aio_nwait Suspends the calling process until n asynchronous I/O requests are completed. aio_read or aio_read64 Reads asynchronously from a file. aio_return or aio_return64 Retrieves the return status of an asynchronous I/O request. aio_suspend or aio_suspend64 Suspends the calling process until one or more asynchronous I/O requests is completed. aio_write or aio_write64 Writes asynchronously to a file. Order and Priority of Asynchronous I/O Calls An application may issue several asynchronous I/O requests on the same file or device. However, because the I/O operations are performed asynchronously, the order in which they are handled may not be the order in which the I/O calls were made. The application must enforce ordering of its own I/O requests if ordering is required. Priority among the I/O requests is not currently implemented. The aio_reqprio field in the control block is currently ignored. For files that support seek operations, seeking is allowed as part of the asynchronous read or write operations. The whence and offset fields are provided in the control block of the request to set the seek parameters. The seek pointer is updated when the asynchronous read or write call returns. Subroutines Affected by Asynchronous I/O The following existing subroutines are affected by asynchronous I/O: •        The close subroutine •        The exit subroutine •        The exec subroutine •        The fork subroutine If the application closes a file, or calls the _exit or exec subroutines while it has some outstanding I/O requests, the requests are canceled. If they cannot be canceled, the application is blocked until the requests have completed. When a process calls the fork subroutine, its asynchronous I/O is not inherited by the child process. One fundamental limitation in asynchronous I/O is page hiding. When an unbuffered (raw) asynchronous I/O is issued, the page that contains the user buffer is hidden during the actual I/O operation. This ensures cache consistency. However, the application may access the memory locations that fall within the same page as the user buffer. This may cause the application to block as a result of a page fault. To alleviate this, allocate page aligned buffers and do not touch the buffers until the I/O request using it has completed. Changing Attributes for Asynchronous I/O You can change attributes relating to asynchronous I/O using the chdev command or SMIT. Likewise, you can use SMIT to configure and remove (unconfigure) asynchronous I/O. (Alternatively, you can use the mkdev and rmdev commands to configure and remove asynchronous I/O). To start SMIT at the main menu for asynchronous I/O, enter smit aio or smit posixaio. MINIMUM number of servers Indicates the minimum number of kernel processes dedicated to asynchronous I/O processing. Because each kernel process uses memory, this number should not be large when the amount of asynchronous I/O expected is small. MAXIMUM number of servers per cpu Indicates the maximum number of kernel processes per cpu that are dedicated to asynchronous I/O processing. This number when multiplied by the number of cpus indicates the limit on I/O requests in progress at one time, and represents the limit for possible I/O concurrency. Maximum number of REQUESTS Indicates the maximum number of asynchronous I/O requests that can be outstanding at one time. This includes requests that are in progress as well as those that are waiting to be started. The maximum number of asynchronous I/O requests cannot be less than the value of AIO_MAX, as defined in the /usr/include/sys/limits.h file, but it can be greater. It would be appropriate for a system with a high volume of asynchronous I/O to have a maximum number of asynchronous I/O requests larger than AIO_MAX. Server PRIORITY Indicates the priority level of kernel processes dedicated to asynchronous I/O. The lower the priority number is, the more favored the process is in scheduling. Concurrency is enhanced by making this number slightly less than the value of PUSER, the priority of a normal user process. It cannot be made lower than the values of PRI_SCHED. Because the default priority is (40+nice), these daemons will be slightly favored with this value of (39+nice). If you want to favor them more, make changes slowly. A very low priority can interfere with the system process that require low priority. Attention: Raising the server PRIORITY (decreasing this numeric value) is not recommended because system hangs or crashes could occur if the priority of the AIO servers is favored too much. There is little to be gained by making big priority changes. PUSER and PRI_SCHED are defined in the /usr/include/sys/pri.h file. STATE to be configured at system restart Indicates the state to which asynchronous I/O is to be configured during system initialization. The possible values are: •        defined, which indicates that the asynchronous I/O will be left in the defined state and not available for use •        available, which indicates that asynchronous I/O will be configured and available for use STATE of FastPath The AIO Fastpath is used only on character devices (raw logical volumes) and sends I/O requests directly to the underlying device. The file system path used on block devices uses the aio kprocs to send requests through file system routines provided to kernel extensions. Disabling this option forces all I/O activity through the aios kprocs, including I/O activity that involves raw logical volumes. In AIX 4.3 and earlier, the fast path is enabled by default and cannot be disabled. 64-bit Enhancements Asynchronous I/O (AIO) has been enhanced to support 64-bit enabled applications. On 64-bit platforms, both 32-bit and 64-bit AIO can occur simultaneously. The struct aiocb, the fundamental data structure associated with all asynchronous I/O operation, has changed. The element of this struct, aio_return, is now defined as ssize_t. Previously, it was defined as an int. AIO supports large files by default. An application compiled in 64-bit mode can do AIO to a large file without any additional #define or special opening of those files.

看来很久之前的以讹传讹是在是影响太厉害了。
1、aio server不是进程，是kernel threading，所以效率才高，要不不如用用户态的thread或者processor实现AIO算了。
2  and 3 显然find使用的syscall是read而不是aio_read，所以只能用阻塞IO，就算你配置了AIO ENABLE也不可能出现你说的3的情况。
4、最大的以讹传讹，AIO可以用于RAW DEVICE，而且效果更好。因为大多数的UNIX的AIO实现是这样的，如果对RAW DEVICE用aio_read aio_write是，是用kernel threading，效率最高；但是对于FS，只能用lwp 或者thread模拟，效率低很多。所以在使用INFORMIX DB2等等DBMS的时候，要充分发挥aio的效果，应该用raw device做数据容器。
5、再次强调，aio server是kernel threading，和用户自己创建的thread或者process都不同。当然AIO SERVER没有必要配置的太多，要根据 应用和使用的磁盘控制器数量等决定。

设计 mmo 服务器，我听过许多老生常谈，说起处理大量连接时， select 是多么低效。我们应该换用 iocp (windows), kqueue(freebsd), 或是 epoll(linux) 。的确，处理大量的连接的读写，select 是够低效的。因为 kernel 每次都要对 select 传入的一组 socket 号做轮询，那次在上海，以陈榕的说法讲，这叫鬼子进村策略。一遍遍的询问“鬼子进村了吗？”，“鬼子进村了吗？”... 大量的 cpu 时间都耗了进去。（更过分的是在 windows 上，还有个万恶的 64 限制。）

使用 kqueue 这些，变成了派一些个人去站岗，鬼子来了就可以拿到通知，效率自然高了许多。不过最近我在反思，真的需要以这些为基础搭建服务器吗？

1、WIN32下面用proactor可以达到几乎RAW　IOCP的效率，由于封装关系，应该是差那么一点。

客户端处理类的常规写法：
//处理客户端连接消息
class ClientHandler : public ACE_Service_Handler
{
public:
 /**构造函数
  *
  *
  */
 ClientHandler(unsigned int client_recv_buf_size=SERVER_CLIENT_RECEIVE_BUF_SIZE)
  :_read_msg_block(client_recv_buf_size),_io_count(0)
 {
 }

 ~ClientHandler(){}

 /**
  *初始化，因为可能要用到ClientHandler内存池，而这个池又不一定要用NEW
  */
 void init();

 /**清理函数，因为可能要用到内存池
  *
  */
 void fini();

//检查是否超时的函数

 void check_time_out(time_t cur_time);

public:

 /**客户端连接服务器成功后调用
  *
  * \param handle 套接字句柄
  * \param &message_block 第一次读到的数据（未用）
  */

//由Acceptor来调用！！！
 virtual void open (ACE_HANDLE handle,ACE_Message_Block &message_block);

 /**处理网络读操作结束消息
  *
  * \param &result 读操作结果
  */
 virtual void handle_read_stream (const ACE_Asynch_Read_Stream::Result &result);

 /**处理网络写操作结束消息
  *
  * \param &result 写操作结果
  */
 virtual void handle_write_stream (const ACE_Asynch_Write_Stream::Result &result);

private:

//**生成一个网络读请求
  *
  * \param void
  * \return 0-成功，-1失败
  */
 int  initiate_read_stream  (void);

 /**生成一个写请求
  *
  * \param mb 待发送的数据
  * \param nBytes 待发送数据大小
  * \return 0－成功，－1失败
  */
 int  initiate_write_stream (ACE_Message_Block & mb, size_t nBytes );
 
 /**
  *
  * \return 检查是否可以删除，用的是一个引用计数。每一个外出IO的时候＋1，每一个IO成功后－1
  */
 int check_destroy();
 
 //异步读
 ACE_Asynch_Read_Stream _rs;

 //异步写
 ACE_Asynch_Write_Stream _ws;

 //接收缓冲区只要一个就够了，因为压根就没想过要多读，我直到现在也不是很清楚为什么要多读，多读的话要考虑很多问题
 ACE_Message_Block _read_msg_block;

 //套接字句柄,这个可以不要了，因为基类就有个HANDLER在里面的。
 //ACE_HANDLE _handle;

 //一个锁，客户端反正有东东要锁的，注意，要用ACE_Recursive_Thread_Mutex而不是用ACE_Thread_Mutex，这里面是可以重入的，而且在WIN32下是直接的EnterCriticalSection，可以达到很高的效率
 ACE_Recursive_Thread_Mutex _lock;
 
 //在外IO数量,其实就是引用计数啦，没啥的。为0的时候就把这个东东关掉啦。
 long _io_count;

//检查超时用的，过段时间没东东就CLOSE他了。

 time_t _last_net_io;

private:

//本来想用另外一种模型的，只用1个或者2个外出读，后来想想，反正一般内存都是足够的，就不管了。

 //ACE_Message_Block _send_msg_blocks[2];

 //ACE_Message_Block &_sending_msg_block;

 //ACE_Message_Block &_idle_msg_block;

private:
 
public:
//TODO:move to prriva and use friend class!!!

//只是为了效率更高，不用STL的LIST是因为到现在我没有可用的Node_Allocator，所以效率上会有问题。
 ClientHandler *_next;

 ClientHandler *next(){return _next;}

 void next(ClientHandler *obj){_next=obj;}

};

//这是具体实现，有些地方比较乱，懒得管了，锁的有些地方不对。懒得改了，反正在出错或者有瓶颈的时候再做也不迟。

void ClientHandler::handle_read_stream (const ACE_Asynch_Read_Stream::Result &result)
{
 _last_net_io=ACE_OS::time(NULL);
 int byterecved=result.bytes_transferred ();
 if ( (result.success ()) && (byterecved != 0))
 {
  //ACE_DEBUG ((LM_DEBUG,  "Receiver completed:%d\n",byterecved));

//处理完数据
  if(handle_received_data()==true)
  {
   //ACE_DEBUG ((LM_DEBUG,  "go on reading...\n"));

//把东东推到头部，处理粘包
   _read_msg_block.crunch();
   initiate_read_stream();
  }
 }

//这个地方不想用ACE_Atom_op，因为反正要有一个锁，而且一般都会用锁，不管了。假如不在意的话，应该直接用ACE_Atom_Op以达到最好的效率

 {
  ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);
  _io_count--;
 }
 check_destroy ();
}

void ClientHandler::init()
{

//初始化数据，并不在构造函数里做。
 _last_net_io=ACE_OS::time(NULL);
 _read_msg_block.rd_ptr(_read_msg_block.base());
 _read_msg_block.wr_ptr(_read_msg_block.base());
 this->handle(ACE_INVALID_HANDLE);
}

bool ClientHandler::handle_received_data()
{

...........自己处理
 return true;
}

//==================================================================
void ClientHandler::handle_write_stream (const ACE_Asynch_Write_Stream::Result &result)
{
 //发送成功，RELEASE掉
 //这个不可能有多个RELEASE，直接XX掉
 //result.message_block ().release ();
 MsgBlockManager::get_instance().release_msg_block(&result.message_block());

 {
  ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);
  _io_count--;
 }
 check_destroy ();
}

//bool ClientHandler::destroy ()
//{
// FUNC_ENTER;
// ClientManager::get_instance().release_client_handle(this);
// FUNC_LEAVE;
// return false ;
//}

int  ClientHandler::initiate_read_stream  (void)
{
 ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);

//考虑到粘包的呀
 if (_rs.read (_read_msg_block, _read_msg_block.space()) == -1)
 {
  ACE_ERROR_RETURN ((LM_ERROR,"%p\n","ACE_Asynch_Read_Stream::read"),-1);
 }
 _io_count++;
 return 0;
}

/**生成一个写请求
*
* \param mb 待发送的数据
* \param nBytes 待发送数据大小
* \return 0－成功，－1失败
*/
int  ClientHandler::initiate_write_stream (ACE_Message_Block & mb, size_t nBytes )
{
 ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);
 if (_ws.write (mb , nBytes ) == -1)
 {
  mb.release ();
  ACE_ERROR_RETURN((LM_ERROR,"%p\n","ACE_Asynch_Write_File::write"),-1);
 }
 _io_count++;
 return 0;
}

void ClientHandler::open (ACE_HANDLE handle,ACE_Message_Block &message_block)
{
 //FUNC_ENTER;
 _last_net_io=ACE_OS::time(NULL);
 _io_count=0;
 if(_ws.open(*this,this->handle())==-1)
 {
  ACE_ERROR ((LM_ERROR,"%p\n","ACE_Asynch_Write_Stream::open"));
 }
 else if (_rs.open (*this, this->handle()) == -1)
 {
  ACE_ERROR ((LM_ERROR,"%p\n","ACE_Asynch_Read_Stream::open"));
 }
 else
 {
  initiate_read_stream ();
 }

 check_destroy();
 //FUNC_LEAVE;
}

void ClientHandler::fini()
{
}

void ClientHandler::check_time_out(time_t cur_time)
{
 //ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);
 //ACE_DEBUG((LM_DEBUG,"cur_time is %u,last io is %u\n",cur_time,_last_net_io));

 //检测是否已经为0了
 if(this->handle()==ACE_INVALID_HANDLE)
  return;
 if(cur_time-_last_net_io>CLIENT_TIME_OUT_SECONDS)
 {
  ACE_OS::shutdown(this->handle(),SD_BOTH);
  ACE_OS::closesocket(this->handle());
  this->handle(ACE_INVALID_HANDLE);
 }
}

int ClientHandler::check_destroy()
{
 {
  ACE_Guard<ACE_Recursive_Thread_Mutex> locker (_lock);
  if (_io_count> 0)
   return 1;
 }
 ACE_OS::shutdown(this->handle(),SD_BOTH);
 ACE_OS::closesocket(this->handle());
 this->handle(ACE_INVALID_HANDLE);

//这个地方给内存池吧。
 ClientManager::get_instance().release_client_handle(this);
 //delete this;
 return 0;
}

一般是用来判断对方（设备，进程或其它网元）是否正常动行，一般采用定时发送简单的通讯包，如果在指定时间段内未收到对方响应，则判断对方已经当掉。用于检测TCP的异常断开。

基本原因是服务器端不能有效的判断客户端是否在线也就是说，服务器无法区分客户端是长时间在空闲，还是已经掉线的情况.所谓的心跳包就是客户端定时发送简单的信息给服务器端告诉它我还在而已。

代码就是每隔几分钟发送一个固定信息给服务端，服务端收到后回复一个固定信息
如果服务端几分钟内没有收到客户端信息则视客户端断开。比如有些通信软件长时间不使用，要想知道它的状态是在线还是离线就需要心跳包，定时发包收包。

发包方：可以是客户也可以是服务端，看哪边实现方便合理。一般是客户端。服务器也可以定时轮询发心跳下去。

一般来说，出于效率的考虑，是由客户端主动向服务器端发包，而不是相反。

epoll Scalability Web Page

• Introduction
• Interface Description
• Man Pages
• Testing
• dphttpd
• dphttpd SMP results
• dphttpd UP results
• pipetest
• pipetest results
• Recent comparison results
• Analysis and Conclusions
• Acknowledgements

Davide Libenzi wrote an event poll implementation and described it at the /dev/epoll home page here. His performance testing led to the conclusion that epoll scaled linearly regardless of the load as defined by the number of dead connections. However, the main hindrance to having epoll accepted into the mainline Linux kernel by Linus Torvalds was the interface to epoll being in /dev. Therefore, a new interface to epoll was added via three new system calls. That interface will hereafter be referred to as sys_epoll. Download sys_epoll here.

• int epoll_create(int maxfds);
  
  

  The system call epoll_create() creates a sys_epoll "object" by allocating space for "maxfds" descriptors to be polled. The sys_epoll "object" is referenced by a file descriptor, and this enables the new interface to :

  • Maintain compatibility with the existing interface
  • Avoid the creation of a epoll_close() syscall
  • Reuse 95% of the existing code
  • Inherit the file* automatic clean up code

• int epoll_ctl(int epfd, int op, int fd, unsigned int events);
  
  

  The system call epoll_ctl() is the controller interface. The "op" parameter is either EP_CTL_ADD, EP_CTL_DEL or EP_CTL_MOD. The parameter "fd" is the target of the operation. The last parameter, "events", is used in both EP_CTL_ADD and EP_CTL_MOD and represents the event interest mask.

• int epoll_wait(int epfd, struct pollfd * events, int maxevents, int timeout);
  
  

  The system call epoll_wait() waits for events by allowing a maximum timeout, "timeout", in milliseconds and returns the number of events ( struct pollfd ) that the caller will find available at "*events".

• Postscript :
  • epoll.ps
    
  • epoll_create.ps
    
  • epoll_ctl.ps
    
  • epoll_wait.ps
    
• ASCI Text :
  • epoll.txt
    
  • epoll_create.txt
    
  • epoll_ctl.txt
    
  • epoll_wait.txt
    

We tested using two applications:

• dphttpd
• pipetest

dphttpd

Software

The http client is httperf from David Mosberger. Download httperf here. The http server is dphttpd from Davide Libenzi. Download dphttpd here. The deadconn client is also provided by Davide Libenzi. Download deadconn here.

Two client programs (deadconn_last and httperf) run on the client machine and establish connections to the HTTP server (dphttpd) running on the server machine. Connections established by deadconn_last are "dead". These send a single HTTP get request at connection setup and remain idle for the remainder of the test. Connections established by httperf are "active". These continuously send HTTP requests to the server at a fixed rate. httperf reports the rate at which the HTTP server replies to its requests. This reply rate is the metric reported on the Y-axis of the graphs below.

For the tests, the number of active connections is kept constant and the number of dead connections increased from 0 to 60000 (X-axis of graphs below). Consequently, dphttpd spends a fixed amount of time responding to requests and a variable amount of time looking for requests to service. The mechanism used to look for active connections amongst all open connections is one of standard poll(), /dev/epoll or sys_epoll. As the number of dead connections is increased, the scalability of these mechanisms starts impacting dphttpd's reply rate, measured by httperf.

dphttpd SMP

Server

• Hardware: 8-way PIII Xeon 700MHz, 2.5 GB RAM, 2048 KB L2 cache
• OS : RedHat 7.3 with 2.5.44 kernel, patched with ONE of:
  • sys_epoll-2.5.44-0.9.diff To reproduce download here.
    To run the latest sys_epoll patch download here.
  • ep_patch-2.5.44-0.32.diff Download /dev/epoll
  • unpatched kernel Download 2.5.44.
• /proc/sys/fs/file-max = 131072
• /proc/sys/net/ipv4/tcp_fin_timeout = 15
• /proc/sys/net/ipv4/tcp_max_syn_backlog = 16384
• /proc/sys/net/ipv4/tcp_tw_reuse = 1
• /proc/sys/net/ipv4/tcp_tw_recycle = 1
• /proc/sys/net/ipv4/ip_local_port_range = 1024 65535
• # ulimit -n 131072
• # dphttpd --maxfds 20000 --stksize 4096
• default size of reply = 128 bytes

Client

• Hardware: 4-way PIII Xeon 500MHz, 3 GB RAM, 512 KB L2 cache
• OS : RedHat 7.3 with 2.4.18-3smp kernel
• /proc/sys/fs/file-max = 131072
• /proc/sys/net/ipv4/tcp_fin_timeout = 15
• /proc/sys/net/ipv4.tcp_tw_recycle = 1
• /proc/sys/net/ipv4.tcp_max_syn_backlog = 16384
• /proc/sys/net/ipv4.ip_local_port_range = 1024 65535
• # ulimit -n 131072
• # deadconn_last $SERVER $SVRPORT num_connections
  where num_connections is one of 0, 50, 100, 200, 400, 800, 1000, 5000, 10000, 15000, 20000, 25000, 30000, 35000, 40000, 45000, 50000, 55000, 60000.
• After deadconn_last reports num_connections established
  # httperf --server=$SERVER --port=$SVRPORT --think-timeout 5 --timeout 5 --num-calls 20000 --num-conns 100 --hog --rate 100

Results for dphttpd SMP

dphttpd UP

Server

• 1-way PIII, 866MHz, 256 MB RAM
• OS: 2.5.44 gcc 2.96 (RedHat 7.3)
• /proc/sys/fs/file-max = 65536
• /proc/sys/net/ipv4/tcp_fin_timeout = 15
• /proc/sys/net/ipv4/tcp_tw_recycle = 1
• # ulimit -n 65536
• # dphttpd --maxfds 20000 --stksize 4096
• default size of reply = 128 bytes

Client

• 1-way PIII, 866MHz, 256 MB RAM
• OS: 2.4.18, gcc 2.96 (RedHat 7.3)
• /proc/sys/fs/file-max = 65536
• /proc/sys/net/ipv4/tcp_fin_timeout = 15
• /proc/sys/net/ipv4.tcp_tw_recycle = 1
• # ulimit -n 65536
• # deadconn_last $SERVER $SVRPORT num_connections
  where num_connections is one of 0, 50, 100, 200, 400, 800, 1000, 2000, 4000, 6000, 8000, 10000, 12000, 14000, 16000.
• After deadconn_last reports num_connections established
  # httperf --server=$SERVER --port=$SVRPORT --think-timeout 5 --timeout 5 --num-calls 20000 --num-conns 100 --hog --rate 100

Results for dphttpd UP

Pipetest

David Stevens added support for sys_epoll to Ben LaHaise's original pipetest.c application. Download Ben LaHaise's Ottawa Linux Symposium 2002 paper including pipetest.c here.Download David Steven's patch to add sys_epoll to pipetest.c here.

Pipetest SMP

System and Configuration

• 8-way PIII Xeon 900MHz, 4 GB RAM, 2048 KB L2 cache
• OS: RedHat 7.2 with 2.5.44 kernel, patched with one of:
  • sys_epoll-2.5.44-0.9.diff To reproduce download here.
    To run the latest sys_epoll patch download here.
  • async poll ported to 2.5.44 download here.
• /proc/sys/fs/file-max = 65536
• # ulimit -n 65536
• # pipetest [poll|aio-poll|sys-epoll] num_pipes message_threads max_generation
  • where num_pipes is one of 10, 100, 500, or 1000-16000 in increments of 1000
  • message_threads is 1
  • max_generantion is 300

Results for pipetest on an SMP

Results for Pipetest on a UP

Same Hardware and Configuration as with the SMP pipetest above
with CONFIG_SMP = n being the only change.

sys_epoll stability comparisons Oct 30, 2002

Following are performance results comparing version 0.14 of the (Download v0.14 here) to the version, v0.9, originally used for the performance testing outlined above. (Download v0.9 here) Testing was done using two measures: pipetest details here. and dphttpd details here..

Analysis and Conclusion

The system call interface to epoll performs as well as the /dev interface to epoll. In addition sys_epoll scales better than poll and AIO poll for large numbers of connections. Other points in favour of sys_epoll are:

• sys_epoll is compatible with synchronous read() and write() and thus makes it usable with existing libraries that have not migrated to AIO.
• Applications using poll() can be easily migrated to sys_epoll while continuing to use the existing I/O infrastructure.
• sys_epoll will be invisible to people who don't want to use it.
• sys_epoll has a low impact on the existing source code.

  
  arch/i386/kernel/entry.S  |    4
  fs/file_table.c           |    4
  fs/pipe.c                 |   36 +
  include/asm-i386/poll.h   |    1
  include/asm-i386/unistd.h |    3
  include/linux/fs.h        |    4
  include/linux/list.h      |    5
  include/linux/pipe_fs_i.h |    4
  include/linux/sys.h       |    2
  include/net/sock.h        |   12
  net/ipv4/tcp.c            |    4
  

Due to these factors sys_epoll should be seriously considered for inclusion in the mainline Linux kernel.

Acknowledgments

Thank You to: Davide Libenzi Who wrote /dev/epoll, sys_epoll and dphttpd.
He was an all around great guy to work with.
Also Thanks to the following people who helped with testing and this web site:
Shailabh Nagar, Paul Larson , Hanna Linder, and David Stevens.