ag旗舰厅官网_ag旗舰厅下载客户端

热门关键词: ag旗舰厅官网,ag旗舰厅下载客户端

ag旗舰厅

当前位置:ag旗舰厅官网 > ag旗舰厅 > twisted异步搜聚程序代码

twisted异步搜聚程序代码

来源:http://www.pedaLyourcycLe.com 作者:ag旗舰厅官网 时间:2019-10-04 22:15

对于大气的多少收罗除了八线程,就唯有异步来贯彻了。本文是透过twisted框架来落到实处异步搜罗,

Twisted 简介,twisted简介

Twisted是用Python完结的基于事件驱动的网络引擎框架,Twisted帮忙广大常见的传导及应用层合同,富含TCP、UDP、SSL/TLS、HTTP、IMAP、SSH、IRC以及FTP。仿佛Python同样,Twisted也是有所“内置电瓶”(batteries-included)的特色。Twisted对于其协理的装有公约都蕴含客商端和服务器完结,同一时候附带有基于命令行的工具,使得配置和布置产等第的Twisted应用变得特别有助于。

Transports
Transports代表网络中多少个通讯结点之间的连接。Transports担任描述连接的细节,比方总是是面向流式的仍然面向数据报的,流控以及可信赖性。TCP、UDP和Unix套接字可作为transports的事例。它们被规划为“知足最小效用单元,同期持有最大程度的可复用性”,况且从左券落到实处中分离出来,这让不菲公约得以选拔一样档期的顺序的传导。Transports达成了ITransports接口,它包罗如下的章程:

write 以非阻塞的形式按顺序依次将数据写到物理连接上
writeSequence 将八个字符串列表写到物理连接上
loseConnection 将具备挂起的数据写入,然后停业连接
getPeer 获得一连中对端的地址信息
getHost 获得延续中本端的地方新闻
将transports从事商业业事务中分离出来也使得对那多个等级次序的测验变得非常简便易行。能够因而简单地写入三个字符串来效仿传输,用这种措施来检查。

Protocols
Protocols描述了何等以异步的法子管理互连网中的事件。HTTP、DNS以及IMAP是应用层合同中的例子。Protocols实现了IProtocol接口,它包涵如下的点子:

makeConnection 在transport对象和服务器之间创设一条连接
connectionMade 连接创设起来后调用
dataReceived 接收数据时调用
connectionLost 关闭连接时调用

Reactor模式
Twisted达成了设计形式中的反应堆(reactor)方式,这种情势在单线程境况中调节五个事件源发生的平地风波到它们分其余事件管理例程中去。

Twisted的主干正是reactor事件循环。Reactor能够感知网络、文件系统以及电火花计时器事件。它等待接下来管理那一个事件,从一定于阳台的一举一动中架空出来,并提供联合的接口,使得在互联网公约栈的其余地点对事件做出响应都变得轻巧。

Deferreds
Deferred对象以抽象化的点子表明了一种思量,即结果还尚一纸空文。它同样能够补助管理产生那一个结果所须要的回调链。当从函数中回到时,Deferred对象承诺在某些时刻函数将发生三个结果。再次来到的Deferred对象中带有全体注册到事件上的回调援引,由此在函数间只供给传递那三个目的就能够,追踪这么些指标比单独保管全体的回调要简单的多。

Deferred对象富含一对回调链,一个是指向操作成功的回调,一个是针对操作失利的回调。开首状态下Deferred对象的两条链都为空。在事件管理的进度中,每一种阶段都为其拉长管理成功的回调养管理失利的回调。当三个异步结果到来时,Deferred对象就被“激活”,那么管理成功的回调弄整理处理退步的回调就足以以合适的诀要遵照它们拉长进去的逐条依次获得调用。

异步版U中华VL获取器采取Deferred对象后的代码如下:

from twisted.internet import reactor
import getPage

def processPage(page):
print page

def logError(error):
print error

def finishProcessing(value):
print "Shutting down..."
reactor.stop()

url = ""
deferred = getPage(url) # getPage returns a Deferred
deferred.addCallbacks(success, failure)
deferred.addBoth(stop)

reactor.run()
在这些本子中调用的事件管理函数与事先同一,但它们都登记到了贰个独立的Deferred对象上,并非散落在代码到处再以参数格局传递给getPage。

事例1:(轻松的tcp服务器实现)
server.py
#-*-coding:utf-8-*-

from twisted.internet import protocol,reactor
#from twisted.protocols.basic import LineReceiver
from time import ctime
import os,sys
from twisted.python import log #导入log

PORT = 23233

class TSServerProtocol(protocol.Protocol): #从Protocol派生出的TSSserverProtocol类
def connectionMade(self): #因为接二连三了Protocol就能够应用和自定义父类的主意了
clnt = self.clnt = self.transport.getPeer().host
log.msg('Client connected from %s' % clnt) #log用法
#self.sendData()
def dataReceived(self, data): #收受多少的议程
addr = self.transport.getHost().host
if data:
log.msg('Cmd received from %s : %s' % (self.clnt,data)) #data接收到的新闻
self.transport.write('那是源于 %s ,的答复:%s' %(addr,ctime())) #write发送
'''for i in os.listdir(os.getcwd()):
self.transport.write('[%s]%s' %(i,data))'''
def connectionLost(self,reason): #reason参数必要求写不然报错,当连接断开时的调用
log.msg('连接断开. ')
#log.msg('连接断开. %s' % reason) # 加上reason会把断开的错误音信答应出来

factory =protocol.Factory() #因为大家地方只导入了protocol,所以那要把Factory函数导进来,每三个protocol的实例都有一个工厂的援用,使用self.factory可以访谈所在的厂子实例
factory.protocol = TSServerProtocol #当有央求步入时调用TSServerProtocol类
print 'waiting for connection......'

log.startLogging(sys.stdout) #以日记方式打字与印刷到窗口
reactor.listenTCP(PORT,factory) #监听诉求
reactor.run() #启航循环

client.py
#-*-coding:utf-8

from twisted.internet import protocol,reactor

HOST = 'localhost' #要一而再的IP地址
PORT = 23233 #端口
class TSClntProtocol(protocol.Protocol): #同服务端同样
def sendData(self): #概念贰个发送数据的不二等秘书诀
data = raw_input('>')
if data:
print '....sending %s.......' %data
self.transport.write(data) #发送数据
else:
self.transport.loseConnection() #当未有输入时断开连接
def connectionMade(self):
self.sendData() #调用方法
def dataReceived(self, data):
print data
self.sendData()

class TSClntFactory(protocol.ClientFactory):#自个儿的领会是客商端写protocol.ClientFactory,服务端写protocol.Factory
protocol = TSClntProtocol #同服务端同样
clientConnectionLost = clientConnectionFailed = lambda self,connector,reason:reactor.stop()

reactor.connectTCP(HOST,PORT,TSClntFactory()) #跟服务端同样
reactor.run()

简单介绍,twisted简要介绍Twisted是用Python达成的依照事件驱动的网络引擎框架,Twisted援助广大广泛的传导及应用层合同,包罗TCP、UDP、SSL/TL...

Async Batching with Twisted: A Walkthrough

Example 1: Just a DefferedList

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
def listCallback(results):
  print results
def finish(ign):
  reactor.stop()
def test():
  d1 = getPage('')
  d2 = getPage('')
  dl = DeferredList([d1, d2])
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

This is one of the simplest examples you’ll ever see for a deferred list in action. Get two deferreds (the getPage function returns a deferred) and use them to created a deferred list. Add callbacks to the list, garnish with a lemon.

Example 2: Simple Result Manipulation

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
def listCallback(results):
  for isSuccess, content in results:
    print "Successful? %s" % isSuccess
    print "Content Length: %s" % len(content)
def finish(ign):
  reactor.stop()
def test():
  d1 = getPage('')
  d2 = getPage('')
  dl = DeferredList([d1, d2])
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

We make things a little more interesting in this example by doing some processing on the results. For this to make sense, just remember that a callback gets passed the result when the deferred action completes. If we look up the API documentation for DeferredList, we see that it returns a list of (success, result) tuples, where success is a Boolean and result is the result of a deferred that was put in the list (remember, we’ve got two layers of deferreds here!).

Example 3: Page Callbacks Too

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
def pageCallback(result):
  return len(result)
def listCallback(result):
  print result
def finish(ign):
  reactor.stop()
def test():
  d1 = getPage('')
  d1.addCallback(pageCallback)
  d2 = getPage('')
  d2.addCallback(pageCallback)
  dl = DeferredList([d1, d2])
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

Here, we mix things up a little bit. Instead of doing processing on all the results at once (in the deferred list callback), we’re processing them when the page callbacks fire. Our processing here is just a simple example of getting the length of the getPage deferred result: the HTML content of the page at the given URL.

Example 4: Results with More Structure

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
def pageCallback(result):
  data = {
    'length': len(result),
    'content': result[:10],
    }
  return data
def listCallback(result):
  for isSuccess, data in result:
    if isSuccess:
      print "Call to server succeeded with data %s" % str(data)
def finish(ign):
  reactor.stop()
def test():
  d1 = getPage('')
  d1.addCallback(pageCallback)
  d2 = getPage('')
  d2.addCallback(pageCallback)
  dl = DeferredList([d1, d2])
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

A follow-up to the last example, here we put the data in which we are interested into a dictionary. We don’t end up pulling any of the data out of the dictionary; we just stringify it and print it to stdout.

Example 5: Passing Values to Callbacks

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
def pageCallback(result, url):
  data = {
    'length': len(result),
    'content': result[:10],
    'url': url,
    }
  return data
def getPageData(url):
  d = getPage(url)
  d.addCallback(pageCallback, url)
  return d
def listCallback(result):
  for isSuccess, data in result:
    if isSuccess:
      print "Call to %s succeeded with data %s" % (data['url'], str(data))
def finish(ign):
  reactor.stop()
def test():
  d1 = getPageData('')
  d2 = getPageData('')
  dl = DeferredList([d1, d2])
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

After all this playing, we start asking ourselves more serious questions, like: “I want to decide which values show up in my callbacks” or “Some information that is available here, isn’t available there. How do I get it there?” This is how :-) Just pass the parameters you want to your callback. They’ll be tacked on after the result (as you can see from the function signatures).

In this example, we needed to create our own deferred-returning function, one that wraps the getPage function so that we can also pass the URL on to the callback.

Example 6: Adding Some Error Checking

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet.defer import DeferredList
urls = [
  '',
  '',
  '',
  '',
  ]
def pageCallback(result, url):
  data = {
    'length': len(result),
    'content': result[:10],
    'url': url,
    }
  return data
def pageErrback(error, url):
  return {
    'msg': error.getErrorMessage(),
    'err': error,
    'url': url,
    }
def getPageData(url):
  d = getPage(url, timeout=5)
  d.addCallback(pageCallback, url)
  d.addErrback(pageErrback, url)
  return d
def listCallback(result):
  for ignore, data in result:
    if data.has_key('err'):
      print "Call to %s failed with data %s" % (data['url'], str(data))
    else:
      print "Call to %s succeeded with data %s" % (data['url'], str(data))
def finish(ign):
  reactor.stop()
def test():
  deferreds = []
  for url in urls:
    d = getPageData(url)
    deferreds.append(d)
  dl = DeferredList(deferreds, consumeErrors=1)
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

As we get closer to building real applications, we start getting concerned about things like catching/anticipating errors. We haven’t added any errbacks to the deferred list, but we have added one to our page callback. We’ve added more URLs and put them in a list to ease the pains of duplicate code. As you can see, two of the URLs should return errors: one a 404, and the other should be a domain not resolving (we’ll see this as a timeout).

Example 7: Batching with DeferredSemaphore

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet import defer
maxRun = 1
urls = [
  '',
  '',
  '',
  '',
  ]
def listCallback(results):
  for isSuccess, result in results:
    print len(result)
def finish(ign):
  reactor.stop()
def test():
  deferreds = []
  sem = defer.DeferredSemaphore(maxRun)
  for url in urls:
    d = sem.run(getPage, url)
    deferreds.append(d)
  dl = defer.DeferredList(deferreds)
  dl.addCallback(listCallback)
  dl.addCallback(finish)
test()
reactor.run()

These last two examples are for more advanced use cases. As soon as the reactor starts, deferreds that are ready, start “firing” — their “jobs” start running. What if we’ve got 500 deferreds in a list? Well, they all start processing. As you can imagine, this is an easy way to run an accidental DoS against a friendly service. Not cool.

For situations like this, what we want is a way to run only so many deferreds at a time. This is a great use for the deferred semaphore. When I repeated runs of the example above, the content lengths of the four pages returned after about 2.5 seconds. With the example rewritten to use just the deferred list (no deferred semaphore), the content lengths were returned after about 1.2 seconds. The extra time is due to the fact that I (for the sake of the example) forced only one deferred to run at a time, obviously not what you’re going to want to do for a highly concurrent task ;-)

Note that without changing the code and only setting maxRun to 4, the timings for getting the the content lengths is about the same, averaging for me 1.3 seconds (there’s a little more overhead involved when using the deferred semaphore).

One last subtle note (in anticipation of the next example): the for loop creates all the deferreds at once; the deferred semaphore simply limits how many get run at a time.

Example 8: Throttling with Cooperator

 代码如下

from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet import defer, task
maxRun = 2
urls = [
  '',
  '',
  '',
  '',
  ]
def pageCallback(result):
  print len(result)
  return result
def doWork():
  for url in urls:
    d = getPage(url)
    d.addCallback(pageCallback)
    yield d
def finish(ign):
  reactor.stop()
def test():
  deferreds = []
  coop = task.Cooperator()
  work = doWork()
  for i in xrange(maxRun):
    d = coop.coiterate(work)
    deferreds.append(d)
  dl = defer.DeferredList(deferreds)
  dl.addCallback(finish)
test()
reactor.run()

就算前段时间还没到研究twisted框架的水准,不过这里先记下下,以备未来用时再体会。

本文由ag旗舰厅官网发布于ag旗舰厅,转载请注明出处:twisted异步搜聚程序代码

关键词:

上一篇:没有了

下一篇:没有了