Ebay的架构发展.ppt

《Ebay的架构发展.ppt》由会员分享，可在线阅读，更多相关《Ebay的架构发展.ppt（37页珍藏版）》请在三一办公上搜索。

1、,eB,ay,I,nc,.,The eBay Architecture,Striking a balance between site stability,feature velocity,performance,and cost,SD Forum 2006,Presented By:Randy Shoup and Dan Pritchett,Date:November 29,2006,ay,I,eB,nc,.,An SUV is soldsells 5 2,What were up against eBay manages Over 212,000,000 registered users,

2、Over 1 Billion photos eBay users worldwide trade more than$1590worth of goods every second eBay averages over 1 billion page views perday At any given time,there are approximately105 million listings on the site eBay stores over 2 Petabytes of data over200 times the size of the Library of Congress!T

3、he eBay platform handles 3 billion API callsper month In a dynamic environment,A sporting good everyeveryminutessecondsOver Million pounds ofKimchi are sold every year!,300+features per quarter We roll 100,000+lines of code every two weeks In 33 countries,in seven languages,24x726 Billion SQL execut

4、ions/day!2 2006 eBay Inc.,ay,I,nc,.,eB,eBays Exponential Growth105MillionListings212MillionUsersQ1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3,1998,1999,2000,2001,2002,2003,2004,2005,2006,4 2006 eBay Inc.,nc,.,eB,ay,I,Velocity of eBay-Software Development Process,Feature,212M

5、 Users,300+FeaturesPer Quarter,FeatureFeature,Train100K LOC/Wk,6M LOC,99.94%,Our site is our product.We change it incrementally through implementing new features.Very predictable development process trains leave on-time at regular intervals(weekly).Parallel development process with significant outpu

6、t-100,000 LOC per release.Always on over 99.94%available.All while supporting a 24x7 environment5 2006 eBay Inc.,eB,ay,I,nc,.,Systemic RequirementsAvailability,ReliabilityMassive ScalabilitySecurityMaintainabilityFaster ProductDeliveryArchitect for thefuture,Enable seamless growthDeliver quality fun

7、ctionality ataccelerating ratesEnable rapid business innovation,10X Growth6 2006 eBay Inc.,eB,ay,I,nc,.,Architectural Lessons,Scale Out,Not Up,Horizontal scaling at every tier.Functional decomposition.,Prefer Asynchronous Integration,Minimize availability coupling.Improve scaling options.,Virtualize

8、 Components,Reduce physical dependencies.Improve deployment flexibility.,Design for Failure,Automated failure detection and notification.“Limp mode”operation of business features.,7,2006 eBay Inc.,ay,I,nc,.,eB,Ongoing Platform Evolution,Registered Users,212M,1998,1999,2000,2001,2002,2003,2004,2005,Q

9、3,V1,2006,V2.0,V2.3,V2.4,V2.5,V3,V4,eBay architecture versions8 2006 eBay Inc.,eB,ay,I,nc,.,V1.0 1995-September 1997Built over a weekend in Pierre Omidyars living room in 1995System hardware was made up of parts that could be bought at FrysEvery item was a separate file,generated by a Perl scriptNo

10、search functionality,only category browsingThis system maxed out at 50,000 active items,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,9 2006 eBay Inc.,eB,ay,I,nc,.,V2.0 September 1997-February 1999 3-tiered conceptual architecture(separation of bus/pres and db access tiers)2-tiered physical

11、 implementation(no application server)C+Library(eBayISAPI.dll)running on IIS on Windows Microsoft index server used for search Items migrated from GDBM to an Oracle database on Solaris,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,10 2006 eBay Inc.,eB,ay,I,nc,.,V2.1 February 1999-November 1

12、999 Servers grouped into pools(small soldiers)Resonate used for front end load balancing and failover Search functionality moved to the Thunderstone indexing system Back-end Oracle database server scaled vertically to a larger machine(Sun E10000),1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,200

13、5,11 2006 eBay Inc.,eB,ay,I,nc,.,V2.3 June 1999-November 1999,Second Database added for failover,CGI pools,Listings,Pages,and Search continued to scale horizontally,However,By November 1999,the database servers approached their limits of physical growth.,12,2006 eBay Inc.,eB,ay,I,nc,.,V2.4 November

14、1999-April 2001,Database split technology.,Logically partition database into separate instances.Horizontal scalability through 2000,but not beyond.,13,2006 eBay Inc.,nc,.,I,ay,eB,December,2002,V2.5 April 2001 December 2002 Horizontal scalability through database splits Items split by category SPOF e

15、limination,User Write,User Read,ACCOUNTS,FEEDBACK,ARCHIVE,BATCH JOBS,BATCH JOBS,Bull,Bear,CATY 1CATY 5CATY 9,CATY 2CATY 6CATY 10,CATY 3CATY 7CATY 11,CATY 4CATY 8CATY 12,TranScratch,November,1999,14,SUNA3500,2006 eBay Inc.,eB,ay,I,nc,.,Now that we have the Database taken care of.,Application Server,M

16、onolithic 2-tier Architecture,3.3 Million Line C+ISAPI DLL(150MB binary),Hundreds of developers,all working on the same codeHitting compiler limits on number of methods per class(!),15,2006 eBay Inc.,eB,ay,I,nc,.,V3 Replace C+/ISAPI with Java 2002-present,Re-wrote the entire application in J2EE appl

17、ication server framework,Gave us a chance to architect the code for reuse and separation of duties,Leveraged the MSXML framework for the presentation layer,Minimizing the development cost for migration,Implemented a development kernel as a foundation for programmers,Allowed for rapid training and de

18、ployment of new engineers,16,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier,eB,ay,I,nc,.,Scaling the Data Tier:Overview,Spread the Load,Segmentation by function.,Horizontal splits within functions.,Minimize the Work,Limit in database work,The Tricks to Scaling,How to survive without transactions.

19、Creating alternate database structures.,18,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier:Functional Segmentation Segment databases into functional areas,User hostsItem hosts,Account hostsFeedback hostsTransaction hostsAnd about 70 more functional categories Rationale Partitions data by different

20、 scaling/usage characteristics Supports functional decoupling and isolation19 2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier:Horizontal Split,Split databases horizontally by primary access path.Different patterns for different use cases,Write Master/Read Slaves,Segmentation by data;Two approaches

21、,Modulo on a key,typically the primary key.,Simple data location if you know the keyNot so simple if you dont.,Map to data location,Supports multiple keys.,Doubles reads required to locate data.,SPOF elimination on map structure is complex.,Rationale,Horizontal scaling of transactional load.,Segment

22、 business impact on database outage.,20,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier:Logical Database Hosts Separate Application notion of a database from physical implementation Databases may be combined and separated with no code changes Reduce cost of creating multiple environments(Dev,QA,)A

23、pplication Servers,Attributes Catalogs,Rules,CATY1.N,User,Account,Feedback,Misc,API,SCRATCH,DB1,DB2,DB3,21 2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier:Minimize DB Resources,No business logic in database,No stored procedures,Only very simple triggers(default value population),Move CPU-intensive

24、 work to applications,Referential Integrity Joins,Sorting,Extensive use of prepared statements and bind variables,22,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Data Tier:Minimize DB Transactions,Auto-commit for vast majority of DB writes Absolutely no client side transactions,Single database transactio

25、ns managed through anonymous PL/SQL blocks.No distributed transactions.,How do we pull it off?,Careful ordering of DB operations Recovery through,Asynchronous recovery events Reconciliation batch Failover to async flow,Rationale,Avoid deadlocks,Avoid coupling availabilityUpdate concurrency,Seamless

26、handling of splits,23,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Application Tier,eB,ay,I,nc,.,Scaling the Application Tier Overview,Spread the Load,Segmentation by function.,Horizontal load-balancing within functions.,Minimize dependencies,Between applications,Between functional areas,From application

27、s to data tier resources,Virtualize data access,25,2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Application Tier Massively Scaling J2EE,Step 1-Throw out most of J2EE,eBay scales on servlets and a rewritten connection pool.,Step 2 Keep Application Tier Completely Stateless,No session state in application

28、tier,Transient state maintained in cookie or scratch database,Step 3 Cache Where Possible,Cache common metadata across requests,with sophisticated cache refresh,procedures,Cache reload from local storage Cache request data in ThreadLocal,26,2006 eBay Inc.,I,nc,eB,ay,.,Scaling the Application Tier Ti

29、ered Application Model Strictly partition application into tiersPresentationBusinessIntegration,Presentation TierBusiness TierIntegration Tier,XSLCommand(View)AO/AOF(View)BO/BOFDO/DAO,XML ModelBuilding LogicBusiness LogicData Access Layer(DAL),27 2006 eBay Inc.,eB,ay,I,nc,.,Scaling the Application T

30、ier Data Access Layer(DAL),What is the DAL?,eBays internally-developed pure Java OR mapping solution.All CRUD(Create Read Update Delete)operations are performed,through DALs abstraction of the data.,Enables horizontal scaling of the Data tier without application code,changes,Dynamic Data Routing abs

31、tracts application developers from,Database splits,Logical/Physical Hosts Markdown,Graceful degradation,Extensive JDBC Prepared Statements cached by DataSources,28,2006 eBay Inc.,I,nc,eB,ay,.,29,Scaling the Application Tier Vertical Code Partitioning Partition code into functional areas Application

32、is specific to a single area(Selling,Buying,etc.)Domain contains common business logic across Applications Restrict inter-dependencies Applications depend on Domains,not on other Applications No dependencies among shared Domains,UserApplication SellingApplication BuyingApplication BillingApplication

33、 SearchApplication,Applications,UserDomain,SellingDomain,BuyingDomain,BillingDomain,SearchDomain,PersonalizationDomainSharedBuyingDomain,UserValidationDomainmyEbayDomain,SharedBillingDomainSharedSearchDomain,SharedDomains,Core-Domain,LookupDomain,API Domain,2006 eBay Inc.,nc,.,I,ay,eB,Scaling the Ap

34、plication Tier Functional Segmentation Segment functions into separate application pools Minimizes/isolates DB dependencies Allows for parallel development,deployment,and monitoring,ViewItem Poolhttp:/,SYI Poolhttp:/,Load Balancing,CGI0,CGI5,IIS WebServers,Load Balancing,IIS,IIS,IIS,IIS,AppServers,A

35、S,AS,AS,AS,AS,AS,Load Balancing,30,User,Acct,Caty1,Caty20+,2006 eBay Inc.,eB,ay,EDA,EDA,SOA,I,nc,.,Scaling the Application Tier Platform Decoupling Domain Partitioning for Deployment Decouple non-transactional domains from transactional flows Search and billing domains are not required in transactio

36、n processing.Fraud domain is required but easier to manage as separate deployment.Integrate with a combination of asynchronous EDA and synchronous SOApatterns.TransactionPlatform,Billing,Search,Fraud,31 2006 eBay Inc.,eB,ay,I,nc,.,Scaling Search,eB,ay,I,nc,.,Scaling Search Overview,In 2002,eBay sear

37、ch had reached its limits,Cost of scaling third-party search engine had become prohibitive 9 hours to update the index,Running on largest systems vendor sold and still not keeping up,eBay has unique search requirements,Real-time updates,Update item on any change(list,bid,sale,etc.)Users expect chang

38、es to be visible immediately,Exhaustive recall,Sellers notice if search results miss any item,Search results require data(“histograms”)from every matching item,Flexible data storage,Keywords,Structured categories and attributes,No off-the-shelf product met these needs,33,2006 eBay Inc.,eB,ay,I,nc,.,

39、Scaling Search Voyager,Real-time feeder infrastructure,Reliable multicast from primary database to search nodes,Real-time indexing,Search nodes update index in real time from messages,In-memory search index Horizontal segmentation,Search index divided into N slices(“columns”)Each slice is replicated

40、 to M instances(“rows”),Aggregator parallelizes query over all N slices,load-balances over M,instances,Caching,Cache results for highly expensive and frequently used queries,34,2006 eBay Inc.,eB,ay,I,nc,.,Scaling Operations,eB,ay,I,nc,.,Scaling Operations Code Deployment,Demanding Requirements,Entir

41、e site rolled every 2 weeks,All deployments require staged rollout with immediate rollback if necessary.More than 100 WAR configurations.,Dependencies exist between pools during some deployment operations.More than 15,000 instances across eight physical data centers.,Rollout Plan,Custom application

42、that works from dependencies provided by projects.Creates transitive closure of dependencies.Generates rollout plan for Turbo Roller.,Automated Rollout Tool(“Turbo Roller”),Manages full deployment cycle onto all application servers.Executes rollout plan.,Built in checkpoints during rollout,including

43、 approvals.,Optimized rollback,including full rollback of dependent pools.,37,2006 eBay Inc.,eB,ay,I,nc,.,Scaling Operations Monitoring,Centralized Activity Logging(CAL),Transaction oriented logging per application server,Transaction boundary starts at request.Nested transactions supported.,Detailed

44、 logging of all application activity,especially database and other external,resources.,Application generated information and exceptions can be reported.,Logging streams gathered and broadcast on a message bus.,Subscriber to log to files(1.5TB/day),Subscriber to capture exceptions and generate operat

45、ional alerts.Subscriber for real time application state monitoring.,Extensive Reporting,Reports on transactions(page and database)per pool.Relationships between URLs and external resources.Inverted relationships between databases and pools/URLs.,Data cube reporting on several key metrics available i

46、n near real time.,38,2006 eBay Inc.,eB,ay,I,nc,.,Recap,AvailabilityReliabilityMassive Scalability,Enabling seamless growth Massive Database and Code Scalability,SecurityDelivering quality functionality at,MaintainabilityFaster ProductDeliveryArchitecting forthe future,accelerating rates Further streamline and optimize the eBaydevelopment modelEnabling rapid business innovation,10X Growth39 2006 eBay Inc.,