存储厂家EMC云计算存储解决方案 AtmosTechnicalOverview.ppt

EMC Atmos:Technical Overview,EMC,Agenda,Atmos OverviewAtmos OfferingsAtmos hardwareArchitectural Overview of AtmosOperational EfficiencyAccess MethodsRole of policiesAtmos Online,Where does Cloud fit?,Market leading storage solutions for all information and applications,Atmos Solution Offerings,Bundled hardware and software solutionLarger footprints(up to 720 TB in one frame)Ease of configurationSoftware-only solutionSmall footprintsVMWare installSupports existing hardwareDeployment terminologyAtmos cloud:Comprised of one to many locationsEach location is a“Resource Management Group”(RMG)Each RMG is comprised of one or more“Installation Segments”(IS)An IS equates to the private network connecting several Atmos“nodes”Rack Two or more Atmos VMs on the same private subnet,Atmos Next Generation Hardware,Dense Compute1:15 Server-to-drive ratioNow up to 480 TB total capacityLower Power/BTU at 2x Capacity,Small/Medium Scale Compute1:15 servers-to-disks60 TB Cloud Entry PointNow up to 240 TB total capacityLower Power/BTU at 2x Capacity,Capacity1:60 Server-to-drive ratioNow up to 720 TB total capacityLower Power/BTU at 2x Capacity,New,Intel Xeon 5500“Nehalem”Microarchitecture,2 TB Low Power SATA Drives,New,Hardware Design Philosophy,Industry standard componentsStandard data center environmentsFactory configured,racked and cabledFlexible,cost effective service modelOptional HA and redundancyUser serviceable30 drives per 3U drawer Up to 720 TB per rackWS2-120,WS2-240 or WS2-360 configurations,WS2-360:Dense Scale Capacity,Up to 720TB total capacity1 or 2 TB Drives44 U“Hulk”cabinet1:60 server to drive ratio6 servers/24 bullet DAEsPartial Rack will be supported Q24-6 Nodes,2 Node increment1TB Drive:240,360 TB2TB Drive:480,720 TBGbe connectivity1 Allied Telesis Gbe switchRedundant power and coolingEnvironmentals(1 TB)82”height 24”width43”depth2548 lbs weight 10,290 W power35,120 BTU/hr,WS2-240:Dense Scale Compute,Up to 480TB total capacity1 or 2 TB Drives44 U“Hulk”cabinet1:15 server to drive ratio16 servers/16 bullet DAEsPartial Rack will be supported Q24 16 Nodes,2 Node increment1TB Drive:60,90,120,180,210,240 TB2TB Drive:120,180,240,300,360,420,480 TBGbe connectivity2 Allied Telesis Gbe switchesRedundant power and cooling Environmentals(1 TB)82”height24”width43”depth2282 lbs weight10,020 W power34,198 BTU/hr,WS2-360/WS2-240 Serviceability and Accessibility,Patented expandable drawer provides full access to embedded components,Full rear access,No special cooling requirements,Clean,rear-only cabling and automated cable management systems,WS2-120:Small Scale Compute,Up to 240 TB total capacity1 or 2 TB drives40 U“Titan”cabinet1:15 server to drive ratioPartial Rack in Q1:4,6,or 8 Nodes1TB:60TB-4 Node,90TB-6 Node,120TB-8 Node2TB:120TB 4 Node,180 TB 6 Node,240 TB 8 NodeGbE connectivity1 Allied Telesis IP x9924 switch2 PDUsEnvironmentals(1 TB)70”height 24”width40”depth1,356 lbs weight4,880 W power16,655 BTU/hr,Additional Cabinet Power Information,SOAP/REST,DNS Load Balancer,Reference Architecture,Access Points,CIFS/NFS/IFS,Location 3“Tokyo”,Location 2“London”,Location 1“Boston”,Access Points,TCP/IP,TCP/IP,PolicyManagement,JobService,StorageService,Atmos High-Level Architecture,Atmos Native Client Access API,MetadataService,Applications,REST,SOAP,Apache HTTPD,Installable File System(IFS),SystemManagement,MetadataLocationService,ResourceManagementService,Administration,1-Client access,2-Data,5-Management,4-Discovery,3-Metadata,Atmos SoftwareCollection of distributed servicesClient access(1)Data(2)Metadata(3)Discovery(4)Management(5)RPC-XML like Protocol,x86 servers,Dense storage,VMware ESX,Atmos Virtual Edition/Atmos Hardware,Clients/Management,NFS,CIFS,Reference Architecture Creating Objects,“London”,Application Server Farm,DNS Load Balancer,Application makes API call to create object,Load balances to“access node”,Meta data written“locally”.,and synchronized,Policy manager makes 2 synchronous replicas,“Boston”,Object identifier returned to application,Reference Architecture Reading Objects,“London”,Application Server Farm,DNS Load Balancer,Application makes read object call,Load balances to“access node”,Meta data server queried,Optimal object found,Data read from copy,“Boston”,Object data and meta data returned to application,Atmos Tenancy Model,Foundation for cloud storage architecture Conceptual grouping of resources Hierarchical model enables flexible application integrationPlatform to support millions of users Provides robust reporting and resource managementImproves operational efficiency Manage access and resources per tenant/subtenantAllows optimal policy definition and management Helps manage node services,Secure Multi-Tenancy,Access control,auditing,separationMultiple“customers”can share the same cloudIsolation between sets of customers(at the Tenant and Sub-tenant level)Maui tenant capabilitiesMultiple Tenants,each with a unique Tenant IDUnique policies per TenantLogical separation of all resources availableNote:servers are dedicated to one TenantExamples of multiple Tenants sharing a single Atmos cloudCustomers(clients of an organization)Business units(within the same organization)Applications(CIFS backups vs.REST access)Geographic regions(e.g.Australia,Germany,United States)Tenants can be divided into Sub-TenantsBy default,each Tenant has a Sub-Tenant with the same Tenant ID,Atmos Tenancy Model,Conceptual grouping of resources Tenants,Sub-tenants and Users Physical and network separation of Tenants,logical separation of Sub-tenants(each have a unique system-wide ID)Unique policies per TenantSeparate admin management,user management and Tenant-defined policies per Sub-TenantSupports thousands of customers,departments,applications or users in one cloud,Supports internal/external service providersSupports tenancy across many locations while providing a unified name space within the TenantSupports optimal policy definition for different groups or applications,providing operational efficiencyAllows granular management of access and resources per Tenant,Subtenant and UserProvides robust reporting,Boston RMG,NODE 1,NODE 2,NODE 3,NODE 4,NODE 5,NODE 6,NODE 7,NODE 8,London RMG,NODE 1,NODE 2,NODE 3,NODE 4,NODE 5,NODE 6,NODE 7,NODE 8,Tenant A,Tenant B,CIFS Sub-tenant,User n,User 1,CIFS Users,NFS Users,NFS Sub-tenant,User n,User 1,Sub-tenant2,User n,User 1,WS Users,WS Users,Sub-tenant1,User n,User 1,Atmos Policy,Key feature of Atmos Cloud storageMaps business requirements Defined and managed per TenantSub-tenants inherit policies from TenantsOne default policy per tenantOptimal and efficient storage managementMaps business objectives to replication and location requirementsIncrease operational efficiencySet it and forgetValue added services are defined per policyData-path serviceData at rest serviceLocation attributesSS attributesExpiration and retentionInitial Design and Implementation of policies is vital,Policy-based Information ManagementIncreasing Operational Efficiency,Audio Transcripts,Digital Images,Object metadata matches policies Policies map to business objectives,Policies drive actions Where to store information What content services to apply The number of copies,Gold:2 synchronous copies Boston1 asynchronous copy Seattle1 asynchronous copy MiamiBronze:1 synchronous copy MiamiDe-dupe,compression1 asynchronous copy Seattle,Availability=Gold,Availability=Bronze,“.mov”,“image”,etc.,“.mp3”,“audio”,etc.,Custom Metadata,Replicated by policyacross your Enterprise,Retained and archived to the cloud,Policy-Driven Management,Additional Data Services,CompressionReduction of physical space required for object storageDe-duplicationReduction of replicated identical objectsSpin-down“Green”,power efficient method for long term storageFAST/StripingData striping within nodes or across nodes for higher throughputAuto-configuration and auto-healingInstall once just add capacity,Node-levelData Services,Storage Server Attributes,Storage Server Attributes are set at install timeDefined per nodeThey cannot be changed on the flyAttributes must be defined on two levelsData Placement Action Type(Transformations)Pick one of each for node profile,Data Placement,Action Type,Optimal,Green,Fast,Balanced,None,Compression,Deduplication,Checksum,Compress+Dedupe,Fine tune your environment,GeoProtect Dial the right performance and protection levels to your contentTwo implementation methods to choose from,Implement both to maximize content distribution efficiency and cost,Managing storage at Global Scale A New Approach,1 Copy,2 Copies,N Copies,N,By Maintaining:,Susceptible to failures of the system,site,and network connectivityMay not meet required performance because limits of device and or location of access,Improves resilience to failures of the system,site,and network connectivity Potentially improves performance of remote access,Greatly improves resilience to failures of the system,site,and network connectivity Greatly improves performance of remote access,Disperse Copies Address Geographic Performance and Reliability,Managing storage at Global Scale A New Approach,Erasure Codes:Location specific content hardening Increase content durability with less overheadHigher Fault Tolerance(tolerates k faults)with less storage cost(n/m)Availability measures if data stored on a system can be returned promptly(i.e.no special action required to generate the data)Durability measures if data stored in a system is not lost due to failures(i.e.data is available but special action may be needed),Erasure Codes:At a Glance,Object written into m data segmentsand k coded segmentsDetermine where to store segmentsObject can still be retrieved when K components are unavailable or failCannot recreate the object without all required segments,Atmos Erasure Code Implementation,Objects are divided into m data fragments and k coding fragmentsSupports two configurations9/12 configuration=m=9,k=3(Default)Tolerates up-to three drive failures33%storage overheadPerformance overhead is under review10/16 configuration High fault tolerance(up-to 6 drive failure)60%storage overheadPerformance overhead is under reviewFragments are stored in independent fault domains1.3 release has the ability to choose fault domainsBased on replication location configuration Fragments can be distributed across RMG Fragments can be distributed across the System,Erasure Code Encode(9/12 configuration),Encode,Write,9 fragments,Encoded fragments,M=9,K=3,Objects are divided into m=9 data fragments and k=3 coding fragmentsK=3 coding fragments are computed using Cauchy Reed-Solomon codeFragments are stored in independent fault domainsEach fragment is stored on a different diskRMG or system level fault domain optionsOverhead=k/(m)Ex:=3/(9)=33%Encode operations has overhead on CPU Fragment level consistency and integrity checks,Configuring Erasure Codes,Click“Erasure Code”Two additional fields appearAlgorithm:CRS(Cauchy Reed-Solomon)Fragmentation:9/12 or 10/16,Policy Layout Replica Options,Standard ReplicasSingle instance of the data perSingle drive Erasure Coded ReplicasPolicy based striped ReplicasSpecify a number of Storage Servers to stripe acrossSpecify a striped depthFederated ReplicasWith an Atmos Online account configured a“federated”location is allowed,Object Recovery:Drive Failure,BenefitsProvides proactive object recovery in the event of a disk failureSupports multiple drive failuresDoes not require drive replacementRecovery distributes objects within the same RMGAlerts and GUI provide information Recovery start timesObjects impactedCompletion status,Disk Failure Alert,Recovery Process has Begun,Alert for Impacted Objects,Recovery Progress,Flexible Access Methods,Web services accessSOAP:Simple Object Access ProtocolREST:REpresentational State TransferFile services accessNFS:Network File SystemCIFS:Common Internet File SystemIFS:Installable File System,CIFSNFS,HTTP,Cross Platform(via SOAP/REST),Linux/Unix(via NFS),Windows(via CIFS),Linux w/Fuse RPM(via IFS),IFS,WSSOAPREST,AtmosNativeClientAPI,Overview,Installable(Atmos)File SystemNative Atmos file systemSame global file system code and library as on Atmos nodesAllows an external client act as the Atmos Client Services that runs on an Atmos node.IP-based communicationRequires FUSE and RHEL5,AtmosFS RPM,AtmosFS,mauifs,IFSRHEL5 support only32 or 64 bitRequires fuse 2.7.4Speaks native AtmosAtmosFS servers are part of the cloudAtmos native file systemMounted to a subtenantClient mountpoint:/mnt/mauifsEven though it is/mnt/mauifs/TID/SIDMultiple AtmosFS clients may point to the same subtenantIFS is inherently clusteredImproves performance and parallelismAtmos ports must be openCan act as a gateway for NFS/CIFSMatch IFS version to Atmos version,RHEL5 Server,“Boston”,Application,Q1:Pushing a File to Atmos:FS Client,MDLS,MDS,RM,SS,SS,SS,ext3,ext3,ext3,MDS,PM,RM,Linux OS,/mnt/mauifsMaui client,1,1,1.File passed to Maui mount point,2,2.Client queries RM for MDLS,4,4.Create request issued to MDS,5,5.Policy queried for layout,6,LSO,6.Layout(LSO)and OID returned to Client,7,7.Client writes data to selected resources Proprietary protocols,3,3.Client queries MDLS for MDS that owns the parent directory,NFS and CIFS Access,NFS or CIFS configured per Atmos NodeNFS service or SMB service startedLayered on top of the AtmosFS(IFS)that runs on each nodeNo automatic failover of access nodes,Unix Clients,Windows Clients,CIFS Configuration,Atmos exports CIFS at node(tenant)level Tenant or subtenant admin manages the shares Regardless of the access method,all objects are defined using global unique identifier(object ID)All files are defined using a global,unique identifierTranslation from traditional namespace to the object ID transparentSub-tenant sharesAbility to access CIFS share if multi-subtenant access is enabledLeverage same authentication model that of TenantsCIFS User Management is managed using AD integrationAtmos CIFS share ACL detailsDirectory and file level permission inheritanceACL inheritanceDefault permissions configurationEase of CIFS share management and configurationMulti-level admin control based on sub-tenant and or tenant access rights,CIFS Access Architecture,EMC Atmos 1.2.x,Share Y,Tenant ID(Tenant 2),Share A,Share B,CIFS Clients,Access Point,Access Point,Access Point,Access Point,Access Point,Access Point,Atmos Linux Driver,SMBd,FUSE 2.4.7,VFS,User,Kernel,Tenant ID(Tenant 1),Atmos NFS Configuration,Atmos supports NFS v3.0Atmos exports NFS at node(tenant)level Tenant admin manages the shares Regardless of the access method,all objects are defined using global unique identifier(object ID)Translation from traditional namespace to the object ID transparentEase of share configuration and management Authentication managementPolicy configuration and management based on system metadata,NFS Access,EMC Atmos v1.2.x,Share Y,Tenant ID(Tenant 2),Share A,Share B,CIFS Clients,Access Point,Access Point,Access Point,Access Point,Access Point,Access Point,Atmos Linux Driver,NFS