HPUNIX内存热点问题及案例分享.ppt

内存热点问题及案例分享,张永军企业客户技术支持中心支持服务事业部中国惠普有限公司2004.2,检查内存,1.dmesg:Memory Information:physical page size=4096 bytes,logical page size=4096 bytes Physical:1048576 Kbytes,lockable:790684 Kbytes,available:918096 Kbytes2.sam:samPerformance MonitorsSystem PropertiesMemory3.syslog.log4.glance,Memory的角色,三个部分:1.Physical memory2.Available MemoryThe amount of main memory not reserved for the kernel is termed available memory.Available memory is used by the system for executing user processes.3.Lockable MemoryPages kept in memory for the lifetime of a process by means of a system call(such as mlock,plock,or shmctl)are termed locked memory.Locked memory cannot be paged and processes with locked memory cannot be deactivated.Kernel parameter:unlockable_mem.Total available system memory-unlockable_mem=lockable memory,检查空闲内存,1.TopMemory:126108K(79812K)real,419716K(324852K)virtual,98184K free2.Glance3.Vmstat#vmstat 1 1 procs memory page faults cpu r b w avm free re at pi po fr de sr in sy cs us sy id 1 0 0 93405 5989 5 1 0 0 0 0 0 448 737 133 1 1 99,与内存相关的内核参数,Buffer cache related:nbuf and bufpages=0 表示使用动态内存 dbc_max_pct50 dbc_min_pct5 Buffer cache used by File system.建议调低dbc_max_pct(尤其大量使用raw device)，否则占用太多内存。,与内存相关的内核参数(续),vx_ninode=0 表示使用动态内存,可能占用很大内存,造成系统内存资源耗尽，还可能造成superpagepool corruption Or fragmentation。unlockable_mem 使用缺省值(=0),内存的分配,内存分配的三种方式，需要相同数量的swap.plain memory as allocated with malloc(3C)system call.shared memory as allocated with shmget(2)system call.memory mapped files as allocated withmmap(2)system call.,内存的分配（续）,一般内存的分配由malloc(3C)系统调用完成a)To report process memory usage,by largest first,run:#ps efl|sort rnk 10|moreAnd look at 10th column(SZ)to see the amount of memory used by this process for data/text and stack(as this value is in pages,you can multiply by 4096 to determine the size in bytes.)Anytime you see that the size(SZ)is a four-digit number,watch it over time and see if it continues to grow.Alternative ps command-Alternatively,you can use the UNIX95 options to look at both Virtual Size as well as the actual Size.Run:#UNIX95=1 ps-efo vsz,sz,pid,args|grep-v grep|sort-rnk 1|more,内存的分配（续）,b)And another way is to sort glances process list(or application list)by RSS(or Res Mem Virt Mem).For example:PROCESS LIST Users=5 User CPU Util Cum Disk ThdProcess Name PID PPID Pri Name(100%max)CPU IO Rate RSS Cnt-pax 13819 13818 148 root 2.7/5.8 273.3 9.4/32.8 284kb 1glance 14464 1822 158 root 2.1/3.1 3.0 0.0/2.1 4.3mb 1scopeux 1715 1 127 root 1.7/0.2 518.4 1.5/0.0 4.1mb 1swapper 0 0 127 root 1.5/0.8 2213.0 0.3/0.0 16kb 1java 10095 1 168 root 1.0/2.7 348.7 0.0/4.2 42.0mb 28vxfsd 35 0 138 root 0.2/0.1 289.4 1.9/1.3 352kb 16c)Unsupported tools:procsize&kmeminfo,内存的分配（续）,2)共享内存的分配:shmget(2)system call.a)To see shared memory usage,ipcs can be used.For example,run:#ipcs mb|more To totalthe shared memory usage,run:#ipcs-mb|sed-n/m/p|awk total+=$NFENDprintf(“%dn”,total)b)unsupported tool called shminfo,内存的分配（续）,3)内存映像文件:mmap(2)system call.No tools available but unsupported shminfo.,用户进程内存限制,maxdsiz,maxssiz,maxtsizShared memory limit:shmmaxswap,32bit app,解释,对通常的程序来说：第一象限用来存放app text第二象限用来存放data&stack第三四象限用来存放共享的数据The default usage for the quadrants is:Quadrant 1 Process textQuadrant 2 Process dataQuadrant 3 Shared libraries Shared memory Memory mapped filesQuadrant 4 Shared libraries Shared memory Memory mapped filesThe last part of quadrant 4 is reserved for system I/O.,32bit app Private 部分,Maxdsiz&maxssiz 所决定的数据段和堆栈段位于同一个象限中，总和不能大于1GB。Maxdsiz&maxssiz应该合理配置，否则互相影响Maxssiz一般设置成64MB已足够正常32bit应用，数据段最大为940MB安装补丁，最大可以1.9GB。编译选项 ld-NOR“chatr+q3p enable executable_name”to enable third quadrant private,32bit app 共享部分,第3、4象限用来做32bit app的共享寻址最大1.75GBSHMEM_MAGIC 将第二象限用于共享数据(这时最大为2.75GB)To get SHMEM_MAGIC,the executable needs to have been linked(ld)with EXEC_MAGIC(the-N option)and chatred with SHMEM_MAGIC(the-M option).,64bit app,For 64 bit processes the quadrant sizes are 4tb,and therefore the maximum sizes for text,data,shared memory,etc,are quite large.For 64 bit 11.00/11i executables there is currently no need to have different memory maps available as the standard one allows up to 4TB for the program text,another 4TB for its private data and a total of 8TB for shared areas.,考虑一下,16GB物理内存将多个数据库置于同一台机器 对32位数据库，只有1.75GB内存可用怎么办？,Memory Windows!,What does the Memory Windows feature do?,Enabling memory windows does change the kernels default allocation policy.With only the global memory window configured,the kernel allocates shared addresses from the 4th quadrant first and then the 3rd quadrant.When max_mem_window is set to a non-zero value(enabling memory windows),the kernel changes the default allocation policy such that the 3rd quadrant is tried first followed by the 4th.,内核设置,To enable the use of memory windows the kernel tunable max_mem_window must be set to the desired amount of memory windows.The disabled value is 0.A good default is 256.Note:This tunable is not dynamic.,内核设置,To enable 256 memory windows in the kernel add the following line to the kernel configuration file(/stand/system):max_mem_window 256Build a new kernel and reboot.Setting max_mem_window to 256 results in creating 256 memory windows,plus the global memory window.Setting max_mem_window to 1 would create 1 memory window,plus the global window,for a total of 2.,Services.window,Modify/etc/services.window.database1 20database2 30database3 40,相关命令,getmemwindow is the command used to extract window ids of user processes from the/etc/services.window file.setmemwindow is the command that changes the window id of a running process or starts a specified program in a particular memory window.,限制,An application cannot dynamically switch between one window and another.Applications are only allowed to change their window at exec time.Applications are not required to change their source code,but instead use the memory window command setmemwindow to start an application in a specified memory window.,例子,For example:setmemwindow-i 10 myprog arg1 arg2 arg3 This would start the executable myprog with 3 arguments in the memory window corresponding to user key 10.There are other options to setmemwindow to control how memory windows are created.,例子（续）,getmemwindow is used to extract the user ids/keys from the/etc/services.window file given a particular string.This avoids having to embed the keys in the scripts themselves.syntax:getmemwindow,例子（续）,Putting it all together in a real example:#cat startDB1.shWinId=$(getmemwindow database1)setmemwindow-i$WinId/home/dave/memwinbb/startDB1 DB1 says hello!,例子（续）,#./startDB1.shwriting to segment:DB1 says hello!Key is 1377312562,例子（续）,#ipcs-mobIPC status from/dev/kmem as of Thu Jul 10 08:52:34 2003T ID KEY MODE OWNER GROUP NATTCH SEGSZShared Memory:m 0 0 x4124288b-rw-rw-rw-root root 0 348m 1 0 x4e000002-rw-rw-rw-root root 1 61760m 2 0 x41280050-rw-rw-rw-root root 1 8192m 5215 0 x52181f32-rw-r-r-root sys 0 1024Our key of 1377312562 converted to hex is 52181f32,例子（续）,#./startDB2.shwriting to segment:DB2 says Bon Jour!Key is 1377312563#./startDB3.shwriting to segment:DB3 says Ola!Key is 1377042734,例子（续）,#more stopDB1.shWinId=$(getmemwindow database1)setmemwindow-i$WinId/home/dave/memwinbb/stopDB1#./stopDB1.sh#./stopDB2.sh#./stopDB3.sh,工具介绍,pstopglance,ps,#ps efl|sort rnk 10|moreAnd look at 10th column(SZ)to see the amount of memory used by this process for data/text and stack(as this value is in pages,you can multiply by 4096 to determine the size in bytes.)Anytime you see that the size(SZ)is a four-digit number,watch it over time and see if it continues to grow.Alternative ps command-Alternatively,you can use the UNIX95 options to look at both Virtual Size as well as the actual Size.For example:#UNIX95=1 ps-efo vsz,sz,pid,args|grep-v grep|sort-rnk 1|moreVSZ SZ PID COMMAND12252 627 2745/opt/OV/bin/ovdbrun-c/var/opt/OV/share/databases/analysis/9060 1214 2362/opt/omni/lbin/rds-d8808 1892 2677/opt/hpwebjet-5.5/hpwebjetd,top,Load averages:0.64,0.57,0.56 195 processes:194 sleeping,1 running.Memory:1444296K(1238320K)real,1967080K(1468152K)virtual,84908K free|1 23 45Memory is not all of physcial,memory,it is:分配给所有进程的物理内存DEDICATED to text,data or stack segments 所有runnable processes的物理内存，不包括sleeping processes的。分配给所有进程的虚拟内存DEDICATED to text,data or stack segments。不是所有都在物理内存，有部分在swap区中。分配给所有进程的虚拟内存，不包括sleeping or stopped processes新进程的可用物理内存，当比较低时，swapping.,Top（续）,Size is virtual memory sizeRES means the part in RAMWhat if too many CPUs?top hWhat if I want to get all process info?top n500 f top.out,glance,Total VM:121.1mb Sys Mem:13.8mb User Mem:91.3mb Phys Mem:144.0mb Active VM:73.7mb Buf Cache:26.4mb Free Mem:12.6mb Total VM:The total private virtual memory(in KBs unless otherwise specified)at the end of the interval.virtual allocation of private data and stack regions for all processes.Active VM:The total virtual memory(in KBs unless otherwise specified)allocated for processes currently are on the run queue or processes that have executed recently.Sys Mem:The amount of physical memory KBs unless otherwise specified)used by the system(kernel)during the interval.User Mem:The amount of physical memory(in KBs unless otherwise specified)allocated to user code and data at the end of the interval.,Memory hog,1.vx_ninode2.dbc_max_pct3.SuperPagePool 4.Sem related5.Other memory leak,Vx_ninode example:,See file:,Dbc_max_pct,缺省值是50，50%内存可以用于bufferDbc_min_pct 缺省是5%在起机时，buffer 很小随buffer增大，free memory急剧减小Vhand 占据 cpu 忙于换页系统hung,SuperPagePool,11.00:High SuperPagePool freelist usage Caused/introduced by PHKL_25098.Corrected in PHKL_28172(SR JAGae32238 related JAGs:JAGad83301 JAGad86735JAGae25556),SuperPagePool freelist example,See file:,Sem related,See file:,