安装、配置Ubuntu 18.04、KVM和群晖
安装Ubuntu 18.04
安装过程很简单。插上u盘，启动后需要等几分钟才能进系统，E6400的usb 2.0的速度太慢了。一路Next。因为只有一块SSD，就不需要LVM，也可以不划/boot和/swap分区，让Ubuntu自动创建。如果需要划/boot分区，至少分512MB。原来/boot分了128MB，在Ubuntu升级了几次内核后，/boot分区 100%了，又没有用LVM，扩都不好扩。
现在把一块256G的msata SSD挂到光驱位了，把硬盘位留给机械硬盘。
用20.04的u盘在T430上启动不了。只有18.04可以。
安装常用软件
更新源
apt-get update
安装ssh
sudo apt-get install ssh
安装screen
窗口管理器，用惯了screen，有空再试试tmux。
apt-get install screen
安装vim
装个完整的vim会比较顺手。
apt-get install vim
安装kvm
vlan是给单臂路由的场景用的。
apt-get install qemu-kvm qemu-system libvirt-bin virt-manager bridge-utils vlan [2020-08-29 周六 21:52:13] ubuntu 20.04 装libvirt-bin报错 apt-get install qemu-kvm qemu-system virt-manager bridge-utils vlan
安装温度监控
E6400和T430都可以，E6400上显示的多些。
sudo apt-get install lm-sensors # 执行命令sensors-detect，进行简单配置，此命令执行后会出现一系列选项，一直yes即可； # 安装完后 sensors
安装htop
颜色好看，可以看进程中的线程。
apt-get install htop
编译、安装tsar，记录性能数据
本来想让ssd能少写点数据，但是没有tsar记录系统资源使用情况有点不方便。只能装了。
编译、安装emacs 27.0.50
系统稳定以后，有空再装了。
系统常用配置
笔记本合盖不休眠
机器是7*24小时开机的，在合盖后机器不休眠。 https://blog.csdn.net/ezhchai/article/details/80548130
vim /etc/systemd/logind.conf #HandleLidSwitch=suspend 改成： HandleLidSwitch=ignore
重启服务
systemctl restart system-logind
修改hostname
hostname如果在安装的时候没配对，可以修改 /etc/hostname 。
修改网卡名为eth0/wlan0
默认的名字是按总线命名的，看着不习惯，改回原来的方式。
root@E6400:~# vim /etc/default/grub GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" root@E6400:~# update-grub
修改.profile，让时间戳和提示符更好看
- 改为英文，看中文的时间戳不习惯。
- 提示符也改成带颜色的。
vim ~/.profile set -o vi export LC_ALL=en_US.UTF-8 export PS1='\n\e[1;37m[\e[m\e[1;32m\u\e[m\e[1;33m@\e[m\e[1;35m\H\e[m \e[4m`pwd`\e[m\e[1;37m]\e[m\e[1;36m\e[m\n\$'
关闭不用的服务
systemctl disable cups
安装驱动
安装bcm43224无线网卡驱动
- 问题
- 使用不同的驱动，发现有些驱动会导致ping的延时高
唉，忘记到底是怎么把无线网卡驱动装上去的了。
参考： https://help.ubuntu.com/community/WifiDocs/Driver/bcm43xx#brcmfmac_driver_.28Open-source.29
Installing STA drivers
root@E6400:~# apt-get --reinstall install bcmwl-kernel-source root@E6400:~# apt-get --reinstall install broadcom-sta-common
# Switching between drivers # If you card is supported by more than one driver then use the modprobe command to test the drivers. First unload all conflicting drivers (this includes removing the driver you're trying to install): sudo modprobe -r b43 bcma sudo modprobe -r brcmsmac bcma sudo modprobe -r wl # To load a specific driver use one of the following commands: sudo modprobe b43 sudo modprobe brcmsmac sudo modprobe wl
这是用了b43(不知道是什么)和brcmsmac(开源驱动，支持bcm43224芯片)的。
root@E6400:~# lsmod|grep wl wl 6447104 0 cfg80211 675840 4 wl,b43,mac80211,brcmsmac
安装b43驱动
sudo apt-get install firmware-b43-installer
网卡直通
updated: 使用 macvtap 方式，不再用直通方式，宿主机只有一个千兆电口。
[albert@T430 /home/albert]
$lspci -nn|grep -i network
00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) [8086:1502] (rev 04)
03:00.0 Network controller [0280]: Intel Corporation Centrino Ultimate-N 6300 [8086:4238] (rev 35)
dsm5.2
dsm5.2的功能正常，可以网卡直通。就是版本稍微有点低。
创建vm：
virt-install --name dsm5.2 --memory 2000 --vcpus sockets=1,cores=1,threads=1 --disk device=cdrom,path=/home/albert/dsm/dsm5.2/XPEnoboot_DS3615xs_5.2-5967.1.iso --network bridge=br0,model=e1000e --noautoconsole --accelerate --hvm --graphics vnc,listen=0.0.0.0,port=20008 --video vga --input tablet,bus=usb --cpu host-passthrough
T430 dsm6.2
virt-install --name dsm6.2 --memory 4000 --vcpus sockets=1,cores=1,threads=1 --disk device=cdrom,path=/home/albert/dsm6.2/synoboot_dsm_ds918_24922_pve.img --network bridge=br0,model=e1000e --noautoconsole --accelerate --hvm --graphics vnc,listen=0.0.0.0,port=20008 --video vga --input tablet,bus=usb --cpu host-passthrough
硬盘优化
参考pve的 https://pve.proxmox.com/wiki/Pci_passthrough
# 增加iommu的支持 $vim /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on" # 更新内核参数，重启生效 $update-grub # 检查iommu的参数是否生效 root@E6400:~# dmesg|grep IOMM [ 0.091713] DMAR: IOMMU enabled [ 0.467216] DMAR: Disabling IOMMU for graphics on this chipset
使用vfio。
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
https://forums.unraid.net/topic/46634-passthrough-hard-drive-to-vm/
https://turlucode.com/qemu-disk-io-performance-comparison-native-or-threads-windows-10-version/#1528572626148-2b30f3e4-f00f kvm的各种模式的解释和对比
Centos6配置KVM使用virtio-scsi（multi-queues）
https://blog.csdn.net/bobpen/article/details/41515119 Centos6下Virtio-SCSI(multi-queues)/Virtio-SCSI/Virtio-blk性能对比 - 多队列的性能也一般
https://www.jianshu.com/p/0288235387c7 深入浅出KVM（三） 丨 I/O 全虚拟化和准虚拟化
只有一个vcpu，设不设都一样 添加driver queues字段至xml文件中，driver queues等于vcpu队列数
<disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='threads'/> <source dev='/dev/disk/by-id/ata-ST320LT007-9ZV142_W0Q5K6ZN'/> <backingStore/> <target dev='sdb' bus='scsi'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> <dirver queues='8' /> </controller>
https://blog.csdn.net/bobpen/article/details/41515263
内存优化 - 使用HugePage
给虚机分配2G内存，需要1000个2MB的页。
# 指定大页需要的内存页面数，永久生效，1024*2MB=2048MB内存作为hugepages sysctl -w vm.nr_hugepages=1024 # 或者 #vim /etc/sysctl.conf vm.nr_hugepages=1024 # 让配置生效 sysctl -p
检查HugePage是否ok：
[root@E6400 /root] #cat /proc/meminfo |grep Huge AnonHugePages: 0 kB ShmemHugePages: 0 kB HugePages_Total: 1024 HugePages_Free: 1024 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 2097152 kB
KSM优化
只起了一个虚机，不需要os来压缩vm的内存。可以关掉ksm。
为什么没地方停ksm呢？
https://serverfault.com/questions/470093/why-kernel-shared-memory-is-0-on-ubuntu-12-04#
echo 2 > /sys/kernel/mm/ksm/run
How to use the Kernel Samepage Merging feature
run - set 0 to stop ksmd from running but keep merged pages,
set 1 to run ksmd e.g. "echo 1 > /sys/kernel/mm/ksm/run",
set 2 to stop ksmd and unmerge all pages currently merged,
but leave mergeable areas registered for next run
Default: 0 (must be changed to 1 to activate KSM,
except if CONFIG_SYSFS is disabled)
只能把扫描时间加长了？提到的配置文件找不到 https://forum.proxmox.com/threads/ksm-is-using-about-30-of-cpu-time.3857/page-2
usb passthrough
为什么group 5的4个设备都要删除，kvm才不会报错？
https://forums.unraid.net/topic/35112-guide-passthrough-entire-pci-usb-controller/
确定usb设备的总线位置，可以插个u盘上去后，看u盘对应的总线号。
root@E6400:~# lsusb Bus 002 Device 005: ID 0781:5567 SanDisk Corp. Cruzer Blade - 左下esata Bus 002 Device 003: ID 0ed1:6981 WinMaxGroup - 左上 Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub - 右上 Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 006 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver - 右下 Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 005 Device 002: ID 0a5c:5801 Broadcom Corp. BCM5880 Secure Applications Processor with fingerprint swipe sensor Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub root@E6400:~# readlink /sys/bus/usb/devices/usb2 ../../../devices/pci0000:00/0000:00:1d.7/usb2
将iommu_group下的设备unbind，将设备unbind之后，目录下面的driver目录会被删除
echo "0000:00:1d.7" >>/sys/bus/pci/devices/0000\:00\:1d.7/driver/unbind echo "0000:00:1d.0" >>/sys/bus/pci/devices/0000\:00\:1d.0/driver/unbind echo "0000:00:1d.1" >>/sys/bus/pci/devices/0000\:00\:1d.1/driver/unbind echo "0000:00:1d.2" >>/sys/bus/pci/devices/0000\:00\:1d.2/driver/unbind
执行后/sys/bus/pci/devices/0000\:00\:1d.7/下的driver目录被删除
group5下面的4个usb controller全部都要删除掉
将设备地vendorid和deviceid绑定到new_id
# usb ehci控制器 echo "8086 293a" >/sys/bus/pci/drivers/vfio-pci/new_id # 17.0 echo "8086 2934" >/sys/bus/pci/drivers/vfio-pci/new_id # 17.1 echo "8086 2935" >/sys/bus/pci/drivers/vfio-pci/new_id # 17.2 echo "8086 2936" >/sys/bus/pci/drivers/vfio-pci/new_id
检测iommu_group是否绑定成功，会发现
/dev/vfio下多了1(intel 82576LM千兆网卡)和5(usb controller)2个组号[root@E6400 /dev/vfio] #ls /dev/vfio 1 5 vfio
查看网卡和usb驱动是否为vfio-pci
# 查看pci具体某个bdf的信息 [2019-09-15 周日 22:41:29] albert@E6400:~$ lspci -nnvv -s 00:1d.7 00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) (prog-if 20 [EHCI]) Subsystem: Dell 82801I (ICH9 Family) USB2 EHCI Controller [1028:0233] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 20 Region 0: Memory at fed1c000 (32-bit, non-prefetchable) [size=1K] Capabilities: <access denied> Kernel driver in use: vfio-pci
nic passthrough
确认IOMMU开启，就是BIOS与IOMMU相关选项打开（Intel VT-d或者AMD IOV）
[albert@E6400 /home/albert] $dmesg|grep IOMM [ 0.091598] DMAR: IOMMU enabled [ 0.581651] DMAR: Disabling IOMMU for graphics on this chipset查看kvm内核模块是否已加载
root@E6400:~# lsmod|grep kvm kvm_intel 212992 3 kvm 598016 1 kvm_intel irqbypass 16384 5 kvm
确认Passthrough的设备
使用
lspci -nn列出所有PCI设备和地址，记录下intel 82567LM ids为 [8086:10f5]root@E6400:~# lspci -nn|grep Eth 00:19.0 Ethernet controller [0200]: Intel Corporation 82567LM Gigabit Network Connection [8086:10f5] (rev 03) root@E6400:~# find /sys/kernel/iommu_groups/ -type l|grep 19 /sys/kernel/iommu_groups/1/devices/0000:00:19.0
查看这两个设备属于的IOMMU组，如果没有显示任何内容，要确认BIOS中相应Intel VT-d打开与否。查找之前记录的网卡id，属于group1。如果在该组中化存在其他设备需要为kernel添加ACS Patch，相应的方法参见链接。如果只有一个组，则进行下一步。
root@E6400:~# find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/7/devices/0000:00:1f.2 /sys/kernel/iommu_groups/7/devices/0000:00:1f.0 /sys/kernel/iommu_groups/7/devices/0000:00:1f.3 /sys/kernel/iommu_groups/5/devices/0000:00:1d.1 ---> 4个都要unbind? /sys/kernel/iommu_groups/5/devices/0000:00:1d.2 /sys/kernel/iommu_groups/5/devices/0000:00:1d.0 /sys/kernel/iommu_groups/5/devices/0000:00:1d.7 ---> usb2.0 左上？ /sys/kernel/iommu_groups/3/devices/0000:00:1b.0 /sys/kernel/iommu_groups/1/devices/0000:00:19.0 ---> intel 82567LM /sys/kernel/iommu_groups/6/devices/0000:03:01.1 /sys/kernel/iommu_groups/6/devices/0000:00:1e.0 /sys/kernel/iommu_groups/6/devices/0000:03:01.2 /sys/kernel/iommu_groups/6/devices/0000:03:01.0 /sys/kernel/iommu_groups/4/devices/0000:0c:00.0 /sys/kernel/iommu_groups/4/devices/0000:00:1c.0 /sys/kernel/iommu_groups/4/devices/0000:00:1c.3 /sys/kernel/iommu_groups/4/devices/0000:00:1c.1 /sys/kernel/iommu_groups/4/devices/0000:00:1c.2 /sys/kernel/iommu_groups/2/devices/0000:00:1a.1 /sys/kernel/iommu_groups/2/devices/0000:00:1a.2 /sys/kernel/iommu_groups/2/devices/0000:00:1a.0 /sys/kernel/iommu_groups/2/devices/0000:00:1a.7 /sys/kernel/iommu_groups/0/devices/0000:00:00.0
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf echo "options vfio-pci ids=10f5:8086" > /etc/modprobe.d/vfio.conf echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
[root@E6400 /root] #virsh start openwrt error: Failed to start domain openwrt error: internal error: process exited while connecting to monitor: 2019-08-18T12:33:10.046528Z qemu-system-x86_64: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x3: vfio error: 0000:00:19.0: failed to setup container for group 1: failed to set iommu for container: Operation not permitted
做个记号，virt中网卡pass through的配置，使用vfio。
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
T430
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
定位cpu高的问题 - 都是和各种timer有关
top时看见没有压力的情况下，宿主机的cpu就到15%。
perf top -p pid # 发现 read_hpet 占得最高，不正常
删除了usb tablet，删除了usb设备，重新把ich9的ehci加上去就ok了。
<timer name='hpet' present='yes'/> - 改了以后到2.3%了
看看qemu的cpu还是不是有4%左右
参考了这个 https://askubuntu.com/questions/1033985/kvm-high-host-cpu-load-after-upgrading-vm-to-windows-10-1803
https://forum.proxmox.com/threads/high-cpu-load-for-windows-10-guests-when-idle.44531/
<timer name='pit' tickpolicy='delay'/>
tickpolicy改为 discard 。
增加 synic和stimer - cpu —> 1.5%以下了
不知道为什么配置了以后，硬盘找不到了，启动的时候可能是时钟相关的 hv_ 模块报错了，但是vnc太快了看不见
https://libvirt.org/formatdomain.html
<features>
<acpi/>
<apic/>
<hyperv>
<synic state='on'/>
<stimer state='on'>
<direct state='on'/>
</stimer>
</hyperv>
</features>
synic Enable Synthetic Interrupt Controller (SynIC) on, off 1.3.3 (QEMU 2.6)
stimer Enable SynIC timers, optionally with Direct Mode support
好像和 kvm:kvm_hv_timer_state event有关
65,201 kvm:kvm_write_tsc_offset
里面有tsc相关内容，不调整了，tsc比较重要 https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Virtualization_Deployment_and_Administration_Guide/sect-Virtualization-Tips_and_tricks-Libvirt_Managed_Timers.html
dsm6.2
dsm6.2是各种问题，搞不定。
修改img - 非LVM分区挂载方法
调整img中的启动选项的时间，timeout太快了，都来不及选。
albert@E6400:~/dsm/dsm6.2$ fdisk -ul synoboot-ds3617_6.2-1.02b-uefi.img
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: B3CAAA25-3CA1-48FA-A5B6-105ADDE4793F
Device Start End Sectors Size Type
synoboot-ds3617_6.2-1.02b-uefi.img1 2048 32767 30720 15M EFI System
synoboot-ds3617_6.2-1.02b-uefi.img2 32768 94207 61440 30M Linux filesystem
synoboot-ds3617_6.2-1.02b-uefi.img3 94208 102366 8159 4M BIOS boot
可以看见linux的文件系统是从第2048块开始的，所以挂载的时候应该从这个地方开始挂载。
由这条信息可以的出，扇区大小为512。
Units = sectors of 1 * 512 = 512 bytes
所以我们需要从512*2048出开始挂载，计算以字节为单位的偏移量。
2048 * 512 = 1048576
修改群晖的mac和sn，mount 第一个分区
Device Start End Sectors Size Type synoboot-ds3617xs-dsm617-20200824.img1 2048 32767 30720 15M EFI System synoboot-ds3617xs-dsm617-20200824.img2 32768 94207 61440 30M Linux filesystem synoboot-ds3617xs-dsm617-20200824.img3 94208 102366 8159 4M BIOS boot mount -o loop,offset=1048576 /home/albert/dsm/dsm6.1/synoboot-ds3617xs-dsm617-20200824.img /mnt/dsm6.1 # 192.168.0.235 set sn=A8ODN02468 set mac1=0011322CA603 # [2020-09-01 周二 11:01:48] https://hlynford.com/124.html set sn=1230ODN515525 set mac1=001132F44E18
mount -o loop,offset=1048576 synoboot-ds3617_6.2-1.02b-uefi.img /mnt/dsm6.2 # 菜单选项改为20s cd /mnt/dsm6.2 vim grub/grub.cfg set timeout='20' mount -o loop,offset=1048576 synoboot-ds3617_6.2-1.02b-uefi.img /mnt/dsm6.2
看看ds3617的6.1.7的extra.lzma
#mount -o loop,offset=16777216 synoboot-ds3617xs-dsm617.img /mnt/dsm6.1 mount -o loop,offset=16777216 /home/albert/dsm/dsm6.1/synoboot-ds3617xs-dsm617-20200824.img /mnt/dsm6.1 # [2020-08-24 周一 15:26:13] 用这个里面的virtio的驱动是可以的，但是没有virtio-scsi的驱动，我自己去年编译的启动就kernel panic。 #mount -o loop,offset=16777216 /home/albert/dsm/dsm6.1/boot_ds3615xs_15132_kvm.img /mnt/dsm6.1 [2020-08-24 周一 18:02:56] #mount -o loop,offset=22020096 /home/albert/dsm/dsm6.1/DS3617xs_DSM6.1_Broadwell.img /mnt/dsm6.1 cp /mnt/dsm6.1/extra.lzma /tmp/ds3617 cd /tmp/ds3617 # 解压 lzma -d extra.lzma cpio -idv < extra rm extra # 打包 cd /tmp/ds3617 (find . -name modprobe && find . \! -name modprobe) | cpio --owner root:root -oH newc | lzma -8 > ../extra.lzma cp /tmp/extra.lzma /mnt/dsm6.1 && ls -ltr /mnt/dsm6.1 # cd /mnt/dsm6.1 umount /mnt/dsm6.1
EXTRA_MODULES="mii mdio libphy atl1 atl1e atl1c alx uio ipg jme skge sky2 ptp_pch pch_gbe qla3xxx qlcnic qlge netxen_nic sfc e1000 pcnet32 vmxnet3 bnx2 li bcrc32c bnx2x cnic e1000e igb ixgbe r8101 r8168 r8169 tg3 usbnet ax88179_178a button evdev ohci-hcd virtio virtio_ring virtio_pci virtio_blk virtio_net" DISK_MODULES="BusLogic vmw_pvscsi megaraid_mm megaraid_mbox megaraid scsi_transport_spi mptbase mptscsih mptspi mptsas mptctl ata_piix megaraid_sas mpt2sa s mpt3sas virtio virtio_ring virtio_pci virtio_blk" EXTRA_FIRMWARES="bnx2/bnx2-rv2p-09ax-6.0.17.fw bnx2/bnx2-rv2p-09-6.0.17.fw bnx2/bnx2-rv2p-06-6.0.15.fw tigon/tg3_tso5.bin tigon/tg3_tso.bin tigon/tg3.bin"
全部重新编译
cd /home/albert/dsm6.1/virtio-20200825 cp -p /home/albert/dsm6.1/linux/drivers/block/virtio_blk.ko . cp -p /home/albert/dsm6.1/linux/drivers/net/virtio_net.ko . cp -p /home/albert/dsm6.1/linux/drivers/virtio/virtio.ko . cp -p /home/albert/dsm6.1/linux/drivers/virtio/virtio_pci.ko . cp -p /home/albert/dsm6.1/linux/drivers/virtio/virtio_ring.ko . cp -p /home/albert/dsm6.1/linux/drivers/scsi/virtio_scsi.ko . scp -rp /home/albert/dsm6.1/virtio-20200825 192.168.0.43:~/dsm/dsm6.1/ cd /home/albert/dsm6.1/linux cp -p /home/albert/dsm6.1/linux/drivers/virtio/virtio_balloon.ko . cp -p /home/albert/dsm6.1/linux/drivers/virtio/virtio_mmio.ko . #cp -p /home/albert/dsm6.1/linux/drivers/ata/ahci_platform.ko . #cp -p /home/albert/dsm6.1/linux/drivers/ata/libata.ko . #cp -p /home/albert/dsm6.1/linux/drivers/ata/libahci.ko . #cp -p /home/albert/dsm6.1/linux/drivers/scsi/scsi_transport_spi.ko . 38943298 64 -rw-rw-r-- 1 albert albert 61720 Aug 25 13:53 ./drivers/ata/ahci.ko ./drivers/ata/libahci.ko
# [2020-08-24 周一 16:46:35] dsm5.2可以查看，就是命令有点慢 lspci -kQ 00:06.0 Mass storage controller [0100]: Red Hat, Inc. Virtio SCSI Subsystem: Red Hat, Inc. Device 0008 Kernel driver in use: virtio-pci 00:07.0 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) Subsystem: Red Hat, Inc. QEMU Virtual Machine Kernel driver in use: ahci
改为q35芯片组
原来的： <os> <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type> <bootmenu enable='yes'/> </os>
dsm6.1
从这里下载 http://down.nas2x.com/synology/dsm/6.1/6.1.7/ds3617xs/
dsm 6.1.7是ok的。cpu指定broadwell，可能性能不会太好。
这个版本的问题是没有virtio-scsi的驱动，总是不认盘。
read_hpet导致cpu高问题
原来的配置
<clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='discard'/> <timer name='hpet' present='no'/> </clock>
virtio和scsi都不认盘
<source file='/home/albert/dsm/dsm6.1/DS3617xs_DSM6.1_Broadwell.img'/>
用virtio后，盘是vd开头的，dsm只认sd开头的。所以dsm不认盘，没法用virtio。只能把sata控制器pass through，但是dsm中要有sata控制器的驱动。
[ 15.907316] md: Autodetecting RAID arrays. [ 15.907332] This is not a kind of scsi disk 252 [ 15.908231] md: deny vda1. [ 15.908233] md: export_rdev(vda1) [ 15.908245] This is not a kind of scsi disk 252 [ 15.908557] md: deny vda2. [ 15.908559] md: export_rdev(vda2) [ 15.908565] This is not a kind of scsi disk 252 [ 15.908654] md: invalid raid superblock magic on vda3 [ 15.910197] md: vda3 does not have a valid v0.90 superblock, not importing! [ 15.910201] md: Scanned 3 and added 0 devices. [ 15.910202] md: autorun ... [ 15.910203] md: ... autorun DONE. [ 18.800910] sd 0:0:0:0: Attached scsi generic sg0 type 0
DS3617xs_DSM6.1_Broadwell.img
用virtio-scsi就报错，core，不识别scsi_dispatch_cmd。为什么呢？
用virtio就不认盘。
- ide 认盘，但是性能不好。
- sata 认盘，但是性能不用virtio好
- scsi dmesg直接就看不见有这个盘的log。
- virtio 不认盘，dsm就是要把盘认成scsi，但是却不是scsi的，本来用virtio以后就不是scsi的，如何才能让dsm把vda认为不是scsi的呢? 搞不定
qemu-system-x86_64 -enable-kvm -name guest=dsm6.2,debug-threads=on -S -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,hv_synic,hv_stimer -m 2000 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 7b9ee21c-502b-4661-bf67-1f139e2bc246 -no-user-config -nodefaults -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x6 -device ahci,id=sata0,bus=pci.0,addr=0x7 -drive file=/home/albert/dsm/dsm6.1/DS3617xs_DSM6.1_Broadwell.img,format=raw,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/dsm-scsi.img,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native -device virtio-blk-pci,scsi=on,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:14109 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on 2019-09-27T07:02:12.535671Z qemu-system-x86_64: -device virtio-blk-pci,scsi=on,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0: Please set scsi=off for virtio-blk devices in order to use virtio 1.0
fdisk /dev/vda d # 删除分区 n # create partition p # primary partition +4000M # 一个分区4G t # partition tag fd # raid tag
用同一个盘建RAID1会提示用了同一个盘。
mdadm -Cv /dev/md0 -a yes -l 0 -n 2 /dev/vda{1,2}
手工创建RAID1，但是在安装时报格式化磁盘失败。(35)
DiskStation> mdadm -Cv /dev/md0 -l1 -n2 /dev/vda[12]
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: size set to 3905408K
DiskStation> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md0 : active raid1 vda2[1] vda1[0]
3905408 blocks super 1.2 [2/2] [UU]
[==>..................] resync = 14.9% (584448/3905408) finish=0.4min speed=116889K/sec
unused devices: <none>
fdisk -ul mount -o loop,offset=1048576 /home/albert/dsm/dsm6.1/DS3617xs_DSM6.1_Broadwell.img /mnt/dsm6.2 mount -o loop,offset=22020096 /home/albert/dsm/dsm6.1/DS3617xs_DSM6.1_Broadwell.img /mnt/dsm6.2_img2
baidu: udev site:xpenology.com 考虑使用udev把 /dev/vda 改为 /dev/sda
http://blog.chinaunix.net/uid-31410005-id-5789856.html 如何让KVM虚拟机显示sda而不是vda - 没地方改啊
https://xpenology.com/forum/topic/12470-how-to-play-around-with-hdd-mapping-with-sataportmap-and-diskidxmap/ 没有用udev改名的内容，但是又 sataportmap的解释
https://xpenology.com/forum/topic/5982-how-does-the-xpenology-bootloader-work/?tab=comments#comment-51884 说了virtio的盘符是vda的问题，但是没解决
http://www.gebi1.com/thread-292649-1-1.html 关于硬盘直通黑裙的问题！
https://xpenology.com/forum/topic/12866-drivers-requests-for-dsm-62/page/3/ 里面有一些如何编译内核的东西
https://github.com/SynologyOpenSource/pkgscripts-ng 编译
https://github.com/kref/scripts/blob/master/mkramdisk 到底用哪个？
T430和ds3617xs、synoboot-ds3617xs-dsm617.img
dsm 6.1.7 启动后可以通过 virsh console dsm617 查看启动的log，而且可以在没挂盘时，root免密码登录。这点很有帮助，可以确定网卡、磁盘是否正常。
ip 192.168.0.235
关闭ksmd
echo 2 > /sys/kernel/mm/ksm/run
安装dsm 6.1.7后报，无法安装此文件，文件可能已经损毁。（13）的问题
这个问题定位了很久，换了3615xs和3617xs的好几个boot文件。
在使用synoboot-ds3617xs-dsm617.img发现，
/var/log/messages中有报错 boot/boot_lock.c(226): failed to mount boot device /dev/synoboot2 /tmp/bootmnt site:xpenology.com
google上面的关键字后，在 https://xpenology.com/forum/topic/7387-tutorial-dsm-6x-on-proxmox/page/5/ 中发现
agent: 1 args: -device 'piix3-usb-uhci,addr=0x18' -drive 'id=synoboot,file=/var/lib/vz/images/102/synoboot.img,if=none,format=raw' -device 'usb-storage,id=synoboot,drive=synoboot' bios: seabios
https://archive.synology.com/download/DSM/release/6.1/15047/
image用的是usb，原来我都是用virtio等类型，思维没有转过来，在virt-manager中改为usb后，重新装系统后，居然ok了。
自己的盘只能用ide，用sata、scsi和virtio都会在黑群晖启动时报没有insert hdisk。。。ide的性能不太好。
物理黑群
https://xpenology.com/forum/topic/25833-tutorial-use-linux-to-create-bootable-xpenology-usb/ 在T430上制作u盘
root@T430:~# lsusb Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 005: ID 5986:02d5 Acer, Inc Bus 001 Device 004: ID 147e:2020 Upek TouchChip Fingerprint Coprocessor (WBF advanced mode) Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 002: ID 0ed1:6981 WinMaxGroup --> Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub # sandisk 64G u盘是这个 [2020-08-17 周一 15:58:53] Bus 002 Device 009: ID 0781:5571 SanDisk Corp. Cruzer Fit [2020-08-20 周四 12:42:20] 32g sandisk Bus 002 Device 017: ID 0781:5567 SanDisk Corp. Cruzer Blade
vid = 0ed1 pid = 6981
fdisk -l Disk /dev/sdb: 7.5 GiB, 8075083776 bytes, 15771648 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x3bc5958c Device Boot Start End Sectors Size Id Type /dev/sdb1 * 1142528 15771647 14629120 7G c W95 FAT32 (LBA) # 确定u盘是/dev/sdb [2020-08-17 周一 15:18:16] sudo dd if=/home/albert/dsm6.1.7/synoboot-ds3617xs-dsm617.img of=/dev/sdb bs=512 102400+0 records in 102400+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 17.0841 s, 3.1 MB/s dd if=/home/albert/dsm6.1.7/synoboot-ds3617xs-dsm617.img of=/dev/sdd bs=512 dd if=/home/albert/dsm/synoboot-ds3617xs-dsm617.img of=/dev/sdc bs=512 # [2020-08-20 周四 12:42:57] dsm3617 dsm6.2 dd if=/home/albert/dsm/dsm6.2/dsm.6.2.synoboot_ds3617xs.1.03b.img of=/dev/sdc bs=512
# 再看看 fdisk -l GPT PMBR size mismatch (102399 != 15771647) will be corrected by w(rite). The backup GPT table is corrupt, but the primary appears OK, so that will be used. Disk /dev/sdb: 7.5 GiB, 8075083776 bytes, 15771648 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: C94E55EA-A4D2-4E78-9D73-46CBAE7A03EF Device Start End Sectors Size Type /dev/sdb1 2048 32767 30720 15M EFI System /dev/sdb2 32768 94207 61440 30M Linux filesystem /dev/sdb3 94208 102366 8159 4M BIOS boot
sudo mount /dev/sdb1 /mnt/usb/part1 sudo mount /dev/sdb2 /mnt/usb/part2 sudo mount /dev/sdb3 /mnt/usb/part3 mount /dev/sdc1 /mnt/usb/part1 mount /dev/sdc2 /mnt/usb/part2 mount /dev/sdc3 /mnt/usb/part3
最低可以到800.00 albert@E6400:~/dsm/dsm6.2$ cat /proc/cpuinfo |grep MH cpu MHz : 821.117 cpu MHz : 828.002
黑群 我们检测到硬盘5,6有错误 - 不小心点了在线安装，把u盘的img升级到最新的6.2.3版本了，所以报这个错。
dsm617 shell在vi mode中tab不是命令补齐的问题
https://stackoverflow.com/questions/2252183/tab-autocompletion-in-bash-vi-shell-mode
admin@dsm617:/$ bind -q complete complete can be invoked via "\C-i", "\e\e". admin@dsm617:/$ set -o vi admin@dsm617:/$ bind -q complete complete is not bound to any keys. # 要重新设置一下，为什么？ bind '"\C-i":complete'
vmm套件
centos 7.5 yum源
更新yum源为aliyun https://www.cnblogs.com/kcxg/p/11457180.html
aliyun官方CentOS镜像 https://developer.aliyun.com/mirror/centos?spm=a2c6h.13651102.0.0.3e221b11PawzPA 阿里云官方镜像站 https://developer.aliyun.com/mirror/
# 1. backup mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak # 2. 下载新的CentOS-Base.repo 到/etc/yum.repos.d/ curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo # 3. 添加EPEL curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo # 4. 清理缓存并生成新的缓存 yum clean all yum makecache
centos dig
https://www.cnblogs.com/wangboyu/p/11934397.html
# 安装dig yum install bind-utils
centos gcc开发环境
yum group install "Development Tools"
git clone https://github.com/alibaba/tsar.git
ubuntu 20.04 server
ubuntu在启动以后，从dsm上top看qemu进程总是有10%左右的cpu，而centos 7.5就没有，奇怪啊。
vm绑定物理核
KVM虚拟机CPU绑定性能调优 https://blog.csdn.net/maoliping455mlp455/article/details/9712641
root@dsm617:~# virsh vcpuinfo 7 VCPU: 0 CPU: 0 State: running CPU time: 316.4s CPU Affinity: yyyy VCPU: 1 CPU: 2 State: running CPU time: 347.6s CPU Affinity: yyyy
Ubuntu 18.04最小安装
1.Ubuntu最小安装
2.添加ppa： sudo add-apt-repository ppa:webupd8team/java sudo apt-get update
3.卸载亚马逊链接： sudo apt-get remove unity-webapps-common 没有
4.卸载一些不常用的链接： sudo apt-get remove thunderbird 没有 totem 没有，只有libtotem rhythmbox 没有 empathy 没有 brasero simple-scan gnome-mahjongg 没有 aisleriot sudo apt-get remove gnome-mines 没有 cheese transmission-common gnome-orca webbrowser-app gnome-sudoku landscape-client-ui-install sudo apt-get remove onboard deja-dup
cheese-common
5.卸载LibreOffice： sudo apt-get remove libreoffice-common
6.卸载Firefox浏览器： sudo apt-get remove firefox*
7.安装Git： sudo apt-get install git
8.安装Vim： sudo apt-get install vim
9.安装Chrome浏览器： wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb sudo dpkg -i *.deb
10.安装oracle-java-installer： sudo apt-get install oracle-java8-installer
11.安装idea： (1).下载idea (2).解压到/opt目录下 sudo tar -zxvf ideaIU-2016.3.3-no-jdk.tar.gz -C /opt (3).进入到bin文件夹： cd /opt/idea-IU-163.11103.6/bin (4).创建快捷方式文件 idea.desktop： vim idea.desktop (5).编辑 idea.desktop 文件： [Desktop Entry] Name=IntelliJ IDEA Comment=IntelliJ IDEA Exec=/home/longsky/Application/idea-IU-163.7743.44/bin/idea.sh Icon=/home/longsky/Application/idea-IU-163.7743.44/bin/idea.png Terminal=false Type=Application Categories=Developer;
ubuntu16.04系统精简
一、更新系统
apt-get update apt-get dist-upgrade
二、查看所有内核
sudo dpkg –get-selections |grep linux
三、清除不用的内核
apt-get purge linux-image-extra-x.x.x-x-generic \ linux-image-x.x.x-x-generic \ linux-headers-x.x.x-x \ linux-headers-x.x.x-x-generic
四、删除无用的软件
apt-get purge mdadm \ git \ acpid \ vim* \ ubuntu-cloudimage-keyring \ ubuntu-core-launcher \ software-properties-common \ lxc* \ lxd* \ perl
apt-get autoremove
五、清理文件
apt-get clean all
KVM clock
在dsm617上使用tsc，看见native_write_msr比较多，可能是和时钟有关系，考虑用kvm_clock，但是kvm clock driver在哪里呢？在kvm.ko kvm-intel.ko里面？ dsm启动时是没看见kvm和kvm-intel.ko的，是要在extra.lzma里面加上去吗？ 加了以后，在dsm617里面没看见kvm.ko
<clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='tsc' present='yes'/> <timer name='kvmclock' present='yes'/> </clock>
在/sys/devices/system/clocksource/clocksource0目录下；
- 当前所有可用的clocksource cat /sys/devices/system/clocksource/clocksource0/available_clocksource
- 当前正在使用的clocksource。 cat /sys/devices/system/clocksource/clocksource0/current_clocksource
grep -iE 'paravirt|KVM_CLOCK' /boot/config-* CONFIG_PARAVIRT_CLOCK=y #y：编译进内核
ptp(Precision Time Protocol)
ptp是用来和host进行时钟同步的，现在dsm用了ntp同步，无所谓了。
https://zshisite.wordpress.com/2017/10/25/sync-your-cloud-with-ptp/
dmesg [ 0.406237] PTP clock support registered
docker
v2ray
- access log写到 /dev/null 吧
- log level用默认的warning，不改为error。
cpu火焰图
https://github.com/brendangregg/FlameGraph
perf + 火焰图分析程序性能 https://www.cnblogs.com/happyliu/p/6142929.html
阮一峰 如何读懂火焰图？ http://www.ruanyifeng.com/blog/2017/09/flame-graph.html
cd /home/albert # 1. 用 perf record采集数据 # perf record -F 99 -p 50713 -g -- sleep 60 root@600g4:/home/albert# perf record -p 50713 -g -- sleep 60 [ perf record: Woken up 7 times to write data ] [ perf record: Captured and wrote 1.951 MB perf.data (10331 samples) ] # 2. 格式转换 root@600g4:/home/albert# perf script > out.perf # For perf_events: root@600g4:/home/albert# FlameGraph/stackcollapse-perf.pl out.perf > out.folded # 3. 生成火焰图 Use flamegraph.pl to render a SVG. FlameGraph/flamegraph.pl out.folded > dsm.svg