原文地址https://www.dell.com/support/article/cn/zh/cnbsd1/sln312382/nvme-on-rhel7?lang=en
Posted on behalf of Lakshmi Narayanan Durairajan (Lakshmi_Narayanan_Du@dell.com)
What is NVMe?
NVM Express [NVMe] or Non-Volatile Memory Host Controller Interface Specification (NVMHCI), is a for accessing (SSDs) attached through the (PCIe) bus. NVM is an acronym for , as used in SSDs.
NVMe defines optimized register interface, command set and feature set for PCIe SSD’s. NVMe focuses to standardize the PCIe SSD’s and improve the performancePCIe SSD devices designed based on the NVMe specification are NVMe based PCIeSSD’sFor more details on the NVMe please refer the link .The NVMe devices used currently are NVMe 1.0c compliantIn this blog we will be looking into RHEL 7 support for the NVMe devices.Currently DELL support the NVMe devices with RHEL 7 out of box [vendor based] driverFollowing are the list of the things that we will look into:- NVMe - Features Supported
- NVMe Device : Listing the device and its Capabilities
- Checking MaxPayLoad
- NVMe Driver : List the driver information
- NVMe Device Node and Naming conventions
- Formatting with xfs and mounting the device
- Using ledmon utility to manage backplane LEDs for NVMe device
NVMe- Features Supported
NVMe driver exposes the following features
- Basic IO operations
- Hot Plug
- Boot Support [UEFI and Legacy]
The following table lists the RHEL 7 [Out of box] driver supported features for NVMe on 12G and 13 G machines
Generation | Basic IO | Hot Plug | UEFI Boot | Legacy Boot |
---|---|---|---|---|
13 G | Yes | Yes | Yes | No |
12 G | Yes | Yes | No | No |
Checking MaxPayLoad
Check the MaxPayload value by executing the following commands. It should set it to 256 bytes [Fig.3] [root@localhost home]# lspci | grep -i Samsung45:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 171X (rev 03) [root@localhost home]# lspci -vvv -s 45:00.0 Figure 3: MaxPayload set to 256 bytesNVMe Driver: List the driver information
1) Use modinfo command to list the diver details [root@localhost ~]# modinfo nvmefilename: /lib/modules/3.10.0-123.el7.x86_64/extra/nvme/nvme.koversion: 0.8-dell1.17license: GPLauthor: Samsung Electronics Corporationsrcversion: AB81DD9D63DD5DADDED9253alias: pci:v0000144Dd0000A820sv*sd*bc*sc*i*depends: vermagic: 3.10.0-123.el7.x86_64 SMP mod_unload modversionsparm: nvme_major:intparm: use_threaded_interrupts:int The below [Fig 4] shows details of the NVMe driver nvme.ko Figure 4: Modinfo listing driver informationNVMe Device Node and Naming conventions
1) cat /proc/partitions displays the device node of nvme. a) Following command run lists the nvme device as nvme0n1 and nvme1n1 [root@localhost ~]# cat /proc/partitionsmajor minor #blocks name 259 0 781412184 nvme0n18 0 1952448512 sda8 1 512000 sda18 2 1951935488 sda211 0 1048575 sr0253 0 52428800 dm-0253 1 16523264 dm-1253 2 1882980352 dm-2259 3 390711384 nvme1n1 Partition the device using the any partitioning tools (fdisk,parted) b) Executing the following command again, lists nvme device along with partitions [root@localhost ~]# cat /proc/partitionsmajor minor #blocks name 259 0 781412184 nvme0n1259 1 390705068 nvme0n1p1259 2 390706008 nvme0n1p28 0 1952448512 sda8 1 512000 sda18 2 1951935488 sda211 0 1048575 sr0253 0 52428800 dm-0253 1 16523264 dm-1253 2 1882980352 dm-2259 3 390711384 nvme1n1259 4 195354668 nvme1n1p1259 5 195354712 nvme1n1p2Naming conventions:
The below [Fig 5] explains the naming convention of the device nodes
The number immediately after the string "nvme" is the device number
Example:
nvme0n1 – Here the device number is 0
Partitions are appended after the device name with the prefix ‘p’
Example:
nvme0n1p1 – partition 1
nvme1n1p2 – partition 2
Example:
nvme0n1p1 – partition 1 of device 0
nvme0n1p2 – partition 2 of device 0
nvme1n1p1 – partition 1 of device 1
nvme1n1p2 – partition 2 of device 1
Figure 5: Device node naming conventionsFormatting with xfs and mounting the device
1) The following command formats the nvme partition 1 on device 1 to xfs [root@localhost ~]# mkfs.xfs /dev/nvme1n1p1meta-data=/dev/nvme1n1p1 isize=256 agcount=4, agsize=12209667 blks= sectsz=512 attr=2, projid32bit=1= crc=0data = bsize=4096 blocks=48838667, imaxpct=25= sunit=0 swidth=0 blksnaming =version 2 bsize=4096 ascii-ci=0 ftype=0log =internal log bsize=4096 blocks=23847, version=2= sectsz=512 sunit=0 blks, lazy-count=1realtime =none extsz=4096 blocks=0, rtextents=0 2) Mount the device to a mount point and list the same [root@localhost ~]# mount /dev/nvme1n1p1 /mnt/[root@localhost ~]# mount | grep -i nvme/dev/nvme1n1p1 on /mnt type xfs (rw,relatime,seclabel,attr2,inode64,noquota) Using ledmon utility to manage backplane LEDs for NVMe deviceLedmon and ledctl are two utilities for Linux that can be used to control LED status on drive backplanes. Normally drive backplane LEDs are controlled by a hardware RAID controller (PERC), but when using Software RAID on Linux (mdadm) for NVMe PCIE SSD, the ledmon daemon will monitor the status of the drive array and update the status of drive LEDs.For extra reading check the link Following are the steps to install and use the ledmon/ledctl utility1) Installing OpenIPMI and ledmon/ledctl utilities:Execute the following commands to install OpenIPMI and ledmon[root@localhost ~]# yum install OpenIPMI[root@localhost ~]# yum install ledmon-0.79-3.el7.x86_64.rpm 2) Use ledmod/ledctl utilities Running ledctl and ledmon concurrently, ledmon will eventually override the ledctl settingsa) Start and check the status of ipmi as shown in the [Fig.6] using the following command[root@localhost ~]# systemctl start ipmiFigure 6: IPMI start and statusa) Start the ledmod
[root@localhost ~]# ledmon
b) [Fig 7] shows LED status after executing ledmon for the working state of the device
Figure 7: LED status after ledmon run for working state of the device (green) a) The below command will blink drive LED [on the device node /dev/nvme0n1 ][root@localhost ~]# ledctl locate=/dev/nvme0n1Below command will blink both the drive LEDs [on the device node /dev/nvme0n1 and /dev/nvme1n1][root@localhost ~]# ledctl locate={ /dev/nvme0n1 /dev/nvme1n1 }And the following command will turn off the locate LED[root@localhost ~]# ledctl locate_off=/dev/nvme0n1