{"id":11581,"date":"2024-04-08T21:13:00","date_gmt":"2024-04-08T19:13:00","guid":{"rendered":"https:\/\/monodes.com\/predaelli\/?p=11581"},"modified":"2024-04-08T17:54:23","modified_gmt":"2024-04-08T15:54:23","slug":"5-tips-to-speed-up-linux-software-raid-rebuilding-and-re-syncing-nixcraft","status":"publish","type":"post","link":"https:\/\/monodes.com\/predaelli\/2024\/04\/08\/5-tips-to-speed-up-linux-software-raid-rebuilding-and-re-syncing-nixcraft\/","title":{"rendered":"5 Tips To Speed Up Linux Software Raid Rebuilding And Re-syncing – nixCraft"},"content":{"rendered":"\n
\n

Speed Up Linux Software Raid: Various commands tips to increase the speed of Linux Software RAID 0\/1\/5\/6\/10 reconstruction and rebuild time<\/p>\n<\/blockquote>\n\n\n\n

Source: 5 Tips To Speed Up Linux Software Raid Rebuilding And Re-syncing – nixCraft<\/a><\/em><\/p>\n\n\n\n\n\n\n\n\n\n\n\n

\n

5 Tips To Speed Up Linux Software Raid Rebuilding And Re-syncing<\/a><\/h1>\n\n\n\n

Author:Vivek GiteLast updated:April 7, 202467 comments<\/a><\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

It is no secret that I am a pretty big fan of excellent Linux Software RAID<\/a>. Creating, assembling and rebuilding small array is fine. But, things started to get nasty when you try to rebuild or re-sync large size array. You may get frustrated when you see it is going to take 22 hours to rebuild the array. You can always increase the speed of Linux Software RAID 0\/1\/5\/6 reconstruction using the following five tips.
<\/p>\n\n\n\n

Why speed up Linux software RAID rebuilding and re-syncing?<\/h2>\n\n\n\n

Recently, I build a small NAS server running Linux for one my client with 5 x 2TB disks in RAID 6 configuration for all in one backup server for Linux, Mac OS X, and Windows XP\/Vista\/7\/10 client computers. Next, I type the command cat \/proc\/mdstat<\/kbd> and it reported that md0 is active and recovery is in progress. The recovery speed was around 4000K\/sec and will complete in approximately in 22 hours. I wanted to finish this early.<\/p>\n\n\n\n

A note about lazy initialization and ext4 file system<\/h2>\n\n\n\n

When creating an ext4 file system, the Linux kernel uses lazy initialization. This feature allows the faster creatation of a file system. A process called \u201cext4lazyinit\u201d runs in the background to create rest of all inode tables. As a result, your RAID rebuild is going to operate at minimal speed. This only affects if you have just created an ext4 filesystem. There is an option to enable or disable this feature while running mkfs.ext4 command:<\/p>\n\n\n\n

    \n
  1. lazy_itable_init[= <0 to disable, 1 to enable>]<\/kbd> \u2013 If enabled and the uninit_bg feature is enabled, the inode table will not be fully initialized by mke2fs. This speeds up filesystem initialization noticeably, but it requires the kernel to finish initializing the filesystem in the background when the filesystem is first mounted. If the option value is omitted, it defaults to 1 to enable lazy inode table zeroing.<\/li>\n\n\n\n
  2. lazy_journal_init[= <0 to disable, 1 to enable>]<\/kbd> \u2013 If enabled, the journal inode will not be fully zeroed out by mke2fs. This speeds up filesystem initialization noticeably, but carries some small risk if the system crashes before the journal has been overwritten entirely one time. If the option value is omitted, it defaults to 1 to enable lazy journal inode zeroing.<\/li>\n<\/ol>\n\n\n\n

    Tip #1: \/proc\/sys\/dev\/raid\/{speed_limit_max,speed_limit_min} kernel variables<\/h2>\n\n\n\n

    The \/proc\/sys\/dev\/raid\/speed_limit_min<\/kbd> is config file that reflects the current \u201cgoal\u201d rebuild speed for times when non-rebuild activity is current on an array. The speed is in Kibibytes per second (1 kibibyte = 210<\/sup> bytes = 1024 bytes), and is a per-device rate, not a per-array rate . The default is 1000.<\/p>\n\n\n\n

    The \/proc\/sys\/dev\/raid\/speed_limit_max<\/kbd> is config file that reflects the current \u201cgoal\u201d rebuild speed for times when no non-rebuild activity is current on an array. The default is 100,000.<\/p>\n\n\n\n

    To see current limits, enter:
    sysctl dev.raid.speed_limit_min<br><\/code><\/p>\n\n\n\n

    sysctl dev.raid.speed_limit_max<\/code>
    Sample outputs:<\/p>\n\n\n\n

    dev.raid.speed_limit_min = 10000\ndev.raid.speed_limit_max = 20000<\/pre>\n\n\n\n
    NOTE<\/strong>: The following hacks are used for recovering Linux software raid, and to increase the speed of RAID rebuilds. Options are good for tweaking rebuilt process and may increase overall system load, high cpu and memory usage.<\/p>\n\n\n\n
    To increase speed, enter:
    echo value > \/proc\/sys\/dev\/raid\/speed_limit_min<\/code>
    OR
    sysctl -w dev.raid.speed_limit_min=value<\/code>
    In this example, set it to 50000 K\/Sec, enter:
    echo 50000 > \/proc\/sys\/dev\/raid\/speed_limit_min<\/code>
    OR<\/p>\n\n\n\n
    sysctl -w dev.raid.speed_limit_min=50000<\/code>
    If you want to override the defaults you could add these two lines to     \/etc\/sysctl.conf<\/a>:<\/p>\n\n\n\n
    #################NOTE ################\n##  You are limited by CPU and memory too #\n###########################################\ndev.raid.speed_limit_min = 50000\n## good for 4-5 disks based array ##\ndev.raid.speed_limit_max = 2000000\n## good for large 6-12 disks based array ###\ndev.raid.speed_limit_max = 5000000<\/pre>\n\n\n\n
    Tip #2: Set read-ahead option<\/h2>\n\n\n\n
    Set readahead (in 512-byte sectors) per raid device. The syntax is:
    blockdev --setra 65536 \/dev\/mdX<br>## Set read-ahead to 32 MiB ##<br><\/code>blockdev --setra 65536 \/dev\/md0<br><\/code><\/p>\n\n\n\n
    blockdev --setra 65536 \/dev\/md1<\/code><\/p>\n\n\n\n
    Tip #3: Set stripe-cache_size for RAID5 or RAID 6<\/h2>\n\n\n\n
    This is only available on RAID5 and RAID6<\/strong> and boost sync performance by 3-6 times. It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256. Valid values are 17 to 32768. Increasing this number can increase performance in some situations, at some cost in system memory. Note, setting this value too high can result in an \u201cout of memory\u201d condition for the system. Use the following formula:<\/p>\n\n\n\n
    memory_consumed = system_page_size * nr_disks * stripe_cache_size\n<\/pre>\n\n\n\n
    To set stripe_cache_size to 16 MiB for \/dev\/md0, type:
    echo 16384 > \/sys\/block\/md0\/md\/stripe_cache_size<\/code>
    To set stripe_cache_size to 32 MiB for \/dev\/md3, type:<\/p>\n\n\n\n
    echo 32768 > \/sys\/block\/md3\/md\/stripe_cache_size<\/code><\/p>\n\n\n\n
    Tip #4: Disable NCQ on all disks<\/h2>\n\n\n\n
    The following will disable NCQ on \/dev\/sda,\/dev\/sdb,..,\/dev\/sde using bash for loop<\/a><\/p>\n\n\n\n
    ## sample for loop ##\nfor i in sd[abcde]\ndo\n  echo 1 > \/sys\/block\/$i\/device\/queue_depth\ndone<\/pre>\n\n\n\n
    Tip #5: Bitmap Option<\/h2>\n\n\n\n
    Bitmaps optimize rebuild time after a crash, or after removing and re-adding a device. Turn it on by typing the following command:
    mdadm --grow --bitmap=internal \/dev\/md0<\/code>
    Once array rebuild or fully synced, disable bitmaps:<\/p>\n\n\n\n
    mdadm --grow --bitmap=none \/dev\/md0<\/code><\/p>\n\n\n\n
    Results<\/h2>\n\n\n\n
    My speed went from 4k to 51k:
    cat \/proc\/mdstat<\/code>
    Sample outputs:<\/p>\n\n\n\n
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]\nmd5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]\n      530048 blocks [2\/2] [UU]\n\nmd0 : active raid6 sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]\n      5855836800 blocks level 6, 64k chunk, algorithm 2 [5\/5] [UUUUU]\n      [============>........]  resync = 61.7% (1205475036\/1951945600) finish=242.9min speed=51204K\/sec\n<\/pre>\n\n\n\n
    Monitoring raid rebuilding\/recovery process like a pro<\/h2>\n\n\n\n
    You cat \/proc\/mdstat file. This read-only file contains information about the status of currently running array and shows rebuilding speed:
    cat \/proc\/mdstat<\/code>
    Alternatively use the     watch command to display \/proc\/mdstat output on screen repeatedly<\/a>, type:<\/p>\n\n\n\n
    watch -n1 <a href="https:\/\/www.cyberciti.biz\/faq\/linux-unix-appleosx-bsd-cat-command-examples\/">cat<\/a> \/proc\/mdstat<\/code>
    Sample outputs:
    <\/p>\n\n\n\n
    <\/a><\/p>\n\n\n\n
    Fig.01: Performance optimization for Linux raid6 for \/dev\/md2 The following command provide details about \/dev\/md2 raid arrray including status and health report:
    mdadm --detail \/dev\/md2<\/code>
    Sample outputs:<\/p>\n\n\n\n
    <\/a><\/p>\n\n\n\n
    Fig.02: Finding out information about md2 raid array You can filter out info using the grep command<\/a> or egrep command<\/a> as follows:
    mdadm --detail \/dev\/md3 | grep 'info-you-want'<\/code>
    Another option is to see     what is actually happening by typing the following iostat command<\/a> to see disk utilization:
    watch iostat -k 1 2<br><\/code>watch -n1 iostat -k 1 2<\/code>
    Sample outputs:<\/p>\n\n\n\n
    <\/a><\/p>\n\n\n\n
    Fig.03: Find out CPU statistics and input\/output statistics for devices and partitions Feel free to use the df command<\/a> or du command to get info about the disk space usage on Linux.<\/a> For example:
    df -hT \/raid1<br><\/code>du -csh \/raid1<\/code><\/p>\n\n\n\n
    Conclusion<\/h2>\n\n\n\n
    We learned how to optimize speed for Linux software RAID devices.<\/p>\n\n\n\n