Idea ID: 1638649

Backup a single file system with multiple streams

Status: Accepted

Brief description:

While file systems continue to grow and the underlaying disk subsystems become faster the backups of large file systems are still one of the major concerns for traditional backup environments. The Data Protector Disk Agent should support multiple streams on a single file system that can be configured similar to device concurrency to drastically speed up the backup and the restore operation.

Picture1.png

Existing Enhancement Requests:

QCCR2A62413: Support multiple disk agents per volume

Benefit:

Reduce the complexitiy of subdividing file systems manually into smaller pieces. Optimize for backup and restore performance using multiple streams even on large file systems with millions of files and folders.

Tags:

Parents
  • With today's multi-TB filesystems, this feature is needed more than ever.  I wrote the original "divide-and-conquer" article back in 2011:
    https://community.hpe.com/t5/Transforming-IT/DPTIPS-Large-Objects-Divide-and-Conquer-with-Data-Protector/ba-p/6805642

    After the first tree walk, a disk agent (VBDA) should have all the info needed to intelligently analyze the filesystem and execute a dynamic divide-and-conquer strategy to read files/folders in an optimal, parallel fashion.

    The potential downside is that such VBDA stream multiplexing would adversely affect inline dedupe which relies upon a single, non-multiplexed data source per media agent (BMA) stream headed to a Catalyst store.  I suppose assigning a unique BMA per VBDA "substream" would be a viable workaround.

    Another consideration is not going crazy with the # of VBDA threads/streams.  The filesystem being read by VBDA lives on a LUN that can easily be pushed to its limit of concurrent IO.  You'll likely reach a LUN's concurrent IO limit with a half dozen streams or less.  In my experience using manual divide-and-conquer, here's what I've found with parallel streams from one filesystem:

    2 streams better than 1?  Always.
    3 streams better than 2?  Quite often.
    4 streams better than 3?  Usually but not always.
    5 streams better than 4?  Less often - depends on the underlying disk storage.
    6 streams better than 5?  Almost never because you've saturated the concurrent IO capability fo the underlying LUN.

    The point of diminishing returns is generally five or six parallel streams coming from the same filesystem - again depending upon a plethora of factors spanning storage, infrastructure, and host resource capability.

Comment
  • With today's multi-TB filesystems, this feature is needed more than ever.  I wrote the original "divide-and-conquer" article back in 2011:
    https://community.hpe.com/t5/Transforming-IT/DPTIPS-Large-Objects-Divide-and-Conquer-with-Data-Protector/ba-p/6805642

    After the first tree walk, a disk agent (VBDA) should have all the info needed to intelligently analyze the filesystem and execute a dynamic divide-and-conquer strategy to read files/folders in an optimal, parallel fashion.

    The potential downside is that such VBDA stream multiplexing would adversely affect inline dedupe which relies upon a single, non-multiplexed data source per media agent (BMA) stream headed to a Catalyst store.  I suppose assigning a unique BMA per VBDA "substream" would be a viable workaround.

    Another consideration is not going crazy with the # of VBDA threads/streams.  The filesystem being read by VBDA lives on a LUN that can easily be pushed to its limit of concurrent IO.  You'll likely reach a LUN's concurrent IO limit with a half dozen streams or less.  In my experience using manual divide-and-conquer, here's what I've found with parallel streams from one filesystem:

    2 streams better than 1?  Always.
    3 streams better than 2?  Quite often.
    4 streams better than 3?  Usually but not always.
    5 streams better than 4?  Less often - depends on the underlying disk storage.
    6 streams better than 5?  Almost never because you've saturated the concurrent IO capability fo the underlying LUN.

    The point of diminishing returns is generally five or six parallel streams coming from the same filesystem - again depending upon a plethora of factors spanning storage, infrastructure, and host resource capability.

Children
No Data