VFS iomap Design and Porting

Definitions

• bufferhead: Shattered remnants of the old buffer cache.

• fsblock: The block size of a file, also known as i_blocksize.

• i_rwsem: The VFS struct inode rwsemaphore.

• invalidate_lock: The pagecache struct address_space rwsemaphore that protects against folio removal.

struct iomap_ops

Every iomap function requires the filesystem to pass an operations structure to obtain a mapping and (optionally) to release the mapping.

struct iomap_ops {
    int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,
                       unsigned flags, struct iomap *iomap,
                       struct iomap *srcmap);

    int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,
                     ssize_t written, unsigned flags,
                     struct iomap *iomap);
};

The ->iomap_begin function is called to obtain one mapping for the range of bytes specified by pos and length for the file inode.

Each iomap operation describes the requested operation through the flags argument. The exact value of flags will be documented in the operation-specific sections below, but these principles apply generally:

• For a write operation, IOMAP_WRITE will be set. Filesystems must not return IOMAP_HOLE mappings.

• For any other operation, IOMAP_WRITE will not be set.

• For any operation targetting direct access to storage (fsdax), IOMAP_DAX will be set.

If it is necessary to read existing file contents from a different device or address range on a device, the filesystem should return that information via srcmap. Only pagecache and fsdax operations support reading from one mapping and writing to another.

After the operation completes, the ->iomap_end function, if present, is called to signal that iomap is finished with a mapping. Typically, implementations will use this function to tear down any context that were set up in ->iomap_begin. For example, a write might wish to commit the reservations for the bytes that were operated upon and unreserve any space that was not operated upon. written might be zero if no bytes were touched. flags will contain the same value passed to ->iomap_begin. iomap ops for reads are not likely to need to supply this function.

Both functions should return a negative errno code on error, or zero.

struct iomap

The filesystem returns the mappings via the following structure. For documentation purposes, the structure has been reordered to group fields that go together logically.

struct iomap {
    loff_t                       offset;
    u64                          length;

    u16                          type;
    u16                          flags;

    u64                          addr;
    struct block_device          *bdev;
    struct dax_device            *dax_dev;
    void                         *inline_data;

    void                         *private;

    const struct iomap_folio_ops *folio_ops;

    u64                          validity_cookie;
};

The information is useful for translating file operations into action. The actions taken are specific to the target of the operation, such as disk cache, physical storage devices, or another part of the kernel.

• offset and length describe the range of file offsets, in bytes, covered by this mapping. These fields must always be set by the filesystem.

• type describes the type of the space mapping:

  • IOMAP_HOLE: No storage has been allocated. This type must never be returned in response to an IOMAP_WRITE operation because writes must allocate and map space, and return the mapping. The addr field must be set to IOMAP_NULL_ADDR. iomap does not support writing (whether via pagecache or direct I/O) to a hole.

  • IOMAP_DELALLOC: A promise to allocate space at a later time ("delayed allocation"). If the filesystem returns IOMAP_F_NEW here and the write fails, the ->iomap_end function must delete the reservation. The addr field must be set to IOMAP_NULL_ADDR.

  • IOMAP_MAPPED: The file range maps to specific space on the storage device. The device is returned in bdev or dax_dev. The device address, in bytes, is returned via addr.

  • IOMAP_UNWRITTEN: The file range maps to specific space on the storage device, but the space has not yet been initialized. The device is returned in bdev or dax_dev. The device address, in bytes, is returned via addr. Reads will return zeroes to userspace. For a write or writeback operation, the ioend should update the mapping to MAPPED.

  • IOMAP_INLINE: The file range maps to the memory buffer specified by inline_data. For write operation, the ->iomap_end function presumably handles persisting the data. The addr field must be set to IOMAP_NULL_ADDR.

• flags describe the status of the space mapping. These flags should be set by the filesystem in ->iomap_begin:

  • IOMAP_F_NEW: The space under the mapping is newly allocated. Areas that will not be written to must be zeroed. If a write fails and the mapping is a space reservation, the reservation must be deleted.

  • IOMAP_F_DIRTY: The inode will have uncommitted metadata needed to access any data written. fdatasync is required to commit these changes to persistent storage. This needs to take into account metadata changes that may be made at I/O completion, such as file size updates from direct I/O.

  • IOMAP_F_SHARED: The space under the mapping is shared. Copy on write is necessary to avoid corrupting other file data.

  • IOMAP_F_BUFFER_HEAD: This mapping requires the use of buffer heads for pagecache operations. Do not add more uses of this.

  • IOMAP_F_MERGED: Multiple contiguous block mappings were coalesced into this single mapping. This is only useful for FIEMAP.

  • IOMAP_F_XATTR: The mapping is for extended attribute data, not regular file data. This is only useful for FIEMAP.

  • IOMAP_F_PRIVATE: Starting with this value, the upper bits can be set by the filesystem for its own purposes.

  These flags can be set by iomap itself during file operations. The filesystem should supply an ->iomap_end function to observe these flags:

  • IOMAP_F_SIZE_CHANGED: The file size has changed as a result of using this mapping.

  • IOMAP_F_STALE: The mapping was found to be stale. iomap will call ->iomap_end on this mapping and then ->iomap_begin to obtain a new mapping.

  Currently, these flags are only set by pagecache operations.

• addr describes the device address, in bytes.

• bdev describes the block device for this mapping. This only needs to be set for mapped or unwritten operations.

• dax_dev describes the DAX device for this mapping. This only needs to be set for mapped or unwritten operations, and only for a fsdax operation.

• inline_data points to a memory buffer for I/O involving IOMAP_INLINE mappings. This value is ignored for all other mapping types.

• private is a pointer to filesystem-private information. This value will be passed unchanged to ->iomap_end.

• folio_ops will be covered in the section on pagecache operations.

• validity_cookie is a magic freshness value set by the filesystem that should be used to detect stale mappings. For pagecache operations this is critical for correct operation because page faults can occur, which implies that filesystem locks should not be held between ->iomap_begin and ->iomap_end. Filesystems with completely static mappings need not set this value. Only pagecache operations revalidate mappings.

  XXX: Should fsdax revalidate as well?

Buffered I/O

Buffered I/O is the default file I/O path in Linux. File contents are cached in memory ("pagecache") to satisfy reads and writes. Dirty cache will be written back to disk at some point that can be forced via fsync and variants.

iomap implements nearly all the folio and pagecache management that filesystems once had to implement themselves. This means that the filesystem need not know the details of allocating, mapping, managing uptodate and dirty state, or writeback of pagecache folios. Unless the filesystem explicitly opts in to buffer heads, they will not be used, which makes buffered I/O much more efficient, and willy much happier.

struct iomap_folio_ops {
    struct folio *(*get_folio)(struct iomap_iter *iter, loff_t pos,
                               unsigned len);
    void (*put_folio)(struct inode *inode, loff_t pos, unsigned copied,
                      struct folio *folio);
    bool (*iomap_valid)(struct inode *inode, const struct iomap *iomap);
};

struct iomap_writeback_ops {
    int (*map_blocks)(struct iomap_writepage_ctx *wpc, struct inode *inode,
                      loff_t offset, unsigned len);
    int (*prepare_ioend)(struct iomap_ioend *ioend, int status);
    void (*discard_folio)(struct folio *folio, loff_t pos);
};

Direct I/O

In Linux, direct I/O is defined as file I/O that is issued directly to storage, bypassing the pagecache.

The iomap_dio_rw function implements O_DIRECT (direct I/O) reads and writes for files. An optional ops parameter can be passed to change the behavior of direct I/O. The done_before parameter should be set if writes have been initiated prior to the call. The direction of the I/O is determined from the iocb passed in.

The flags argument can be any of the following values:

• IOMAP_DIO_FORCE_WAIT: Wait for the I/O to complete even if the kiocb is not synchronous.

• IOMAP_DIO_OVERWRITE_ONLY: Allocating blocks, zeroing partial blocks, and extensions of the file size are not allowed. The entire file range must to map to a single written or unwritten extent. This flag exists to enable issuing concurrent direct IOs with only the shared i_rwsem held when the file I/O range is not aligned to the filesystem block size. -EAGAIN will be returned if the operation cannot proceed.

• IOMAP_DIO_PARTIAL: If a page fault occurs, return whatever progress has already been made. The caller may deal with the page fault and retry the operation.

These struct kiocb flags are significant for direct I/O with iomap:

• IOCB_NOWAIT: Only proceed with the I/O if mapping data are already in memory, we do not have to initiate other I/O, and we acquire all filesystem locks without blocking.

• IOCB_SYNC: Ensure that the device has persisted data to disk before completing the call. In the case of pure overwrites, the I/O may be issued with FUA enabled.

• IOCB_HIPRI: Poll for I/O completion instead of waiting for an interrupt. Only meaningful for asynchronous I/O, and only if the entire I/O can be issued as a single struct bio.

• IOCB_DIO_CALLER_COMP: Try to run I/O completion from the caller's process context. See linux/fs.h for more details.

Filesystems should call iomap_dio_rw from ->read_iter and ->write_iter, and set FMODE_CAN_ODIRECT in the ->open function for the file. They should not set ->direct_IO, which is deprecated.

If a filesystem wishes to perform its own work before direct I/O completion, it should call __iomap_dio_rw. If its return value is not an error pointer or a NULL pointer, the filesystem should pass the return value to iomap_dio_complete after finishing its internal work.

struct iomap_dio_ops {
    void (*submit_io)(const struct iomap_iter *iter, struct bio *bio,
                      loff_t file_offset);
    int (*end_io)(struct kiocb *iocb, ssize_t size, int error,
                  unsigned flags);
    struct bio_set *bio_set;
};

DAX I/O

Storage devices that can be directly mapped as memory support a new access mode known as "fsdax".

SEEK_DATA

The iomap_seek_data function implements the SEEK_DATA "whence" value for llseek. IOMAP_REPORT will be passed as the flags argument to ->iomap_begin.

For unwritten mappings, the pagecache will be searched. Regions of the pagecache with a folio mapped and uptodate fsblocks within those folios will be reported as data areas.

Callers commonly hold i_rwsem in shared mode.

SEEK_HOLE

The iomap_seek_hole function implements the SEEK_HOLE "whence" value for llseek. IOMAP_REPORT will be passed as the flags argument to ->iomap_begin.

For unwritten mappings, the pagecache will be searched. Regions of the pagecache with no folio mapped, or a !uptodate fsblock within a folio will be reported as sparse hole areas.

Callers commonly hold i_rwsem in shared mode.

Swap File Activation

The iomap_swapfile_activate function finds all the base-page aligned regions in a file and sets them up as swap space. The file will be fsync()'d before activation. IOMAP_REPORT will be passed as the flags argument to ->iomap_begin. All mappings must be mapped or unwritten; cannot be dirty or shared, and cannot span multiple block devices. Callers must hold i_rwsem in exclusive mode; this is already provided by swapon.

Extent Map Reporting (FS_IOC_FIEMAP)

The iomap_fiemap function exports file extent mappings to userspace in the format specified by the FS_IOC_FIEMAP ioctl. IOMAP_REPORT will be passed as the flags argument to ->iomap_begin. Callers commonly hold i_rwsem in shared mode.

Block Map Reporting (FIBMAP)

iomap_bmap implements FIBMAP. The calling conventions are the same as for FIEMAP. This function is only provided to maintain compatibility for filesystems that implemented FIBMAP prior to conversion. This ioctl is deprecated; do not add a FIBMAP implementation to filesystems that do not have it. Callers should probably hold i_rwsem in shared mode, but this is unclear.

Why Convert to iomap?

There are several reasons to convert a filesystem to iomap:

1. The classic Linux I/O path is not terribly efficient. Pagecache operations lock a single base page at a time and then call into the filesystem to return a mapping for only that page. Direct I/O operations build I/O requests a single file block at a time. This worked well enough for direct/indirect-mapped filesystems such as ext2, but is very inefficient for extent-based filesystems such as XFS.

2. Large folios are only supported via iomap; there are no plans to convert the old buffer_head path to use them.

3. Direct access to storage on memory-like devices (fsdax) is only supported via iomap.

4. Lower maintenance overhead for individual filesystem maintainers. iomap handles common pagecache related operations itself, such as allocating, instantiating, locking, and unlocking of folios. No ->write_begin(), ->write_end() or direct_IO address_space_operations are required to be implemented by filesystem using iomap.

How to Convert to iomap?

First, add #include <linux/iomap.h> from your source code and add select FS_IOMAP to your filesystem's Kconfig option. Build the kernel, run fstests with the -g all option across a wide variety of your filesystem's supported configurations to build a baseline of which tests pass and which ones fail.

The recommended approach is first to implement ->iomap_begin (and ->iomap->end if necessary) to allow iomap to obtain a read-only mapping of a file range. In most cases, this is a relatively trivial conversion of the existing get_block() function for read-only mappings. FS_IOC_FIEMAP is a good first target because it is trivial to implement support for it and then to determine that the extent map iteration is correct from userspace. If FIEMAP is returning the correct information, it's a good sign that other read-only mapping operations will do the right thing.

Next, modify the filesystem's get_block(create = false) implementation to use the new ->iomap_begin implementation to map file space for selected read operations. Hide behind a debugging knob the ability to switch on the iomap mapping functions for selected call paths. It is necessary to write some code to fill out the bufferhead-based mapping information from the iomap structure, but the new functions can be tested without needing to implement any iomap APIs.

Once the read-only functions are working like this, convert each high level file operation one by one to use iomap native APIs instead of going through get_block(). Done one at a time, regressions should be self evident. You do have a regression test baseline for fstests, right? It is suggested to convert swap file activation, SEEK_DATA, and SEEK_HOLE before tackling the I/O paths. A likely complexity at this point will be converting the buffered read I/O path because of bufferheads. The buffered read I/O paths doesn't need to be converted yet, though the direct I/O read path should be converted in this phase.

At this point, you should look over your ->iomap_begin function. If it switches between large blocks of code based on dispatching of the flags argument, you should consider breaking it up into per-operation iomap ops with smaller, more cohesive functions. XFS is a good example of this.

The next thing to do is implement get_blocks(create == true) functionality in the ->iomap_begin/->iomap_end methods. It is strongly recommended to create separate mapping functions and iomap ops for write operations. Then convert the direct I/O write path to iomap, and start running fsx w/ DIO enabled in earnest on filesystem. This will flush out lots of data integrity corner case bugs that the new write mapping implementation introduces.

Now, convert any remaining file operations to call the iomap functions. This will get the entire filesystem using the new mapping functions, and they should largely be debugged and working correctly after this step.

Most likely at this point, the buffered read and write paths will still to be converted. The mapping functions should all work correctly, so all that needs to be done is rewriting all the code that interfaces with bufferheads to interface with iomap and folios. It is much easier first to get regular file I/O (without any fancy features like fscrypt, fsverity, compression, or data=journaling) converted to use iomap. Some of those fancy features (fscrypt and compression) aren't implemented yet in iomap. For unjournalled filesystems that use the pagecache for symbolic links and directories, you might also try converting their handling to iomap.

The rest is left as an exercise for the reader, as it will be different for every filesystem. If you encounter problems, email the people and lists in get_maintainers.pl for help.