WinFsp Kernel Mode File Systems

Since release 2019.3 WinFsp supports development of file systems in kernel mode. Such file systems are implemented as kernel drivers that use WinFsp as the primary FSD (File System Driver) as well as the software library that they interface with to retrieve and service file system requests. This document discusses how to write such file systems.

Motivation

The primary goal of WinFsp is to enable the easy creation of user mode file systems using an easy to use API. There are however legitimate reasons for wanting to develop a file system as a kernel mode driver, but the difficulty of doing so often deters people because Windows kernel file system development is notoriously difficult and has many pitfalls.

Some of the reasons that a kernel mode file system may be preferrable to a user mode one:

A kernel driver may be able to achieve better performance than user mode processes.
A kernel driver may be able to access advanced OS features that are not easily available to user mode processes.
A kernel driver may be able to present an alternative interface than the one presented by WinFsp to user mode processes.

Since release 2019.3 WinFsp supports development of file systems as kernel mode drivers. The primary motivation for this work was to support the WinFuse project, which exposes the FUSE protocol from the Windows kernel in a way that allows FUSE file systems to interface with it, either directly or via libfuse. The support was added in such a way that it is generic enough to be used by other kernel mode file systems.

Overview

A WinFsp "volume" (file system) is typically created using the FspFsctlCreateVolume API. This is simply a wrapper around a CreateFileW call on one of the WinFsp control devices, either WinFsp.Disk or WinFsp.Net. The CreateFileW call returns a HANDLE to the newly created volume, which acts either as a "disk" or a "network" file system, depending on which control device was used to create it.

User mode file systems interact with the WinFsp FSD via DeviceIoControl/FSP_FSCTL_TRANSACT messages on the volume HANDLE. (This is usually done indirectly via the WinFsp DLL, which hides this detail behind an easy to use API.)

Since release 2019.3 (v1.5) WinFsp supports the following:

Registration and on-demand loading of a third party driver when a volume that is destined to be handled by the third party driver is created.
Forwarding of custom DeviceIoControl messages to a third party driver. This allows the third party driver to handle custom FSCTL requests directed to a WinFsp volume HANDLE.
Kernel-mode API’s (called DDI’s) that allow a third party driver to register itself with the FSD and interact with its I/O queues.

Such drivers are called within the WinFsp sources and header/library files by the name of "Fsext Providers" (File System Extension Providers). In the text below we will usually refer to them as simply "Providers".

A Provider is a kernel mode driver that uses the FSD as a frontend file system driver to interface with Windows and implement all the complex details that this entails. The Provider fetches I/O requests from the WinFsp I/O queues and may filter them, transform them into a different protocol (as is the case with WinFuse) or use them to fully implement a file system in kernel mode.

Provider Development

The WinFsp installer includes a "Kernel Developer" feature. When installed this feature adds header and library files in the opt\fsext installation subdirectory.

The primary header file used for Provider development is <winfsp/fsext.h> and can be found in opt\fsext\inc. Additionally the file <winfsp/fsctl.h> found in the inc installation subdirectory must be in the compiler’s include path.

Providers must also be linked with one of the import libraries that can be found in opt\fsext\lib. Use the winfsp-x64.lib import library when linking the 64-bit version of a Provider and the winfsp-x86.lib import library when linking the 32-bit version of a Provider.

Provider DDI’s

The <winfsp/fsext.h> header file includes definitions for the following important DDI’s:

FspFsextProviderRegister: This DDI is used by a Provider to register itself with the FSD. Typically the Provider prepares an FSP_FSEXT_PROVIDER structure and calls FspFsextProviderRegister in its DriverEntry routine. The structure’s fields are as follows:
- Version: Set to sizeof(FSP_FSEXT_PROVIDER).
- DeviceTransactCode: The DeviceIoControl code that should be forwarded to the Provider.
  - The code must be defined as follows: CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 0xC00 + ProviderSpecific, METHOD_BUFFERED, FILE_ANY_ACCESS).
  - Please note that this scheme allows for up to 1024 codes. If you want to define a new Provider code I will appreciate it if you email me at <billziss at navimatics.com> so that we can keep track of Provider codes, avoid conflicts and see when we need to extend this scheme.
- DeviceExtensionSize: The size of private data that the Provider wants for itself in the DeviceExtension of the FSD volume’s device object.
- DeviceInit: Called when a new volume is created. The DeviceObject parameter is the volume’s device object. The VolumeParams are the initial parameters passed to FspFsctlCreateVolume and can be modified by the Provider.
- DeviceFini: Called when a volume is going away. This usually happens when the volume HANDLE is closed.
- DeviceExpirationRoutine: Called once a second as part of the FSD’s expiration timer. The FSD uses this timer to expire caches, long-pending IRP’s, etc. A Provider can use this call for a similar purpose. The ExpirationTime parameter contains the current interrupt time as determined by the FSD.
- DeviceTransact: Called whenever the FSD receives a DeviceIoControl request with the DeviceTransactCode. The Irp parameter contains the relevant IRP_MJ_FILE_SYSTEM_CONTROL.
- DeviceExtensionOffset: Set to 0 on input. On successful return from FspFsextProviderRegister it will contain the offset to use for accessing the Provider’s private data in the DeviceExtension of the FSD volume’s device object. Given a DeviceObject, the data can be accessed as (PVOID)((PUINT8)DeviceObject→DeviceExtension + Provider.DeviceExtensionOffset).
FspFsextProviderTransact: This DDI is used by a Provider to interact with the FSD I/O queues. The DeviceObject is the FSD volume’s device object. The FileObject is the FILE_OBJECT that corresponds to the volume HANDLE (created by FspFsctlCreateVolume). The Response is the response to send and can be NULL. The Request is a pointer that receives a pointer to a new request from the WinFsp I/O queue and can be NULL; if the received pointer is not NULL it must be freed with ExFreePool.

Provider I/O

When the FSD receives a file system IRP it is often able to handle and complete it without help from an external user mode or kernel mode file system. However in most cases the IRP has to be seen and acted upon by an external file system. For this reason the FSD preprocesses the IRP and places it in an I/O queue in the form of a "request". At a later time the external file system retrieves the request, processes it and sends back a "response". The FSD uses the response to locate the original IRP, perform any necessary postprocessing and finally complete the IRP.

Providers typically use the FspFsextProviderTransact DDI to receive requests and send back responses. Requests are of type FSP_FSCTL_TRANSACT_REQ and responses are of type FSP_FSCTL_TRANSACT_RSP. Requests have a Kind field which describes what kind of file system operation is being requested. The following request kinds are currently defined in <winfsp/fsctl.h>:

    FspFsctlTransactCreateKind,
    FspFsctlTransactOverwriteKind,
    FspFsctlTransactCleanupKind,
    FspFsctlTransactCloseKind,
    FspFsctlTransactReadKind,
    FspFsctlTransactWriteKind,
    FspFsctlTransactQueryInformationKind,
    FspFsctlTransactSetInformationKind,
    FspFsctlTransactQueryEaKind,
    FspFsctlTransactSetEaKind,
    FspFsctlTransactFlushBuffersKind,
    FspFsctlTransactQueryVolumeInformationKind,
    FspFsctlTransactSetVolumeInformationKind,
    FspFsctlTransactQueryDirectoryKind,
    FspFsctlTransactFileSystemControlKind,
    FspFsctlTransactDeviceControlKind,
    FspFsctlTransactShutdownKind,
    FspFsctlTransactLockControlKind,
    FspFsctlTransactQuerySecurityKind,
    FspFsctlTransactSetSecurityKind,
    FspFsctlTransactQueryStreamInformationKind,

When request processing is complete the Provider must prepare a response and send it to the FSD using FspFsextProviderTransact as mentioned above. It is particularly important that the Provider initializes the Kind and Hint fields by copying the values from the corresponding request.

This document does not describe in detail how each request kind is supposed to be handled. For the full details refer to the implementation for the WinFsp DLL in the WinFsp sources: src/dll/fsop.c. Although this implementation is for user mode file systems, similar logic and techniques should be used for Providers.

Provider Registration

Providers are loaded on demand and must be properly registered:

A provider must be registered as a kernel driver. This can be achieved by using the command sc create PROVIDER type=kernel binPath=X:\PATH\TO\PROVIDER.SYS or by using the Service Control Manager API’s (OpenServiceW, CreateServiceW, etc.). You do not need an INF file or to use the Setup API in order to register a Provider driver.
A provider must be registered under the registry key HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\WinFsp\Fsext. Create a string value with name the textual representation of the Provider’s transact code (see DeviceTransactCode) in "%08lx" format and value the Provider’s driver name.

For example the WinFuse Provider registers its driver under the name WinFuse and adds a registry value of 00093118 → WinFuse.

Provider Lifetime

Providers are loaded on demand by the FSD during volume creation. This process works as follows:

During volume creation (e.g. by using FspFsctlCreateVolume) a non-zero FsextControlCode must be specified in VolumeParams.
If the FSD sees the FsextControlCode as non-zero it attempts to find a corresponding Provider driver.
- It first checks an internal mapping of codes to Provider drivers. If the code is found, the FSD proceeds to the DeviceInit step below.
- If the code is not found in the internal mapping, the FSD checks the registry under the registry key HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\WinFsp\Fsext. If the code is not found the volume creation fails.
- If the code is found the FSD loads the Provider driver using ZwLoadDriver. The Provider is supposed to register itself with the FSD during DriverEntry by calling FspFsextProviderRegister.
- Finally the internal mapping of codes to Providers is rechecked. Assuming that everything worked as intended, the corresponding Provider driver is now loaded and we can proceed to the DeviceInit step.
The FSD proceeds to call the DeviceInit callback of the Provider. The Provider can use this call to initialize itself in relation to the new volume device object.
Assuming that the volume device object is created successfully, the FSD will do the following:
- Forward any FsextControlCode==DeviceTransactCode requests that it gets in its IRP_MJ_FILE_SYSTEM_CONTROL to the Provider via DeviceTransact.
- Call the Provider’s DeviceExpirationRoutine once a second as part of the FSD’s expiration process.
Eventually the volume device object will be torn down (e.g. because the corresponding HANDLE is closed). In this case the FSD will call the Provider’s DeviceFini callback.

Finally note that once loaded a Provider driver cannot be unloaded (without a reboot).