FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148

tmds · 2025-06-30T14:42:38Z

(By default) Linux allows 128 inotify instances per user. By sharing the inotify instance between the FileSystemWatchers we reduce contention with other applications.

Fixes #62869.

…atch tracking.

tmds · 2025-06-30T14:43:58Z

@dotnet/area-system-io @stephentoub ptal.

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

…-Windows configurations.

… or we won't update _wdToWatch.

adamsitnik

@tmds big thanks for your contribution! For now I've reviewed 25% of this PR (I need to wrap my head around all the locks and it's going to take me a while), PTAL at my comments.

src/libraries/Common/src/Interop/Linux/System.Native/Interop.INotify.cs

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

tmds · 2025-07-02T18:05:09Z

For the locking, this overview may be helpful:

watcherslock: guards the list of Watcher
lock on Watcher: guards its tree of WatchedDirectory
lock on Watch: guards its list of WatchedDirectory
addLock: this reader writer lock enables concurrent adding of watches ("readers"), but no watches may be added while an inotify watch is removed ("writer")

To prevent deadlocks, the locks (as needed) are taken in this order: watchersLock, addLock, lock on Watcher, lock on Watch.

…rectly.

tmds · 2025-07-07T07:42:04Z

@adamsitnik this is a challenge to review so feel free to ask any questions you have while looking at the code.

I'm also going to make some time this week to look at the PR with my reviewer's hat on.

…taken.

tmds · 2025-07-10T07:28:14Z

@adamsitnik I'll be on an extended break starting next week. I wonder if you have any additional feedback/questions that I can still look into tomorrow.

If we'd like to address #62869, I think this is the way to go.
There's a cost in code complexity as we can no longer consider each inotify watch to be owned by a single FileSystemWatcher.

I don't think we need to rush this in. It would be good to have some target date in mind so this doesn't get postponed indefinitely.

tmds · 2025-08-18T08:25:01Z

@adamsitnik @stephentoub where do you want to go with this? Do you want to target .NET 10? Or perhaps defer to early .NET 11?

jeffhandley · 2025-09-02T02:44:53Z

We will target .NET 11 for this, @tmds. Thanks for your patience. It might take some more time for @adamsitnik to get back to reviewing it. I'm also adding @jozkee as a reviewer to help and load-balance with Adam (I'll let them coordinate if/how to divide the reviewing).

tmds · 2025-11-05T12:00:19Z

@adamsitnik @jozkee can we target .NET 11 preview 1 for this PR?

tmds · 2025-11-18T08:53:10Z

@jeffhandley @adamsitnik @jozkee it would be nice if we can work towards getting this merged.

Copilot

Pull request overview

This PR refactors the Linux implementation of FileSystemWatcher to use a single shared inotify instance across all FileSystemWatcher objects, addressing the low per-user limit of 128 inotify instances on Linux. The refactoring introduces a new architecture with improved watch management and event processing.

Key changes:

Introduces a shared inotify instance pattern to reduce contention with other applications
Implements a dedicated thread for reading inotify events with ThreadPool-based event handler invocation to prevent blocking
Adds new internal classes (INotify, Watcher, WatchedDirectory, Watch) to track watch state and manage the relationship between paths and watch descriptors

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 7 comments.

File	Description
System/IO/FileSystemWatcher.cs	Extracts CreateBufferOverflowException method for reuse in Linux implementation
System/IO/FileSystemWatcher.Linux.cs	Major refactoring: implements shared inotify instance, new watch tracking structures, improved event processing with Channel-based queuing
System.IO.FileSystem.Watcher.csproj	Adds required dependencies for System.Collections.Concurrent and System.Threading.Channels on non-Windows platforms
Interop.INotify.cs	Adds IN_MASK_ADD flag support needed for sharing inotify instances across multiple watchers

Copilot · 2025-12-10T22:23:17Z

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

+                public NotifyFilters NotifyFilters { get; }
+                public Interop.Sys.NotifyEvents WatchFilters { get; }
+                public bool IncludeSubdirectories { get; }
+                public bool IsStopped { get; set; }


The IsStopped property is accessed from multiple threads without synchronization (e.g., set on line 1073 within a lock, but read on line 231 without holding the same lock). This should be marked as volatile or use Interlocked operations to ensure proper memory visibility across threads.

Suggested change

public bool IsStopped { get; set; }

private volatile bool _isStopped;

public bool IsStopped

{

get => _isStopped;

set => _isStopped = value;

}

Synchronization happens with other locks, like the one on the watcher instance, and IsStopped being set before signalling the writer completion.

Copilot · 2025-12-10T22:23:18Z

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

+                private readonly WeakReference<FileSystemWatcher> _weakFsw;
+                private readonly Channel<WatcherEvent> _eventQueue;
+                private INotify? _inotify;
+                private bool _emitEvents;


The _emitEvents field is accessed from multiple threads without synchronization (set on lines 1058 and 1074, read on line 1207). This should be marked as volatile to ensure proper memory visibility across threads.

Suggested change

private bool _emitEvents;

private volatile bool _emitEvents;

Same as #117148 (comment).

Copilot · 2025-12-10T22:23:18Z

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

+                    void Append(Span<char> pathBuffer, ReadOnlySpan<char> path)
+                    {
+                        if (path.Length == 0)
+                        {
+                            return;
+                        }
+
+                        if (length != 0 && pathBuffer[length - 1] != '/')
                        {
-                            builder.Append(System.IO.Path.DirectorySeparatorChar);
+                            pathBuffer[length] = '/';
+                            length++;
                        }
+
+                        path.CopyTo(pathBuffer.Slice(length));
+                        length += path.Length;
                    }


The Append local function in GetPath does not perform bounds checking before writing to pathBuffer. If the total path length exceeds the buffer size (PATH_MAX = 4096), this will throw an IndexOutOfRangeException. Consider adding a check to ensure 'length + path.Length' (plus 1 for separator) does not exceed pathBuffer.Length before copying, or handle potential exceptions gracefully.

Yes, the code assumes path length does not to exceed PATH_MAX.

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

adamsitnik

I've reviewed 30-40% of the code. Since I am not very familiar with this code and I know that historically we have disabled plenty of FSW tests (they were flaky) my next step will be to implement stress tests (and when the implementation passes them review the rest).

Big thanks for your contribution @tmds !

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs

Co-authored-by: Copilot <[email protected]>

…y and RemoveUnusedINotifyWatches.

adamsitnik

@tmds Could you please run these stress tests with just .NET 10 and with your new implementation and report back the results?

For the latter you should be able to do that with corerun $pathToCoreRun FswStressTests.dll

tmds · 2025-12-12T11:21:18Z

@adamsitnik, thanks for looking at this! I'll run the stress tests on Monday and report how it went.

tmds · 2025-12-15T13:00:29Z

I built vmr's 10.0 branch with this implementation and ran the stress tests:

$ dotnet run -c Release
=== FileSystemWatcher Stress Tests ===
Started at: 2025-12-15 13:48:48

[1/15] Running: Basic File Creation
    ✓ PASSED in 0.02s

[2/15] Running: Multiple File Operations
    ✓ PASSED in 0.31s

[3/15] Running: Directory Creation
    ✓ PASSED in 0.00s

[4/15] Running: Recursive Directory Monitoring
    ✓ PASSED in 0.15s

[5/15] Running: High-Frequency File Operations
    ✓ PASSED in 2.01s

[6/15] Running: Large File Monitoring
    ✓ PASSED in 0.01s

[7/15] Running: Concurrent Watchers
    ✓ PASSED in 1.00s

[8/15] Running: Rename Operations
    ✓ PASSED in 0.41s

[9/15] Running: Rapid Create/Delete Cycles
    ✓ PASSED in 3.02s

[10/15] Running: Nested Directory Operations
    ✓ PASSED in 2.37s

[11/15] Running: Parallel Watchers and File Creation
    ✓ PASSED in 2.01s

[12/15] Running: Parallel Create/Delete Cycles
    ✓ PASSED in 8.01s

[13/15] Running: Rapid Watcher Start/Stop with Producer Thread
    ✓ PASSED in 10.02s

[14/15] Running: Aggressive Watcher Lifecycle Management
    ✓ PASSED in 0.78s

[15/15] Running: Global Watcher Contention Test
    Created 6492 watchers, 4037 received events, total 8336495 events
    ✓ PASSED in 8.55s

=== Summary ===
Total scenarios: 15
Passed: 15
Failed: 0
Total execution time: 38.67s

All tests passed!

To verify I used this implementation for the test run, I checked the inotify_add_watch syscalls, and all of them were using the IN_MASK_ADD flag (which is added in this PR).

adamsitnik · 2025-12-17T10:51:53Z

I built vmr's 10.0 branch with this implementation and ran the stress tests:

$ dotnet run -c Release
=== FileSystemWatcher Stress Tests ===
Started at: 2025-12-15 13:48:48

[1/15] Running: Basic File Creation
    ✓ PASSED in 0.02s

[2/15] Running: Multiple File Operations
    ✓ PASSED in 0.31s

[3/15] Running: Directory Creation
    ✓ PASSED in 0.00s

[4/15] Running: Recursive Directory Monitoring
    ✓ PASSED in 0.15s

[5/15] Running: High-Frequency File Operations
    ✓ PASSED in 2.01s

[6/15] Running: Large File Monitoring
    ✓ PASSED in 0.01s

[7/15] Running: Concurrent Watchers
    ✓ PASSED in 1.00s

[8/15] Running: Rename Operations
    ✓ PASSED in 0.41s

[9/15] Running: Rapid Create/Delete Cycles
    ✓ PASSED in 3.02s

[10/15] Running: Nested Directory Operations
    ✓ PASSED in 2.37s

[11/15] Running: Parallel Watchers and File Creation
    ✓ PASSED in 2.01s

[12/15] Running: Parallel Create/Delete Cycles
    ✓ PASSED in 8.01s

[13/15] Running: Rapid Watcher Start/Stop with Producer Thread
    ✓ PASSED in 10.02s

[14/15] Running: Aggressive Watcher Lifecycle Management
    ✓ PASSED in 0.78s

[15/15] Running: Global Watcher Contention Test
    Created 6492 watchers, 4037 received events, total 8336495 events
    ✓ PASSED in 8.55s

=== Summary ===
Total scenarios: 15
Passed: 15
Failed: 0
Total execution time: 38.67s

All tests passed!

To verify I used this implementation for the test run, I checked the inotify_add_watch syscalls, and all of them were using the IN_MASK_ADD flag (which is added in this PR).

Excellent! Our plan is that @jozkee is going to review the PR and I am going to merge it in the 2nd week of January.

FileSystemWatcher.Linux: use a single inotify instance and refactor w…

d67ff45

…atch tracking.

github-actions bot added the area-System.IO label Jun 30, 2025

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jun 30, 2025

tmds commented Jun 30, 2025

View reviewed changes

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs Outdated Show resolved Hide resolved

This was referenced Jun 30, 2025

browser-wasm Windows build error #116746

Closed

wasm build failure in CI #117017

Open

tmds added 2 commits July 1, 2025 09:24

Fix xunit test hang.

3fbd374

csproj: try fix CI leg build failures by adding References in all non…

07e86f3

…-Windows configurations.

build-analysis bot mentioned this pull request Jul 1, 2025

Occasional failure in "browser-wasm windows Release LibraryTests: Build Product" #116671

Closed

Don't consider ignoreFd in RemoveWatchedDirectoryFromParentAndWatches…

3cb75b0

… or we won't update _wdToWatch.

adamsitnik reviewed Jul 2, 2025

View reviewed changes

tmds added 6 commits July 2, 2025 20:33

Explain why RemoveUnusedINotifyWatches takes the watchers lock.

4d64e82

Detect when the watchers dropped to zero in a single place.

2941f9f

Check if watchers didn't drop to zero on every IN_IGNORED.

3d91dee

Fix and simplify how we detect when all watchers stopped.

41546fb

Don't accept new Watchers when all watchers stopped.

680c50a

Rename variable.

eeab032

DequeueEvents: stop emitting events when we see IsStopped.

bf5c14d

build-analysis bot mentioned this pull request Jul 3, 2025

[browser] HalfTests.ExplicitConversion_FromSingle failing due to NaN != NaN #103347

Open

tmds added 2 commits July 7, 2025 09:21

Start the Watcher after setting enabled. Use static s_watchersLock di…

a9ac9b3

…rectly.

Remove unneeded lock.

679643d

tmds added 4 commits July 8, 2025 19:59

Assign RootDirectory in AddOrUpdateWatchedDirectory.

042b4a1

Add additional Debug.Assert Monitor.IsEntered for checking locks are …

e3dd435

…taken.

Merge branch 'main' into inotify

4df6f8e

Add some notes about the buffer/queue sizes.

8e65598

build-analysis bot mentioned this pull request Jul 10, 2025

CI failure in System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.ThresholdExceeded_ThrowsException #117484

Closed

simonegli8 mentioned this pull request Jul 12, 2025

Page or Control modifications don't cause Recompilation webformsforcore/WebFormsForCore#3

Open

jeffhandley added this to the 11.0.0 milestone Sep 2, 2025

jeffhandley requested a review from jozkee September 2, 2025 02:45

jeffhandley assigned jozkee Sep 2, 2025

Merge branch 'main' into inotify

c9b2ea4

Mbucari mentioned this pull request Nov 5, 2025

Installer manipulates fs.inotify.max_user_instances and commits immediately rmcrackan/Libation#1409

Open

Non-functional cleanup.

db7cd33

build-analysis bot mentioned this pull request Dec 4, 2025

Host Activation test failures on OSX #122169

Closed

Copilot AI mentioned this pull request Dec 4, 2025

Increase HostActivation test hang dump timeout to 8 minutes for OSX CI #122171

Merged

adamsitnik requested a review from Copilot December 10, 2025 22:12

Copilot started reviewing on behalf of adamsitnik December 10, 2025 22:13 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

adamsitnik reviewed Dec 10, 2025

View reviewed changes

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs Show resolved Hide resolved

src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs Outdated Show resolved Hide resolved

tmds and others added 2 commits December 11, 2025 09:02

Apply suggestions from Copilot.

219fbe5

Co-authored-by: Copilot <[email protected]>

Make more clear how StopINotify syncs with AddOrUpdateWatchedDirector…

fcd6057

…y and RemoveUnusedINotifyWatches.

Copilot AI mentioned this pull request Dec 11, 2025

Add demanding stress tests for global FileSystemWatcher implementation adamsitnik/fsw-stress-tests#3

Merged

8 tasks

adamsitnik reviewed Dec 11, 2025

View reviewed changes

-                public bool IsStopped { get; set; }
+                private volatile bool _isStopped;
+                public bool IsStopped
+                {
+                    get => _isStopped;
+                    set => _isStopped = value;
+                }

FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148

Are you sure you want to change the base?

FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148

Uh oh!

Conversation

tmds commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmds commented Jun 30, 2025

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tmds commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmds commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmds commented Jul 10, 2025

Uh oh!

tmds commented Aug 18, 2025

Uh oh!

jeffhandley commented Sep 2, 2025

Uh oh!

tmds commented Nov 5, 2025

Uh oh!

tmds commented Nov 18, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

tmds Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

tmds Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

tmds Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

tmds commented Dec 12, 2025

Uh oh!

tmds commented Dec 15, 2025

Uh oh!

adamsitnik commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

tmds commented Jun 30, 2025 •

edited

Loading

tmds commented Jul 2, 2025 •

edited

Loading

tmds commented Jul 7, 2025 •

edited

Loading