node.js scandir confuses files and directories in a bind mount in a Windows container

[mwfriedm]

Describe the bug
node.js scandir confuses files and directories in a bind mount in a Windows container.

See isaacs/node-glob#436. This is the same issue, but with more detailed repro steps.

To Reproduce
Dockerfile:

FROM windows/servercore:ltsc2019 

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

ENV NODE_VERSION=16.13.1

# node
RUN \
  Write-Host INFO: Downloading node ${env:NODE_VERSION}...; \
  Invoke-WebRequest -Uri "https://nodejs.org/dist/v${env:NODE_VERSION}/node-v${env:NODE_VERSION}-x64.msi" -OutFile C:\node-x64.msi; \
  Write-Host INFO: Installing node...; \
  Start-Process -Wait -NoNewWindow -FilePath msiexec.exe -ArgumentList @('/package', 'C:\node-x64.msi', '/quiet', '/norestart'); \
  Remove-Item C:\node-x64.msi; \
  Write-Host INFO: Completed

RUN \
  npm install --global yarn

Create another folder for source files, outside of the docker context.
mkdir demo

Add this package.json file to demo

{
  "name": "nodejs-docker-bind-mount",
  "version": "1.0.0",
  "description": "Repro demo",
  "private": true,
  "license": "UNLICENSED",
  "workspaces": [
    "*"
  ]
}

Create a subfolder containing a yarn workspace
mkdir demo\subfolder

Add this package.json file to demo\subfolder

{
  "name": "subfolder",
  "version": "1.0.0",
  "description": "Repro demo",
  "private": true,
  "license": "UNLICENSED"
}

docker build the Dockerfile
docker run -it -m4g -v c:\path\to\demo:c\mountpoint -w c:\mountpoint [image id from docker build]

Inside the container:

C:\mountpoint >yarn install
yarn install v1.22.18
glob error [Error: EISDIR: illegal operation on a directory, scandir 'C:\mountpoint \package.json'] {
  errno: -4068,
  code: 'EISDIR',
  syscall: 'scandir',
  path: 'C:\\mountpoint \\package.json'
}

Expected behavior
Run the same command outside the container (assuming you have node.js and yarn installed), and it works without an error. Yarn is calling glob over the contents of the directory, and iterating the results based on the entry type. In the bind mount, it is getting confused by the files -- presumably because they are some sort of unexpected reparse point? I have no idea if this is an issue in hcsshim, or in the node.js libuv IO provider.

c:\path\to\demo> yarn install
yarn install v1.22.10
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.

Configuration:

• Edition: Windows 10 Enterprise Version 1909
• Base Image being used: Windows Server Core, ltsc2019
• Container engine: docker
• Container Engine version: 20.10.11

Additional context

docker.exe info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-rc.2)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Server Version: 20.10.11
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 18363 (18362.1.amd64fre.19h1_release.190318-1202)
 Operating System: Windows 10 Enterprise Version 1909 (OS Build 18363.2158)
 OSType: windows
 Architecture: x86_64

[cwilhit]

Thanks for the details, confirming this reproed for me as well on a WS2022 machine. It works fine if you run the container under process isolation, but with Hyper-V isolation, it threw the EISDIR error as you said. Acknowledging that process isolation is not a mitigation for you since you're on a 1909 host and are targeting a 2019 image.

I've referred this issue to some engineers on my side and will get back to you when we have an update.

[cwilhit]

Opened MSFT internal 38776845 to track. Not much to share yet.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

Definitely continues to repro. The impact for my workload is that I can't use bind mounts to run my node.js applications (or build scripts) in a Windows container. You can workaround this issue by making a local copy of the bind mount inside the container's filesystem. But that's unnecessary overhead. The same scenario works fine in Ubuntu or Centos containers using the Linux docker platform.

This is a common pattern in continuous integration (CI) workloads. The CI server checks source out on an agent machine, then bind mounts the source directory into a container to run something. Examples:

• https://docs.cloudbees.com/docs/admin-resources/latest/plugins/docker-workflow
• https://www.jetbrains.com/help/teamcity/docker-wrapper.html#how-it-works-1

[travnick]

It totally blocks using volumes for builds directories, as it's advised - use volumes to not store data as layers in container.

host:

docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.17
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19044 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2009 (OS Build 19044.1826)
 OSType: windows
 Architecture: x86_64

inside container:

systeminfo.exe

Host Name:                 5DF10581AACD
OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.19042 N/A Build 19042

System Model:              Virtual Machine
System Type:               x64-based PC

BIOS Version:              Microsoft Corporation Hyper-V UEFI Release v4.0, 11/1/2019

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[fady-azmy-msft]

No updates to share yet, still working on this issue.

[ghost]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[mwfriedm]

No updates, just bumping so that the bot doesn't close this. I can confirm that this is still an issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[plachta11b]

Any news? I hope this message prevents the bot from closing the issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[fjs4]

This issue has been sitting idle for too long and deserves an update. Within Microsoft we've been looking into this issue. The root problem is related to differences between how mapped folders are exposed to Hyper-V containers. With process-isolated containers, we can do some straightforward namespace virtualization to expose the host folder into the container. Things are a bit more complicated with Hyper-V containers. We still leverage namespace virtualization, but we also need a way to expose the folder between machines. Therefore, with Hyper-V containers we use an SMB share from the host to share content with the container and then use namespace virtualization to make it appear properly within the container namespace.

SMB, however, blocks certain operations for security reasons. For example, SMB blocks the creation of certain reparse points as that would potentially allow the requestor a way to escape the share and get access to other host content. With the issue described in this thread, node.js is trying to create a junction and it is being blocked by SMB.

So, in a sense the behavior is by design for SMB. However, the use of SMB is just an implementation detail, and our desire is functional parity between container types. We are therefore investigating options for how we can maintain the required security but still allow this type of functionality within a container. As mentioned earlier in this thread, a work around for this problem is to copy content from the shared folder to the local volume for processing.

We'll update the thread with more details as our internal investigation concludes.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@mwfriedm, please provide an update or close this issue.