How To Speed Up Shared File Access In Docker For Mac

Troubleshooting Network File Copy Slowness https: 10 and mac OS. If it takes more than a minute to copy a 100MB file, your connection is too slow to be. Docker for Mac: Overcoming Slow Mounted Volumes For about two years, I've.

This page contains information on how to diagnose and troubleshoot Docker Desktop issues, send logs and communicate with the Docker Desktop team, use our forums and Success Center, browse and log issues on GitHub, and find workarounds for known problems.

Troubleshoot

Choose > Troubleshootfrom the menu bar to see the troubleshoot options.

The Troubleshoot page contains the following options:

• Restart Docker Desktop: Select to restart Docker Desktop.

• Run Diagnostics: Select this option to diagnose any issues on Docker Desktop. For detailed information about diagnostics, see Diagnose problems, send feedback, and create GitHub issues.

• Reset Kubernetes cluster: Select this option to delete all stacks and Kubernetes resources. For more information, see Kubernetes.

• Reset disk image: This option resets all Docker data without areset to factory defaults. Selecting this option results in the loss of existing settings.

• Reset to factory defaults: Choose this option to reset all options onDocker Desktop to their initial state, the same as when Docker Desktop was first installed.

• Uninstall: Choose this option to remove Docker Desktop from yoursystem.

Uninstall Docker Desktop from the command line

To uninstall Docker Desktop from a terminal, run: <DockerforMacPath>--uninstall. If your instance is installed in the default location, thiscommand provides a clean uninstall:

You might want to use the command-line uninstall if, for example, you find thatthe app is non-functional, and you cannot uninstall it from the menu.

Diagnose problems, send feedback, and create GitHub issues

In-app diagnostics

If you encounter problems for which you do not find solutions in thisdocumentation, on Docker Desktop issues onGitHub, or the Docker Desktop forum, we can help you troubleshootthe log data.

Choose > Troubleshoot > Run Diagnostics.

Once the diagnostics are available, you can upload them and obtain aDiagnostic ID, which must be provided when communicating with the Dockerteam. For more information on our policy regarding personal data, seehow is personal data handled in DockerDesktop.

If you click Report an issue, this opens Docker Desktop for Mac issues on GitHub in your web browser in a “New issue” template. Add the details before submitting the issue. Do not forget to copy/paste your diagnostic ID.

Diagnosing from the terminal

In some cases, it is useful to run the diagnostics yourself, for instance, ifDocker Desktop cannot start.

First, locate the com.docker.diagnose tool. If you have installed Docker Desktop in the Applications directory, then it is located at/Applications/Docker.app/Contents/MacOS/com.docker.diagnose.

To create and upload diagnostics, run:

After the diagnostics have finished, you should have the following output,containing your diagnostics ID:

The diagnostics ID (here BE9AFAAF-F68B-41D0-9D12-84760E6B8740/20190905152051) iscomposed of your user ID (BE9AFAAF-F68B-41D0-9D12-84760E6B8740) and a timestamp(20190905152051). Ensure you provide the full diagnostics ID, and not just the user ID.

To view the contents of the diagnostic file, run:

Check the logs

In addition to using the diagnose and feedback option to submit logs, you canbrowse the logs yourself. The following documentation is about macOS 10.12onwards; for older versions, see olderdocumentation.

In a terminal

To watch the live flow of Docker Desktop logs in the command line, run the following script from your favorite shell.

Alternatively, to collect the last day of logs (1d) in a file, run:

In the Console app

Macs provide a built-in log viewer, named “Console”, which you can use to checkDocker logs.

The Console lives in /Applications/Utilities; you can search for it withSpotlight Search.

To read the Docker app log messages, type docker in the Console window search bar and press Enter. Then select ANY to expand the drop-down list next to your docker search entry, and select Process.

You can use the Console Log Query to search logs, filter the results in variousways, and create reports.

Troubleshooting

Make sure certificates are set up correctly

Docker Desktop ignores certificates listed under insecure registries, and doesnot send client certificates to them. Commands like docker run that attempt topull from the registry produces error messages on the command line, for example:

As well as on the registry. For example:

For more about using client and server side certificates, see Adding TLScertificates in the Getting Started topic.

Docker Desktop does not start if Mac user account and home folder are renamed after installing the app

See Do I need to reinstall Docker Desktop if I change the name of my macOSaccount?in the FAQs.

Volume mounting requires file sharing for any project directories outside of /Users

If you are using mounted volumes and get runtime errors indicating anapplication file is not found, access to a volume mount is denied, or a servicecannot start, such as when using Docker Compose,you might need to enable file sharing.

Volume mounting requires shared drives for projects that live outside of the/Users directory. Go to >Preferences > Resources > File sharing and share the drive that contains the Dockerfile and volume.

Incompatible CPU detected

Docker Desktop requires a processor (CPU) that supports virtualization and, morespecifically, the Apple Hypervisorframework.Docker Desktop is only compatible with Mac systems that have a CPU that supports the Hypervisor framework. Most Macs built in 2010 and later support it,as described in the Apple Hypervisor Framework documentation about supported hardware:

Generally, machines with an Intel VT-x feature set that includes Extended PageTables (EPT) and Unrestricted Mode are supported.

To check if your Mac supports the Hypervisor framework, run the following command in a terminal window.

If your Mac supports the Hypervisor Framework, the command printskern.hv_support: 1.

If not, the command prints kern.hv_support: 0.

See also, Hypervisor FrameworkReferencein the Apple documentation, and Docker Desktop Mac system requirements.

Workarounds for common problems

• If Docker Desktop fails to install or start properly on Mac:

  • Make sure you quit Docker Desktop before installing a new version of theapplication ( > Quit Docker Desktop). Otherwise, you get an “application in use” error when you try tocopy the new app from the .dmg to /Applications.

  • Restart your Mac to stop / discard any vestige of the daemon running fromthe previously installed version.

  • Run the uninstall commands from the menu.

• If docker commands aren’t working properly or as expected, you may need tounset some environment variables, to make sure you are not using the legacyDocker Machine environment in your shell or command window. Unset theDOCKER_HOST environment variable and related variables.

  • If you use bash, use the following command: unset ${!DOCKER_*}

  • For other shells, unset each environment variable individually as describedin Setting up to run Docker Desktop onMac in Docker Desktop on Mac vs. Docker Toolbox.

• Network connections fail if the macOS Firewall is set to “Block all incomingconnections”. You can enable the firewall, but bootpd must be allowedincoming connections so that the VM can get an IP address.

• For the hello-world-nginx example, Docker Desktop must be running to get tothe web server on http://localhost/. Make sure that the Docker icon isdisplayed on the menu bar, and that you run the Docker commands in a shell that is connected to the Docker Desktop Engine (not Engine from Toolbox).Otherwise, you might start the webserver container but get a “web page notavailable” error when you go to localhost. For more information on distinguishing between the two environments, see Docker Desktop on Mac vs. Docker Toolbox.

• If you see errors like Bind for 0.0.0.0:8080 failed: port is alreadyallocated or listen tcp:0.0.0.0:8080: bind: address is already in use:

  • These errors are often caused by some other software on the Mac using thoseports.

  • Run lsof -i tcp:8080 to discover the name and pid of the other process anddecide whether to shut the other process down, or to use a different port inyour docker app.

Known issues

• IPv6 is not (yet) supported on Docker Desktop.

  A workaround is provided that auto-filters out the IPv6 addresses in DNSserver lists and enables successful network access. For example,2001:4860:4860::8888 would become 8.8.8.8. To learn more, see theseissues on GitHub and Docker Desktop forums:

• You might encounter errors when using docker-compose up with Docker Desktop(ValueError: Extra Data). We’ve identified this is likely related to dataand/or events being passed all at once rather than one by one, so sometimesthe data comes back as 2+ objects concatenated and causes an error.

• Force-ejecting the .dmg after running Docker.app from it can cause thewhale icon to become unresponsive, Docker tasks to show as not responding inthe Activity Monitor, and for some processes to consume a large amount of CPUresources. Reboot and restart Docker to resolve these issues.

• Docker does not auto-start on login even when it is enabled in > Preferences. This is related to aset of issues with Docker helper, registration, and versioning.

• Usb and midi drivers for mac for yamaha ypg-635. Docker Desktop uses the HyperKit hypervisor(https://github.com/docker/hyperkit) in macOS 10.10 Yosemite and higher. Ifyou are developing with tools that have conflicts with HyperKit, such asIntel Hardware Accelerated Execution Manager(HAXM),the current workaround is not to run them at the same time. You can pauseHyperKit by quitting Docker Desktop temporarily while you work with HAXM.This allows you to continue work with the other tools and prevent HyperKitfrom interfering.

• If you are working with applications like ApacheMaven that expect settings for DOCKER_HOST andDOCKER_CERT_PATH environment variables, specify these to connect to Dockerinstances through Unix sockets. For example:

• docker-compose 1.7.1 performs DNS unnecessary lookups forlocalunixsocket.local which can take 5s to timeout on some networks. Ifdocker-compose commands seem very slow but seem to speed up when the networkis disabled, try appending 127.0.0.1 localunixsocket.local to the file/etc/hosts. Alternatively you could create a plain-text TCP proxy onlocalhost:1234 using:

  and then export DOCKER_HOST=tcp://localhost:1234.

• There are a number of issues with the performance of directories bind-mountedwith osxfs. In particular, writes of small blocks, and traversals of largedirectories are currently slow. Additionally, containers that perform largenumbers of directory operations, such as repeated scans of large directorytrees, may suffer from poor performance. Applications that behave in this wayinclude:

  • rake
  • ember build
  • Symfony
  • Magento
  • Zend Framework
  • PHP applications that use Composer to installdependencies in a vendor folder
    

  As a work-around for this behavior, you can put vendor or third-party librarydirectories in Docker volumes, perform temporary file system operationsoutside of osxfs mounts, and use third-party tools like Unison or rsync tosynchronize between container directories and bind-mounted directories. We areactively working on osxfs performance using a number of differenttechniques. To learn more, see the topic on Performance issues, solutions,and roadmap.

• If your system does not have access to an NTP server, then after a hibernatethe time seen by Docker Desktop may be considerably out of sync with the host.Furthermore, the time may slowly drift out of sync during use. To manuallyreset the time after hibernation, run:

  Or, to resolve both issues, you can add the local clock as a low-priority(high stratum) fallback NTP time source for the host. To do this, edit thehost’s /etc/ntp-restrict.conf to add:

  Then restart the NTP service with:

Docker just released a native MacOS runtime environment to run containers on Macs with ease. They fixed many issues, but the bitter truth is they missed something important. The read and write access for mounted volumes is terrible.

Benchmarks

We can spin up a container and write to a mounted volume by executing the following commands:

1. Start a container
2. Mount the current directory
3. Write random data to a file in this directory

Let’s compare the results of Windows, Cent OS and Mac OS:

Windows 10

Cent OS

Mac OS

So the winner is…. 19 seconds for writing. For reading it is quiet similar. When you develop a big dockerized application then you are in a bad spot. Usually you would work on your source code and expect no slowdowns for building. But the bitter truth is it will take ages.

This GitHub issue tracks the current state. There is a lot of hate so better listen to the “members” instead of reading all the frustrations.

@dsheetz from the Docker for Mac team nailed the issue:

Perhaps the most important thing to understand is that shared file system performance is multi-dimensional. This means that, depending on your workload, you may experience exceptional, adequate, or poor performance with osxfs, the file system server in Docker for Mac. File system APIs are very wide (20-40 message types) with many intricate semantics involving on-disk state, in-memory cache state, and concurrent access by multiple processes. Additionally, osxfs integrates a mapping between OS X's FSEvents API and Linux's inotify API which is implemented inside of the file system itself complicating matters further (cache behavior in particular).

At the highest level, there are two dimensions to file system performance: throughput (read/write IO) and latency (roundtrip time). In a traditional file system on a modern SSD, applications can generally expect throughput of a few GB/s. With large sequential IO operations, osxfs can achieve throughput of around 250 MB/s which, while not native speed, will not be the bottleneck for most applications which perform acceptably on HDDs.

Latency is the time it takes for a file system system call to complete. For instance, the time between a thread issuing write in a container and resuming with the number of bytes written. With a classical block-based file system, this latency is typically under 10μs (microseconds). With osxfs, latency is presently around 200μs for most operations or 20x slower. For workloads which demand many sequential roundtrips, this results in significant observable slow down. To reduce the latency, we need to shorten the data path from a Linux system call to OS X and back again. This requires tuning each component in the data path in turn -- some of which require significant engineering effort. Even if we achieve a huge latency reduction of 100μs/roundtrip, we will still 'only' see a doubling of performance. This is typical of performance engineering, which requires significant effort to analyze slowdowns and develop optimized components.

Many people created workarounds with different approaches. Some of them use nfs, Docker in Docker, Unison 2 way sync or rsync. I tried some solutions but non of them worked for my docker container that contains a big Java monolith. Either they install extra tools like vagrant to reduce the pain. Vagrant uses nfs but this is still slow compared to native write and read performance. Or they are unreliable, hard to setup and hard to maintain.

I made a step back and thought about the root issue again. A very good approach is docker-sync. It’s a ruby application with a lot of options. One very mature option is file synchronisation based upon rsync.

Rsync

Rsync initial release was in 1996 (20 years ago). It’s used for transferring files across computer systems. One important use case is 1-way synchronization.

Sounds good…, right ?

Docker-sync supports rsync for synchronization. In the beginning it worked but a few days later I got some connection issues between my host and my container.

Do you know the feeling when you want to fix something but it feels so far away? You realise you don’t understand what’s happing behind the scenes.

The rsync approach sounds right. It tackles the root of the issue: operating on mounted files right now is damn slow.

I tried other solutions but without real success.

Build a custom image

So let’s try to get our hands dirty. You start a rsync server in the container and connect to it using rsync. This approach works for many years for other use-cases.

Let’s setup a docker Centos 6 container with an installed and configured rsync service.

1. The Dockerfile

2. Build the container within the repository directory.

3. Start the container and map the rsync server port to a specific host port.

Now we need to sync our share directory and sync any changes again as soon as anything changes. Rsync will only sync changes after an initial sync.

Fswatch utilizes rsync to talk to the container as soon as something changes. We do not use any kind of docker volume mounting. Hence all file operations will stay in the container and will be fast. Whenever we change something rsync transfers it to the container using . For sure you can use all rsync features like delete rules or exclude patterns.

If we change something (it does not matter if it’s a small project or a huge one) then we see something like

0.02 seconds, great !

Fswatch uses file system events on Mac OS. Thus is still very fast and you can event tweak it. For example by excluding build related directories like target or node_modules.

Sources are available on GitHub.

For small projects the bad performance is not a critical issue. For huge application rsync is our hero. Good old tools, and still reliable and important.

Especially for all guys who love Mac OS and need to use a VM know the pain. Issues like the command key mapping are annoying. Either you map it to the Windows key or in the end you don’t use it anymore. So on Mac OS you use cmd+c to copy something and in your container you use control. For sure you can also map your host control to command but then you have again other issues. Everything is better when you can work in Mac OS instead of in a virtual machine as a mac user.

I hope you enjoyed the reading. If you like it and feel a round of applause, drop a comment and subscribe our mailing list.

I work with ubuntu-technology. We are a very young software engineering company located in Dresden, Germany. We are looking for long term projects and partners. If you need support in certain software engineering or AI-related fields, drop a message in our contact form.

Happy coding :)