%ProgramData%\Microsoft\VisualStudio\Packages

This is the default directory where Visual Studio stores package manifests and (optionally) payloads. Non-default locations are stored in the CachePath value below the first registry key that is found in the following list:

• HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VisualStudio\Setup]
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\Setup`]
• HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\Setup`

Apparently, this directory is also referred to as the download cache.

Files in this directory can be deleted. But, of course, they'll have to be downloaded again in order to update, modify or deinstall Visual Studio.