# Plugin GUI Magnification in a Bridge Plugin
### Windows.Graphics.Capture + DWM Cloak + In-Process Input Injection
Implementation guide for adding interactive, arbitrarily-scaled rendering of hosted plugin GUIs
inside a bridge/offload host (e.g. GrandBridge FX offload process), without moving the real
window off-screen and without touching the audio path.
**Target:** Windows 10 2004+ / Windows 11, x64, C++17+, D3D11.
**Scope:** the hosted plugin's editor HWND lives in *your* offload process — this guide relies on that.
---
## 1. Architecture Overview
```
┌──────────────────────────── Offload process ────────────────────────────┐
│ │
│ Plugin GUI thread Magnifier render thread │
│ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ Real plugin HWND │── DWM compose ─▶│ WGC FramePool (D3D11) │ │
│ │ (CLOAKED) │ │ → scale shader → present│ │
│ │ child HWND tree │◀─ posted msgs ──│ MagnifierWindow (visible)│ │
│ └─────────────────┘ └──────────────────────────┘ │
│ ▲ ▲ │
│ │ └── IAT/inline hooks: GetCursorPos, GetAsyncKeyState, ... │
│ └────── WinEvent hook: popup detection / reposition │
│ │
│ Audio threads (realtime prio) ──── never touched by any of the above │
└──────────────────────────────────────────────────────────────────────────┘
```
Components:
1. **CaptureSession** — one per hosted plugin editor. WGC item + frame pool + D3D11 texture.
2. **MagnifierWindow** — the visible window the user interacts with. Owns a swapchain, draws
the latest captured frame scaled, translates and forwards input.
3. **CursorStateVirtualizer** — process-wide hooks that lie about cursor position / button state
to the plugin GUI thread while a magnified interaction is in progress.
4. **PopupManager** — WinEvent hook that catches transient top-level windows (menus, dropdowns,
tooltips) created by plugin GUI threads and repositions (or captures) them.
5. **Registry** — HWND → PluginInstance mapping so input routing and popup attribution never guess.
---
## 2. Prerequisites & Project Setup
- C++/WinRT (header-only): `winrt/Windows.Graphics.Capture.h`,
`winrt/Windows.Graphics.DirectX.Direct3D11.h`
- Link: `d3d11.lib`, `dxgi.lib`, `dwmapi.lib`, `windowsapp.lib` (or `RuntimeObject.lib` for older setups)
- Interop header: `windows.graphics.capture.interop.h` (for `IGraphicsCaptureItemInterop::CreateForWindow`)
- Direct3D interop: `windows.graphics.directx.direct3d11.interop.h` (`CreateDirect3D11DeviceFromDXGIDevice`)
- Hooking library: **MinHook** (small, license-friendly) or Detours.
- Call `winrt::init_apartment(winrt::apartment_type::single_threaded)` on the thread that creates
the frame pool if you use `Create` (DispatcherQueue required), or use
**`CreateFreeThreaded`** (recommended here — no dispatcher, callbacks arrive on a pool thread).
Feature detection at startup:
```cpp
bool wgcAvailable = winrt::Windows::Graphics::Capture::GraphicsCaptureSession::IsSupported();
bool canHideBorder = winrt::Windows::Foundation::Metadata::ApiInformation::IsPropertyPresent(
L"Windows.Graphics.Capture.GraphicsCaptureSession", L"IsBorderRequired"); // Win10 2104+/Win11
```
If `wgcAvailable == false` (very old Win10), fall back to a `PrintWindow(hwnd, dc,
PW_RENDERFULLCONTENT)` polling path (Section 10).
> **Note on `IsBorderRequired`:** setting it to `false` requires the
> `GraphicsCaptureAccess::RequestAccessAsync(GraphicsCaptureAccessKind::Borderless)` consent on
> some builds for packaged apps; for a normal Win32 process it generally just works on Win11.
> Test on your minimum supported OS. The yellow border only matters cosmetically — worst case,
> the *cloaked* window has a border nobody sees.
---
## 3. Cloaking the Real Editor Window
Never move the window off-screen and never `SW_HIDE` it (hidden windows stop being composed
and WGC delivers nothing). Cloak it instead — DWM keeps composing a cloaked window, but it is
invisible on the desktop and excluded from Alt-Tab thumbnails.
```cpp
#include <dwmapi.h>
void SetCloaked(HWND hwnd, bool cloak)
{
BOOL v = cloak ? TRUE : FALSE;
DwmSetWindowAttribute(hwnd, DWMWA_CLOAK, &v, sizeof(v));
}
```
Rules:
- Cloak the **top-level** window that hosts the plugin editor. If your bridge parents the plugin
editor into a child HWND of a bridge-owned top-level window, cloak that top-level window and
capture it (WGC captures top-level windows; child HWNDs are captured as part of their root).
- Keep the window **restored** (not minimized) and at a fixed, known desktop position. Its real
screen rect is your coordinate anchor for popups and wheel messages. A convenient trick:
park all cloaked editors at the same location, e.g. the top-left of the primary monitor —
overlap is irrelevant since nothing is visible and WGC capture is per-window.
- Some plugin GUIs throttle repaint timers when they believe they are occluded/minimized.
Cloaked ≠ minimized, so most keep painting. If one goes lazy: periodic
`RedrawWindow(hwnd, nullptr, nullptr, RDW_INVALIDATE | RDW_ALLCHILDREN)` at your capture
cadence, or keep its animation timer alive (you own the process). Handle this per-plugin,
only when observed.
---
## 4. Capture Session (WGC → D3D11 texture)
One `CaptureSession` per hosted editor. Everything stays on the GPU.
```cpp
#include <winrt/Windows.Graphics.Capture.h>
#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
#include <windows.graphics.capture.interop.h>
#include <windows.graphics.directx.direct3d11.interop.h>
using namespace winrt;
using namespace winrt::Windows::Graphics;
using namespace winrt::Windows::Graphics::Capture;
using namespace winrt::Windows::Graphics:
irectX;
class CaptureSession
{
public:
// d3dDevice: create ONE shared D3D11 device for all capture sessions + magnifier rendering.
// Use D3D11_CREATE_DEVICE_BGRA_SUPPORT.
void Start(HWND source, ID3D11Device* d3dDevice)
{
m_source = source;
// Wrap the D3D device for WinRT
com_ptr<IDXGIDevice> dxgi;
d3dDevice->QueryInterface(IID_PPV_ARGS(dxgi.put()));
com_ptr<::IInspectable> inspectable;
check_hresult(CreateDirect3D11DeviceFromDXGIDevice(dxgi.get(), inspectable.put()));
m_winrtDevice = inspectable.as<DirectX:irect3D11::IDirect3DDevice>();
// Create the capture item for the (cloaked) window
auto interop = get_activation_factory<GraphicsCaptureItem,
IGraphicsCaptureItemInterop>();
check_hresult(interop->CreateForWindow(
source, guid_of<GraphicsCaptureItem>(),
put_abi(m_item)));
m_lastSize = m_item.Size();
// Free-threaded pool: no DispatcherQueue needed; 2 buffers is enough.
m_framePool = Direct3D11CaptureFramePool::CreateFreeThreaded(
m_winrtDevice,
DirectXPixelFormat::B8G8R8A8UIntNormalized,
/*numberOfBuffers*/ 2,
m_lastSize);
m_frameArrived = m_framePool.FrameArrived(auto_revoke,
{ this, &CaptureSession::OnFrameArrived });
m_session = m_framePool.CreateCaptureSession(m_item);
m_session.IsCursorCaptureEnabled(false); // we draw our own cursor if needed
TrySetBorderless(m_session); // IsBorderRequired(false) if available
m_session.StartCapture();
}
// Returns the most recent frame texture (thread-safe swap). May be null before first frame.
com_ptr<ID3D11Texture2D> LatestFrame()
{
std::scoped_lock lk(m_mutex);
return m_latest;
}
SizeInt32 ContentSize() const { return m_lastSize; }
// Call when the plugin resizes its editor (WM_WINDOWPOSCHANGED / IPlugFrame::resizeView)
void NotifySourceResized()
{
m_pendingRecreate = true; // handled on next FrameArrived
}
private:
void OnFrameArrived(Direct3D11CaptureFramePool const& pool,
winrt::Windows::Foundation::IInspectable const&)
{
auto frame = pool.TryGetNextFrame();
if (!frame) return;
auto size = frame.ContentSize();
if (m_pendingRecreate ||
size.Width != m_lastSize.Width || size.Height != m_lastSize.Height)
{
m_lastSize = size;
m_pendingRecreate = false;
pool.Recreate(m_winrtDevice,
DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, size);
}
// Extract the D3D11 texture and keep a reference; DO NOT copy on CPU.
auto surface = frame.Surface();
com_ptr<ID3D11Texture2D> tex;
auto access = surface.as<::Windows::Graphics:irectX:irect3D11::IDirect3DDxgiInterfaceAccess>();
check_hresult(access->GetInterface(IID_PPV_ARGS(tex.put())));
// Copy into our own texture (frame buffer is recycled by the pool).
// One GPU CopyResource per dirty frame — cheap.
com_ptr<ID3D11Texture2D> own = GetOrCreateStagingSizedTexture(tex.get(), size);
{
com_ptr<ID3D11DeviceContext> ctx; // use a deferred/immediate ctx guarded by a lock,
m_device->GetImmediateContext(ctx.put()); // or D3D11_CREATE_DEVICE flags + mutex
std::scoped_lock lk(m_ctxMutex);
ctx->CopyResource(own.get(), tex.get());
}
{
std::scoped_lock lk(m_mutex);
m_latest = own;
}
m_dirty.store(true, std::memory_order_release); // wake/flag the magnifier render
}
// ... members: m_item, m_framePool, m_session, m_winrtDevice, m_device,
// m_latest, m_lastSize, mutexes, m_dirty, m_pendingRecreate ...
};
```
Key decisions in this design:
- **`CreateFreeThreaded`** — callbacks arrive on a WinRT pool thread, never on the plugin GUI
thread and never on your audio threads. No DispatcherQueue plumbing.
- **2 buffers** — minimal latency and memory; you are mirroring, not recording.
- **Event-driven** — WGC only delivers frames when the source repaints. A static GUI costs ~0.
- **One shared D3D11 device** for all sessions and the renderer, so textures are directly usable
without cross-device copies. Guard the immediate context with a mutex (or use
`ID3D10Multithread::SetMultithreadProtected`).
---
## 5. Magnifier Window & Scaled Rendering
The magnifier window is a plain top-level window (or a child view inside your bridge editor UI)
with a DXGI flip-model swapchain.
Render loop (on a dedicated render thread, or driven by a `WM_TIMER`/composition tick):
1. Wait until `m_dirty` or a resize/interaction event; **cap at 30–60 fps** with a waitable
timer. Do not free-run.
2. `LatestFrame()` → draw a fullscreen quad sampling the texture with your scale filter:
- integer factors (2x, 3x): point/nearest for pixel-crisp look, or bilinear for smooth;
- non-integer (1.5x like your screenshot): bicubic (Catmull-Rom) in a pixel shader, or
bilinear + a light sharpen (FSR1/RCAS-style) — subjectively best for knob labels.
3. Present with `DXGI_SWAP_EFFECT_FLIP_DISCARD`, `Present(1, 0)` (vsync keeps GPU polite).
Sizing model:
- `scale = magnifierClientSize / captureContentSize` (keep aspect; letterbox or lock the
magnifier window's aspect via `WM_SIZING`).
- When the plugin resizes itself (expand/collapse panels), you get the new `ContentSize`
from the frame — resize the magnifier window to `content * userZoom` and continue. Handle
`WM_WINDOWPOSCHANGED` on the cloaked window too (subclass it) and call
`NotifySourceResized()` for immediate response.
Performance guardrails (audio-safety checklist):
- All scaling on GPU. Zero CPU pixel work. No `Map`/readback anywhere in the hot path.
- Render thread priority: `THREAD_PRIORITY_NORMAL` or `BELOW_NORMAL`. Never raise it.
- Frame-rate cap ≤ 60, default 30. Expose in settings.
- On `WM_SHOWWINDOW(FALSE)` / magnifier minimized: **pause the session**
(`m_frameArrived.revoke()` or stop presenting) — don't burn GPU for invisible output.
- Test with a meter-heavy plugin and measure: GPU % (Task Manager per-process), and your
audio callback headroom before/after. Expected impact: comparable to one extra animated
plugin GUI.
---
## 6. Input Injection
### 6.1 Coordinate mapping
```cpp
// magnifier client point → real (cloaked) window client point
POINT MapToSource(POINT magPt, SIZE magClient, SizeInt32 src)
{
POINT p;
p.x = MulDiv(magPt.x, src.Width, magClient.cx);
p.y = MulDiv(magPt.y, src.Height, magClient.cy);
return p; // client coords of the cloaked top-level window
}
```
### 6.2 Which HWND receives the message
Plugin GUIs are often HWND trees (JUCE = usually one HWND; VSTGUI/iPlug/custom = sometimes
children). Resolve the deepest child at the point:
```cpp
HWND HitTestChild(HWND root, POINT clientPt)
{
// ChildWindowFromPointEx walks one level; RealChildWindowFromPoint or a manual
// recursive descent handles nesting:
HWND h = root;
for (;
{
POINT p = clientPt;
MapWindowPoints(root, h, &p, 1);
HWND next = RealChildWindowFromPoint(h, p);
if (!next || next == h) return h;
h = next;
}
}
```
### 6.3 Posting mouse messages
Forward from the magnifier's WndProc. Maintain a small state machine per interaction
(idle → hover → drag) so capture semantics behave.
```cpp
void ForwardMouse(UINT msg, WPARAM keys, POINT magPt)
{
POINT src = MapToSource(magPt, ClientSize(), m_capture.ContentSize());
HWND target = m_dragTarget ? m_dragTarget // during drag, keep target stable
: HitTestChild(m_sourceRoot, src);
POINT cp = src;
MapWindowPoints(m_sourceRoot, target, &cp, 1);
LPARAM lp = MAKELPARAM(cp.x, cp.y);
switch (msg) {
case WM_LBUTTONDOWN: case WM_RBUTTONDOWN: case WM_MBUTTONDOWN:
m_dragTarget = target;
g_virtualCursor.BeginInteraction(m_sourceRoot, src, msg); // Section 7
break;
case WM_LBUTTONUP: case WM_RBUTTONUP: case WM_MBUTTONUP:
g_virtualCursor.EndInteraction(msg);
break;
}
PostMessage(target, msg, keys, lp);
if (msg == WM_LBUTTONUP || msg == WM_RBUTTONUP || msg == WM_MBUTTONUP)
m_dragTarget = nullptr;
}
```
Details that matter:
- **`WM_MOUSEMOVE` cadence:** forward every move; also keep the virtual cursor position
updated *before* posting, because many GUIs read `GetCursorPos()` in the handler rather
than trusting `lParam`.
- **`WM_MOUSEWHEEL` / `WM_MOUSEHWHEEL`:** lParam is **screen** coordinates. Compute the screen
point as `cloakedWindowScreenOrigin + src` (the cloaked window has a real screen rect —
use it). Post to the hit-test target (plugins don't rely on the focus-window wheel routing
when messages are posted directly).
- **Double clicks:** synthesize `WM_LBUTTONDBLCLK` yourself when two downs land within
`GetDoubleClickTime()` and a small radius — posted messages don't get the system's
double-click conversion (that happens at the queue level for real input).
- **Fine-adjust drags (Ctrl/Shift-drag on knobs):** pass through modifier state in `wParam`
(`MK_CONTROL|MK_SHIFT`) *and* make sure your `GetKeyState` hook (Section 7) reports them,
since JUCE reads `GetKeyState(VK_CONTROL)` rather than wParam.
- **Keyboard:** on click, call `SetFocus` on the target (you're in the same process — legal
across threads only with `AttachThreadInput(magThread, pluginGuiThread, TRUE)` bracketing,
or marshal the `SetFocus` call onto the plugin GUI thread via `SendMessage` to a helper).
Then forward `WM_KEYDOWN/CHAR/KEYUP`. Text fields in plugin GUIs then behave normally.
- **Cursor shape:** the plugin calls `SetCursor` on its thread; mirror it by hooking
`SetCursor` (optional, cosmetic) or by handling `WM_SETCURSOR` in the magnifier with a
forwarded `WM_SETCURSOR` to the source and reusing the resulting cursor handle.
---
## 7. Cursor/Key State Virtualization (the reliability core)
Naive message injection breaks on knob drags because GUIs poll:
- `GetCursorPos` / `GetPhysicalCursorPos` / `GetCursorInfo` / `GetMessagePos`
- `GetAsyncKeyState(VK_LBUTTON)` / `GetKeyState(VK_LBUTTON | VK_CONTROL | VK_SHIFT)`
- `WindowFromPoint` (hit tests during drag)
- `SetCursorPos` (some GUIs re-center the cursor for endless-rotary drags!)
- `ClipCursor` (drag confinement)
Install **inline hooks (MinHook)** for these in the offload process. The hooks are pass-through
unless a magnified interaction is active, then they report the virtual state:
```cpp
struct VirtualCursorState
{
std::atomic<bool> active{false};
std::atomic<DWORD> guiThreadId{0}; // only lie to the plugin GUI thread
POINT screenPos; // virtual cursor in SCREEN coords
SHORT lButton, rButton, mButton; // 0x8000 pressed
SHORT ctrl, shift, alt;
std::mutex m;
};
VirtualCursorState g_vc;
BOOL WINAPI Hook_GetCursorPos(LPPOINT p)
{
if (g_vc.active.load(std::memory_order_acquire) &&
GetCurrentThreadId() == g_vc.guiThreadId.load())
{
std::scoped_lock lk(g_vc.m);
*p = g_vc.screenPos;
return TRUE;
}
return Real_GetCursorPos(p);
}
SHORT WINAPI Hook_GetAsyncKeyState(int vk)
{
if (g_vc.active.load(std::memory_order_acquire) &&
GetCurrentThreadId() == g_vc.guiThreadId.load())
{
std::scoped_lock lk(g_vc.m);
switch (vk) {
case VK_LBUTTON: return g_vc.lButton;
case VK_RBUTTON: return g_vc.rButton;
case VK_MBUTTON: return g_vc.mButton;
case VK_CONTROL: return g_vc.ctrl;
case VK_SHIFT: return g_vc.shift;
case VK_MENU: return g_vc.alt;
}
}
return Real_GetAsyncKeyState(vk);
}
// Same pattern for GetKeyState, GetCursorInfo, GetMessagePos, GetPhysicalCursorPos.
HWND WINAPI Hook_WindowFromPoint(POINT p)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId) {
// If the virtual point is inside the cloaked window's rect, resolve within it:
RECT r; GetWindowRect(g_vc_sourceRoot, &r);
if (PtInRect(&r, p)) {
POINT c = p; ScreenToClient(g_vc_sourceRoot, &c);
return HitTestChild(g_vc_sourceRoot, c);
}
}
return Real_WindowFromPoint(p);
}
BOOL WINAPI Hook_SetCursorPos(int x, int y)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId) {
// Endless-knob recenter: update virtual pos, tell the magnifier to
// warp ITS cursor correspondingly (post a message to the magnifier).
std::scoped_lock lk(g_vc.m);
g_vc.screenPos = {x, y};
NotifyMagnifierCursorWarp(x, y);
return TRUE; // swallow: never move the real cursor
}
return Real_SetCursorPos(x, y);
}
BOOL WINAPI Hook_ClipCursor(const RECT* r)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId)
return TRUE; // swallow; optionally clip magnifier cursor instead
return Real_ClipCursor(r);
}
```
Lifecycle, driven by the magnifier (Section 6.3):
- `BeginInteraction(root, srcClientPt, buttonMsg)` → set `guiThreadId =
GetWindowThreadProcessId(root)`, compute virtual **screen** pos =
cloaked-window-origin + srcClientPt, set button state, `active = true`.
- Every forwarded `WM_MOUSEMOVE` updates `screenPos` first, then posts the message.
- `EndInteraction` → clear button, and after the `WM_?BUTTONUP` is processed set
`active = false` (a short deferred clear — e.g. clear on the *next* mouse-move or after
50 ms — avoids a race where the GUI polls right after up).
- **Hover-only virtualization:** also activate (buttons unpressed) while the magnifier cursor
is over the view, so tooltips and hover highlights that poll `GetCursorPos` work. Keep it
thread-scoped to the plugin GUI thread so the rest of the process is unaffected.
Why thread-scoped: your own bridge UI, other plugins' GUI threads (if they share a thread you
must scope per-interaction instead — see 9), and any host machinery keep seeing the real cursor.
`SetCapture`: posted button messages do **not** set the system capture, but that's fine —
your magnifier holds the *real* capture (`SetCapture` on mouse-down in the magnifier), so you
keep receiving moves outside your window and keep forwarding them; the plugin sees a consistent
stream plus consistent poll results. Verify drag-outside-window on: JUCE sliders, VSTGUI knobs,
iPlug2 controls, and one custom-framework plugin (u-he or FabFilter style).
---
## 8. Popup Handling (menus, dropdowns, tooltips)
Native popups (`TrackPopupMenu` menus of class `#32768`, ComboLBox, tooltips_class32, and
custom borderless popup HWNDs) are separate top-level windows positioned in screen coordinates
near the *cloaked* window. Strategy: **detect → reposition over the magnifier (unscaled) →
let them run natively.** This is the minimum viable, fully functional answer; scaled popup
capture can come later.
```cpp
void CALLBACK OnWinEvent(HWINEVENTHOOK, DWORD event, HWND hwnd,
LONG idObject, LONG, DWORD, DWORD)
{
if (event != EVENT_OBJECT_SHOW || idObject != OBJID_WINDOW) return;
if (GetAncestor(hwnd, GA_ROOT) != hwnd) return; // top-level only
DWORD tid = GetWindowThreadProcessId(hwnd, nullptr);
PluginInstance* plug = g_registry.FindByGuiThread(tid); // attribute by thread
if (!plug) {
// fallback: owner-window attribution
HWND owner = GetWindow(hwnd, GW_OWNER);
plug = owner ? g_registry.FindByRootWindow(GetAncestor(owner, GA_ROOT)) : nullptr;
}
if (!plug || hwnd == plug->cloakedRoot || g_registry.IsOurWindow(hwnd)) return;
// Transient popup heuristics: WS_POPUP, not WS_EX_APPWINDOW, small-ish, class name
LONG style = GetWindowLong(hwnd, GWL_STYLE);
if (!(style & WS_POPUP)) return;
RECT pr; GetWindowRect(hwnd, &pr);
RECT src; GetWindowRect(plug->cloakedRoot, &src);
// Position relative to the cloaked window → map to magnifier space (scaled offset)
double sx = plug->magnifier->Scale();
POINT magOrigin = plug->magnifier->ScreenOrigin();
int nx = magOrigin.x + (int)((pr.left - src.left) * sx);
int ny = magOrigin.y + (int)((pr.top - src.top ) * sx);
SetWindowPos(hwnd, HWND_TOPMOST, nx, ny, 0, 0,
SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOZORDER);
// Do NOT cloak popups — the user interacts with them directly, with the REAL cursor.
}
```
Install with `SetWinEventHook(EVENT_OBJECT_SHOW, EVENT_OBJECT_SHOW, nullptr, OnWinEvent,
GetCurrentProcessId(), 0, WINEVENT_OUTOFCONTEXT)`.
Interaction rules during a native popup:
- The popup runs a **modal tracking loop** reading the *real* cursor — so while a repositioned
popup is up, **deactivate** cursor virtualization (`g_vc.active = false`). The user clicks
the real popup with the real mouse; everything is native. Reactivate on
`EVENT_OBJECT_HIDE`/`EVENT_OBJECT_DESTROY` of that popup.
- Edge case — the *opening click position*: `TrackPopupMenu` is usually called with
coordinates from `GetCursorPos()`, which your hook virtualized to a point over the cloaked
window; your reposition then moves the menu to the right place. If a plugin instead uses the
message lParam or its own math, the reposition still catches it. Either way the WinEvent
fires after the window is shown but typically before the first paint is visible; slight
flicker is possible and acceptable.
- Tooltips: same mechanism; optionally filter class `tooltips_class32` to reposition without
the virtualization toggling (tooltips have no modal loop).
**Phase-2 option (cosmetic):** capture popups with their own WGC items and composite them
scaled into the magnifier. Only attempt after the reposition path is solid; the modal-loop +
virtual-cursor interplay is the hardest test surface in the project.
---
## 9. Multi-Plugin Bookkeeping
- `Registry`: `cloakedRoot HWND ↔ PluginInstance ↔ guiThreadId ↔ MagnifierWindow`.
Populate at editor-open, clear at editor-close.
- **Prefer one GUI thread per hosted plugin** in the offload process. It makes thread-scoped
cursor virtualization and popup attribution exact. If plugins share a GUI thread, scope the
virtualization to *(thread AND active-interaction plugin)* and accept that two simultaneous
drags on different plugins on the same thread are impossible anyway (one mouse).
- All cloaked roots may share the same desktop position; capture is per-window and unaffected
by overlap or z-order.
- N plugins = N frame pools = N small GPU copies per dirty frame. Linear, modest. Pause
sessions whose magnifier is hidden.
---
## 10. Fallback Path (no WGC)
For pre-2004 Windows 10, poll on a low-priority thread at ≤ 30 fps:
```cpp
HDC hdc = GetDC(nullptr); // memory DC + DIB section sized to window
PrintWindow(hwnd, memDC, PW_RENDERFULLCONTENT); // flag 0x2 — routes through DWM,
// captures GPU-rendered content
```
Upload the DIB to a D3D11 dynamic texture and reuse the same scaler/injector. This path has
CPU cost (readback-style copy) — keep the rate low and treat WGC as the primary path.
---
## 11. Integration Points in the Bridge
- **Editor open:** create cloaked host window → attach plugin view (VST3
`IPlugView::attached`) → register in Registry → start CaptureSession → create
MagnifierWindow at `content * savedZoom`.
- **VST3 resize:** in your `IPlugFrame::resizeView`, resize the cloaked host window, call
`CaptureSession::NotifySourceResized()`, resize magnifier.
- **Native scaling first:** before enabling the magnifier for a plugin, try
`IPlugViewContentScaleSupport::setContentScaleFactor(zoom)`; if the view accepts and
resizes, you don't need capture-scaling for that plugin (vector-crisp beats bitmap-scaled).
Keep a per-plugin override in settings ("Force bitmap zoom").
- **Settings:** zoom factor (1.25/1.5/2.0/custom), filter (nearest/bilinear/bicubic/sharpen),
fps cap, per-plugin native-vs-bitmap preference.
- **Teardown order:** stop session → revoke FrameArrived → close frame pool → uncloak (or
just destroy) host window → unregister. Hooks stay installed for the process lifetime
(they're pass-through when inactive).
---
## 12. Test Matrix (budget real time here)
| Area | Cases |
|---|---|
| Frameworks | JUCE (single HWND), VSTGUI, iPlug2, custom (u-he, FabFilter, Eventide) |
| Drag | knob fine-drag with Ctrl/Shift, drag past magnifier edge, endless-rotary (`SetCursorPos` recenter), `ClipCursor` users |
| Wheel | over knobs, over scrollable preset lists (screen-coord lParam) |
| Popups | right-click context menus, combo dropdowns, tooltip hover, preset browser popups |
| Resize | expand/collapse panels, plugin-initiated resize, user zoom change mid-drag |
| Text | double-click value entry, type + Enter/Escape, focus loss |
| Perf | meter-heavy plugin @ 30/60 fps cap, 4 plugins magnified simultaneously, audio headroom before/after at 64-sample buffers |
| Lifecycle | open/close editor 100×, cloak/uncloak, monitor DPI change, magnifier minimized |
| Repaint-while-cloaked | verify each framework keeps painting; add `RedrawWindow` nudge where needed |
---
## 13. Known Sharp Edges (honest list)
1. **Modal popup loops + virtual cursor** — the interaction in Section 8 (deactivate during
popup, reactivate after) must be airtight or menus mis-track. This is the #1 debugging sink.
2. **Double-click synthesis** — easy to forget; value-entry on knobs silently breaks without it.
3. **Plugins that throttle painting when cloaked** — per-plugin nudge list.
4. **`GetMessagePos`** — returns queue-time cursor pos; JUCE uses it in places. Hook it or
accept rare off-by-a-frame hover artifacts.
5. **Per-Monitor DPI** — keep the cloaked window on a known monitor; if the magnifier moves to
a different-DPI monitor, only your scale factor changes, but re-check wheel screen-coord math.
6. **`IsBorderRequired` availability** — cosmetic only (border is on the cloaked window), but
verify no border artifacts leak into captured frames on Win10 (they don't — the border is a
compositor adornment, excluded from `CreateForWindow` content).
7. **Touch/pen** — out of scope above; `WM_POINTER` forwarding is a separate (similar) effort.
8. **Focus stealing** — `AttachThreadInput` bracketing for `SetFocus`; detach immediately after.
---
## 14. Phased Implementation Plan
1. **Phase 0 (1–2 days):** WGC capture of a cloaked JUCE plugin → scaled display, no input.
Validates capture-while-cloaked + repaint behavior + perf on your machines.
2. **Phase 1:** mouse forwarding without hooks. Clicks/hover work; drags glitch on polling
GUIs — expected. Establishes coordinate mapping + hit-testing.
3. **Phase 2:** MinHook virtualization (`GetCursorPos`, `GetAsyncKeyState`, `GetKeyState`,
`SetCursorPos`, `ClipCursor`, `WindowFromPoint`, `GetMessagePos`). Drags become solid.
Run the drag test matrix.
4. **Phase 3:** PopupManager (reposition strategy) + virtualization pause/resume around popups.
5. **Phase 4:** keyboard/focus, double-click synthesis, cursor-shape mirroring, wheel polish.
6. **Phase 5:** multi-plugin registry, session pause on hidden magnifier, settings UI,
`PrintWindow` fallback, native `setContentScaleFactor` preference path.
### Windows.Graphics.Capture + DWM Cloak + In-Process Input Injection
Implementation guide for adding interactive, arbitrarily-scaled rendering of hosted plugin GUIs
inside a bridge/offload host (e.g. GrandBridge FX offload process), without moving the real
window off-screen and without touching the audio path.
**Target:** Windows 10 2004+ / Windows 11, x64, C++17+, D3D11.
**Scope:** the hosted plugin's editor HWND lives in *your* offload process — this guide relies on that.
---
## 1. Architecture Overview
```
┌──────────────────────────── Offload process ────────────────────────────┐
│ │
│ Plugin GUI thread Magnifier render thread │
│ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ Real plugin HWND │── DWM compose ─▶│ WGC FramePool (D3D11) │ │
│ │ (CLOAKED) │ │ → scale shader → present│ │
│ │ child HWND tree │◀─ posted msgs ──│ MagnifierWindow (visible)│ │
│ └─────────────────┘ └──────────────────────────┘ │
│ ▲ ▲ │
│ │ └── IAT/inline hooks: GetCursorPos, GetAsyncKeyState, ... │
│ └────── WinEvent hook: popup detection / reposition │
│ │
│ Audio threads (realtime prio) ──── never touched by any of the above │
└──────────────────────────────────────────────────────────────────────────┘
```
Components:
1. **CaptureSession** — one per hosted plugin editor. WGC item + frame pool + D3D11 texture.
2. **MagnifierWindow** — the visible window the user interacts with. Owns a swapchain, draws
the latest captured frame scaled, translates and forwards input.
3. **CursorStateVirtualizer** — process-wide hooks that lie about cursor position / button state
to the plugin GUI thread while a magnified interaction is in progress.
4. **PopupManager** — WinEvent hook that catches transient top-level windows (menus, dropdowns,
tooltips) created by plugin GUI threads and repositions (or captures) them.
5. **Registry** — HWND → PluginInstance mapping so input routing and popup attribution never guess.
---
## 2. Prerequisites & Project Setup
- C++/WinRT (header-only): `winrt/Windows.Graphics.Capture.h`,
`winrt/Windows.Graphics.DirectX.Direct3D11.h`
- Link: `d3d11.lib`, `dxgi.lib`, `dwmapi.lib`, `windowsapp.lib` (or `RuntimeObject.lib` for older setups)
- Interop header: `windows.graphics.capture.interop.h` (for `IGraphicsCaptureItemInterop::CreateForWindow`)
- Direct3D interop: `windows.graphics.directx.direct3d11.interop.h` (`CreateDirect3D11DeviceFromDXGIDevice`)
- Hooking library: **MinHook** (small, license-friendly) or Detours.
- Call `winrt::init_apartment(winrt::apartment_type::single_threaded)` on the thread that creates
the frame pool if you use `Create` (DispatcherQueue required), or use
**`CreateFreeThreaded`** (recommended here — no dispatcher, callbacks arrive on a pool thread).
Feature detection at startup:
```cpp
bool wgcAvailable = winrt::Windows::Graphics::Capture::GraphicsCaptureSession::IsSupported();
bool canHideBorder = winrt::Windows::Foundation::Metadata::ApiInformation::IsPropertyPresent(
L"Windows.Graphics.Capture.GraphicsCaptureSession", L"IsBorderRequired"); // Win10 2104+/Win11
```
If `wgcAvailable == false` (very old Win10), fall back to a `PrintWindow(hwnd, dc,
PW_RENDERFULLCONTENT)` polling path (Section 10).
> **Note on `IsBorderRequired`:** setting it to `false` requires the
> `GraphicsCaptureAccess::RequestAccessAsync(GraphicsCaptureAccessKind::Borderless)` consent on
> some builds for packaged apps; for a normal Win32 process it generally just works on Win11.
> Test on your minimum supported OS. The yellow border only matters cosmetically — worst case,
> the *cloaked* window has a border nobody sees.
---
## 3. Cloaking the Real Editor Window
Never move the window off-screen and never `SW_HIDE` it (hidden windows stop being composed
and WGC delivers nothing). Cloak it instead — DWM keeps composing a cloaked window, but it is
invisible on the desktop and excluded from Alt-Tab thumbnails.
```cpp
#include <dwmapi.h>
void SetCloaked(HWND hwnd, bool cloak)
{
BOOL v = cloak ? TRUE : FALSE;
DwmSetWindowAttribute(hwnd, DWMWA_CLOAK, &v, sizeof(v));
}
```
Rules:
- Cloak the **top-level** window that hosts the plugin editor. If your bridge parents the plugin
editor into a child HWND of a bridge-owned top-level window, cloak that top-level window and
capture it (WGC captures top-level windows; child HWNDs are captured as part of their root).
- Keep the window **restored** (not minimized) and at a fixed, known desktop position. Its real
screen rect is your coordinate anchor for popups and wheel messages. A convenient trick:
park all cloaked editors at the same location, e.g. the top-left of the primary monitor —
overlap is irrelevant since nothing is visible and WGC capture is per-window.
- Some plugin GUIs throttle repaint timers when they believe they are occluded/minimized.
Cloaked ≠ minimized, so most keep painting. If one goes lazy: periodic
`RedrawWindow(hwnd, nullptr, nullptr, RDW_INVALIDATE | RDW_ALLCHILDREN)` at your capture
cadence, or keep its animation timer alive (you own the process). Handle this per-plugin,
only when observed.
---
## 4. Capture Session (WGC → D3D11 texture)
One `CaptureSession` per hosted editor. Everything stays on the GPU.
```cpp
#include <winrt/Windows.Graphics.Capture.h>
#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
#include <windows.graphics.capture.interop.h>
#include <windows.graphics.directx.direct3d11.interop.h>
using namespace winrt;
using namespace winrt::Windows::Graphics;
using namespace winrt::Windows::Graphics::Capture;
using namespace winrt::Windows::Graphics:
irectX;class CaptureSession
{
public:
// d3dDevice: create ONE shared D3D11 device for all capture sessions + magnifier rendering.
// Use D3D11_CREATE_DEVICE_BGRA_SUPPORT.
void Start(HWND source, ID3D11Device* d3dDevice)
{
m_source = source;
// Wrap the D3D device for WinRT
com_ptr<IDXGIDevice> dxgi;
d3dDevice->QueryInterface(IID_PPV_ARGS(dxgi.put()));
com_ptr<::IInspectable> inspectable;
check_hresult(CreateDirect3D11DeviceFromDXGIDevice(dxgi.get(), inspectable.put()));
m_winrtDevice = inspectable.as<DirectX:
irect3D11::IDirect3DDevice>();// Create the capture item for the (cloaked) window
auto interop = get_activation_factory<GraphicsCaptureItem,
IGraphicsCaptureItemInterop>();
check_hresult(interop->CreateForWindow(
source, guid_of<GraphicsCaptureItem>(),
put_abi(m_item)));
m_lastSize = m_item.Size();
// Free-threaded pool: no DispatcherQueue needed; 2 buffers is enough.
m_framePool = Direct3D11CaptureFramePool::CreateFreeThreaded(
m_winrtDevice,
DirectXPixelFormat::B8G8R8A8UIntNormalized,
/*numberOfBuffers*/ 2,
m_lastSize);
m_frameArrived = m_framePool.FrameArrived(auto_revoke,
{ this, &CaptureSession::OnFrameArrived });
m_session = m_framePool.CreateCaptureSession(m_item);
m_session.IsCursorCaptureEnabled(false); // we draw our own cursor if needed
TrySetBorderless(m_session); // IsBorderRequired(false) if available
m_session.StartCapture();
}
// Returns the most recent frame texture (thread-safe swap). May be null before first frame.
com_ptr<ID3D11Texture2D> LatestFrame()
{
std::scoped_lock lk(m_mutex);
return m_latest;
}
SizeInt32 ContentSize() const { return m_lastSize; }
// Call when the plugin resizes its editor (WM_WINDOWPOSCHANGED / IPlugFrame::resizeView)
void NotifySourceResized()
{
m_pendingRecreate = true; // handled on next FrameArrived
}
private:
void OnFrameArrived(Direct3D11CaptureFramePool const& pool,
winrt::Windows::Foundation::IInspectable const&)
{
auto frame = pool.TryGetNextFrame();
if (!frame) return;
auto size = frame.ContentSize();
if (m_pendingRecreate ||
size.Width != m_lastSize.Width || size.Height != m_lastSize.Height)
{
m_lastSize = size;
m_pendingRecreate = false;
pool.Recreate(m_winrtDevice,
DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, size);
}
// Extract the D3D11 texture and keep a reference; DO NOT copy on CPU.
auto surface = frame.Surface();
com_ptr<ID3D11Texture2D> tex;
auto access = surface.as<::Windows::Graphics:
irectX:irect3D11::IDirect3DDxgiInterfaceAccess>();check_hresult(access->GetInterface(IID_PPV_ARGS(tex.put())));
// Copy into our own texture (frame buffer is recycled by the pool).
// One GPU CopyResource per dirty frame — cheap.
com_ptr<ID3D11Texture2D> own = GetOrCreateStagingSizedTexture(tex.get(), size);
{
com_ptr<ID3D11DeviceContext> ctx; // use a deferred/immediate ctx guarded by a lock,
m_device->GetImmediateContext(ctx.put()); // or D3D11_CREATE_DEVICE flags + mutex
std::scoped_lock lk(m_ctxMutex);
ctx->CopyResource(own.get(), tex.get());
}
{
std::scoped_lock lk(m_mutex);
m_latest = own;
}
m_dirty.store(true, std::memory_order_release); // wake/flag the magnifier render
}
// ... members: m_item, m_framePool, m_session, m_winrtDevice, m_device,
// m_latest, m_lastSize, mutexes, m_dirty, m_pendingRecreate ...
};
```
Key decisions in this design:
- **`CreateFreeThreaded`** — callbacks arrive on a WinRT pool thread, never on the plugin GUI
thread and never on your audio threads. No DispatcherQueue plumbing.
- **2 buffers** — minimal latency and memory; you are mirroring, not recording.
- **Event-driven** — WGC only delivers frames when the source repaints. A static GUI costs ~0.
- **One shared D3D11 device** for all sessions and the renderer, so textures are directly usable
without cross-device copies. Guard the immediate context with a mutex (or use
`ID3D10Multithread::SetMultithreadProtected`).
---
## 5. Magnifier Window & Scaled Rendering
The magnifier window is a plain top-level window (or a child view inside your bridge editor UI)
with a DXGI flip-model swapchain.
Render loop (on a dedicated render thread, or driven by a `WM_TIMER`/composition tick):
1. Wait until `m_dirty` or a resize/interaction event; **cap at 30–60 fps** with a waitable
timer. Do not free-run.
2. `LatestFrame()` → draw a fullscreen quad sampling the texture with your scale filter:
- integer factors (2x, 3x): point/nearest for pixel-crisp look, or bilinear for smooth;
- non-integer (1.5x like your screenshot): bicubic (Catmull-Rom) in a pixel shader, or
bilinear + a light sharpen (FSR1/RCAS-style) — subjectively best for knob labels.
3. Present with `DXGI_SWAP_EFFECT_FLIP_DISCARD`, `Present(1, 0)` (vsync keeps GPU polite).
Sizing model:
- `scale = magnifierClientSize / captureContentSize` (keep aspect; letterbox or lock the
magnifier window's aspect via `WM_SIZING`).
- When the plugin resizes itself (expand/collapse panels), you get the new `ContentSize`
from the frame — resize the magnifier window to `content * userZoom` and continue. Handle
`WM_WINDOWPOSCHANGED` on the cloaked window too (subclass it) and call
`NotifySourceResized()` for immediate response.
Performance guardrails (audio-safety checklist):
- All scaling on GPU. Zero CPU pixel work. No `Map`/readback anywhere in the hot path.
- Render thread priority: `THREAD_PRIORITY_NORMAL` or `BELOW_NORMAL`. Never raise it.
- Frame-rate cap ≤ 60, default 30. Expose in settings.
- On `WM_SHOWWINDOW(FALSE)` / magnifier minimized: **pause the session**
(`m_frameArrived.revoke()` or stop presenting) — don't burn GPU for invisible output.
- Test with a meter-heavy plugin and measure: GPU % (Task Manager per-process), and your
audio callback headroom before/after. Expected impact: comparable to one extra animated
plugin GUI.
---
## 6. Input Injection
### 6.1 Coordinate mapping
```cpp
// magnifier client point → real (cloaked) window client point
POINT MapToSource(POINT magPt, SIZE magClient, SizeInt32 src)
{
POINT p;
p.x = MulDiv(magPt.x, src.Width, magClient.cx);
p.y = MulDiv(magPt.y, src.Height, magClient.cy);
return p; // client coords of the cloaked top-level window
}
```
### 6.2 Which HWND receives the message
Plugin GUIs are often HWND trees (JUCE = usually one HWND; VSTGUI/iPlug/custom = sometimes
children). Resolve the deepest child at the point:
```cpp
HWND HitTestChild(HWND root, POINT clientPt)
{
// ChildWindowFromPointEx walks one level; RealChildWindowFromPoint or a manual
// recursive descent handles nesting:
HWND h = root;
for (;
{POINT p = clientPt;
MapWindowPoints(root, h, &p, 1);
HWND next = RealChildWindowFromPoint(h, p);
if (!next || next == h) return h;
h = next;
}
}
```
### 6.3 Posting mouse messages
Forward from the magnifier's WndProc. Maintain a small state machine per interaction
(idle → hover → drag) so capture semantics behave.
```cpp
void ForwardMouse(UINT msg, WPARAM keys, POINT magPt)
{
POINT src = MapToSource(magPt, ClientSize(), m_capture.ContentSize());
HWND target = m_dragTarget ? m_dragTarget // during drag, keep target stable
: HitTestChild(m_sourceRoot, src);
POINT cp = src;
MapWindowPoints(m_sourceRoot, target, &cp, 1);
LPARAM lp = MAKELPARAM(cp.x, cp.y);
switch (msg) {
case WM_LBUTTONDOWN: case WM_RBUTTONDOWN: case WM_MBUTTONDOWN:
m_dragTarget = target;
g_virtualCursor.BeginInteraction(m_sourceRoot, src, msg); // Section 7
break;
case WM_LBUTTONUP: case WM_RBUTTONUP: case WM_MBUTTONUP:
g_virtualCursor.EndInteraction(msg);
break;
}
PostMessage(target, msg, keys, lp);
if (msg == WM_LBUTTONUP || msg == WM_RBUTTONUP || msg == WM_MBUTTONUP)
m_dragTarget = nullptr;
}
```
Details that matter:
- **`WM_MOUSEMOVE` cadence:** forward every move; also keep the virtual cursor position
updated *before* posting, because many GUIs read `GetCursorPos()` in the handler rather
than trusting `lParam`.
- **`WM_MOUSEWHEEL` / `WM_MOUSEHWHEEL`:** lParam is **screen** coordinates. Compute the screen
point as `cloakedWindowScreenOrigin + src` (the cloaked window has a real screen rect —
use it). Post to the hit-test target (plugins don't rely on the focus-window wheel routing
when messages are posted directly).
- **Double clicks:** synthesize `WM_LBUTTONDBLCLK` yourself when two downs land within
`GetDoubleClickTime()` and a small radius — posted messages don't get the system's
double-click conversion (that happens at the queue level for real input).
- **Fine-adjust drags (Ctrl/Shift-drag on knobs):** pass through modifier state in `wParam`
(`MK_CONTROL|MK_SHIFT`) *and* make sure your `GetKeyState` hook (Section 7) reports them,
since JUCE reads `GetKeyState(VK_CONTROL)` rather than wParam.
- **Keyboard:** on click, call `SetFocus` on the target (you're in the same process — legal
across threads only with `AttachThreadInput(magThread, pluginGuiThread, TRUE)` bracketing,
or marshal the `SetFocus` call onto the plugin GUI thread via `SendMessage` to a helper).
Then forward `WM_KEYDOWN/CHAR/KEYUP`. Text fields in plugin GUIs then behave normally.
- **Cursor shape:** the plugin calls `SetCursor` on its thread; mirror it by hooking
`SetCursor` (optional, cosmetic) or by handling `WM_SETCURSOR` in the magnifier with a
forwarded `WM_SETCURSOR` to the source and reusing the resulting cursor handle.
---
## 7. Cursor/Key State Virtualization (the reliability core)
Naive message injection breaks on knob drags because GUIs poll:
- `GetCursorPos` / `GetPhysicalCursorPos` / `GetCursorInfo` / `GetMessagePos`
- `GetAsyncKeyState(VK_LBUTTON)` / `GetKeyState(VK_LBUTTON | VK_CONTROL | VK_SHIFT)`
- `WindowFromPoint` (hit tests during drag)
- `SetCursorPos` (some GUIs re-center the cursor for endless-rotary drags!)
- `ClipCursor` (drag confinement)
Install **inline hooks (MinHook)** for these in the offload process. The hooks are pass-through
unless a magnified interaction is active, then they report the virtual state:
```cpp
struct VirtualCursorState
{
std::atomic<bool> active{false};
std::atomic<DWORD> guiThreadId{0}; // only lie to the plugin GUI thread
POINT screenPos; // virtual cursor in SCREEN coords
SHORT lButton, rButton, mButton; // 0x8000 pressed
SHORT ctrl, shift, alt;
std::mutex m;
};
VirtualCursorState g_vc;
BOOL WINAPI Hook_GetCursorPos(LPPOINT p)
{
if (g_vc.active.load(std::memory_order_acquire) &&
GetCurrentThreadId() == g_vc.guiThreadId.load())
{
std::scoped_lock lk(g_vc.m);
*p = g_vc.screenPos;
return TRUE;
}
return Real_GetCursorPos(p);
}
SHORT WINAPI Hook_GetAsyncKeyState(int vk)
{
if (g_vc.active.load(std::memory_order_acquire) &&
GetCurrentThreadId() == g_vc.guiThreadId.load())
{
std::scoped_lock lk(g_vc.m);
switch (vk) {
case VK_LBUTTON: return g_vc.lButton;
case VK_RBUTTON: return g_vc.rButton;
case VK_MBUTTON: return g_vc.mButton;
case VK_CONTROL: return g_vc.ctrl;
case VK_SHIFT: return g_vc.shift;
case VK_MENU: return g_vc.alt;
}
}
return Real_GetAsyncKeyState(vk);
}
// Same pattern for GetKeyState, GetCursorInfo, GetMessagePos, GetPhysicalCursorPos.
HWND WINAPI Hook_WindowFromPoint(POINT p)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId) {
// If the virtual point is inside the cloaked window's rect, resolve within it:
RECT r; GetWindowRect(g_vc_sourceRoot, &r);
if (PtInRect(&r, p)) {
POINT c = p; ScreenToClient(g_vc_sourceRoot, &c);
return HitTestChild(g_vc_sourceRoot, c);
}
}
return Real_WindowFromPoint(p);
}
BOOL WINAPI Hook_SetCursorPos(int x, int y)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId) {
// Endless-knob recenter: update virtual pos, tell the magnifier to
// warp ITS cursor correspondingly (post a message to the magnifier).
std::scoped_lock lk(g_vc.m);
g_vc.screenPos = {x, y};
NotifyMagnifierCursorWarp(x, y);
return TRUE; // swallow: never move the real cursor
}
return Real_SetCursorPos(x, y);
}
BOOL WINAPI Hook_ClipCursor(const RECT* r)
{
if (g_vc.active && GetCurrentThreadId() == g_vc.guiThreadId)
return TRUE; // swallow; optionally clip magnifier cursor instead
return Real_ClipCursor(r);
}
```
Lifecycle, driven by the magnifier (Section 6.3):
- `BeginInteraction(root, srcClientPt, buttonMsg)` → set `guiThreadId =
GetWindowThreadProcessId(root)`, compute virtual **screen** pos =
cloaked-window-origin + srcClientPt, set button state, `active = true`.
- Every forwarded `WM_MOUSEMOVE` updates `screenPos` first, then posts the message.
- `EndInteraction` → clear button, and after the `WM_?BUTTONUP` is processed set
`active = false` (a short deferred clear — e.g. clear on the *next* mouse-move or after
50 ms — avoids a race where the GUI polls right after up).
- **Hover-only virtualization:** also activate (buttons unpressed) while the magnifier cursor
is over the view, so tooltips and hover highlights that poll `GetCursorPos` work. Keep it
thread-scoped to the plugin GUI thread so the rest of the process is unaffected.
Why thread-scoped: your own bridge UI, other plugins' GUI threads (if they share a thread you
must scope per-interaction instead — see 9), and any host machinery keep seeing the real cursor.
`SetCapture`: posted button messages do **not** set the system capture, but that's fine —
your magnifier holds the *real* capture (`SetCapture` on mouse-down in the magnifier), so you
keep receiving moves outside your window and keep forwarding them; the plugin sees a consistent
stream plus consistent poll results. Verify drag-outside-window on: JUCE sliders, VSTGUI knobs,
iPlug2 controls, and one custom-framework plugin (u-he or FabFilter style).
---
## 8. Popup Handling (menus, dropdowns, tooltips)
Native popups (`TrackPopupMenu` menus of class `#32768`, ComboLBox, tooltips_class32, and
custom borderless popup HWNDs) are separate top-level windows positioned in screen coordinates
near the *cloaked* window. Strategy: **detect → reposition over the magnifier (unscaled) →
let them run natively.** This is the minimum viable, fully functional answer; scaled popup
capture can come later.
```cpp
void CALLBACK OnWinEvent(HWINEVENTHOOK, DWORD event, HWND hwnd,
LONG idObject, LONG, DWORD, DWORD)
{
if (event != EVENT_OBJECT_SHOW || idObject != OBJID_WINDOW) return;
if (GetAncestor(hwnd, GA_ROOT) != hwnd) return; // top-level only
DWORD tid = GetWindowThreadProcessId(hwnd, nullptr);
PluginInstance* plug = g_registry.FindByGuiThread(tid); // attribute by thread
if (!plug) {
// fallback: owner-window attribution
HWND owner = GetWindow(hwnd, GW_OWNER);
plug = owner ? g_registry.FindByRootWindow(GetAncestor(owner, GA_ROOT)) : nullptr;
}
if (!plug || hwnd == plug->cloakedRoot || g_registry.IsOurWindow(hwnd)) return;
// Transient popup heuristics: WS_POPUP, not WS_EX_APPWINDOW, small-ish, class name
LONG style = GetWindowLong(hwnd, GWL_STYLE);
if (!(style & WS_POPUP)) return;
RECT pr; GetWindowRect(hwnd, &pr);
RECT src; GetWindowRect(plug->cloakedRoot, &src);
// Position relative to the cloaked window → map to magnifier space (scaled offset)
double sx = plug->magnifier->Scale();
POINT magOrigin = plug->magnifier->ScreenOrigin();
int nx = magOrigin.x + (int)((pr.left - src.left) * sx);
int ny = magOrigin.y + (int)((pr.top - src.top ) * sx);
SetWindowPos(hwnd, HWND_TOPMOST, nx, ny, 0, 0,
SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOZORDER);
// Do NOT cloak popups — the user interacts with them directly, with the REAL cursor.
}
```
Install with `SetWinEventHook(EVENT_OBJECT_SHOW, EVENT_OBJECT_SHOW, nullptr, OnWinEvent,
GetCurrentProcessId(), 0, WINEVENT_OUTOFCONTEXT)`.
Interaction rules during a native popup:
- The popup runs a **modal tracking loop** reading the *real* cursor — so while a repositioned
popup is up, **deactivate** cursor virtualization (`g_vc.active = false`). The user clicks
the real popup with the real mouse; everything is native. Reactivate on
`EVENT_OBJECT_HIDE`/`EVENT_OBJECT_DESTROY` of that popup.
- Edge case — the *opening click position*: `TrackPopupMenu` is usually called with
coordinates from `GetCursorPos()`, which your hook virtualized to a point over the cloaked
window; your reposition then moves the menu to the right place. If a plugin instead uses the
message lParam or its own math, the reposition still catches it. Either way the WinEvent
fires after the window is shown but typically before the first paint is visible; slight
flicker is possible and acceptable.
- Tooltips: same mechanism; optionally filter class `tooltips_class32` to reposition without
the virtualization toggling (tooltips have no modal loop).
**Phase-2 option (cosmetic):** capture popups with their own WGC items and composite them
scaled into the magnifier. Only attempt after the reposition path is solid; the modal-loop +
virtual-cursor interplay is the hardest test surface in the project.
---
## 9. Multi-Plugin Bookkeeping
- `Registry`: `cloakedRoot HWND ↔ PluginInstance ↔ guiThreadId ↔ MagnifierWindow`.
Populate at editor-open, clear at editor-close.
- **Prefer one GUI thread per hosted plugin** in the offload process. It makes thread-scoped
cursor virtualization and popup attribution exact. If plugins share a GUI thread, scope the
virtualization to *(thread AND active-interaction plugin)* and accept that two simultaneous
drags on different plugins on the same thread are impossible anyway (one mouse).
- All cloaked roots may share the same desktop position; capture is per-window and unaffected
by overlap or z-order.
- N plugins = N frame pools = N small GPU copies per dirty frame. Linear, modest. Pause
sessions whose magnifier is hidden.
---
## 10. Fallback Path (no WGC)
For pre-2004 Windows 10, poll on a low-priority thread at ≤ 30 fps:
```cpp
HDC hdc = GetDC(nullptr); // memory DC + DIB section sized to window
PrintWindow(hwnd, memDC, PW_RENDERFULLCONTENT); // flag 0x2 — routes through DWM,
// captures GPU-rendered content
```
Upload the DIB to a D3D11 dynamic texture and reuse the same scaler/injector. This path has
CPU cost (readback-style copy) — keep the rate low and treat WGC as the primary path.
---
## 11. Integration Points in the Bridge
- **Editor open:** create cloaked host window → attach plugin view (VST3
`IPlugView::attached`) → register in Registry → start CaptureSession → create
MagnifierWindow at `content * savedZoom`.
- **VST3 resize:** in your `IPlugFrame::resizeView`, resize the cloaked host window, call
`CaptureSession::NotifySourceResized()`, resize magnifier.
- **Native scaling first:** before enabling the magnifier for a plugin, try
`IPlugViewContentScaleSupport::setContentScaleFactor(zoom)`; if the view accepts and
resizes, you don't need capture-scaling for that plugin (vector-crisp beats bitmap-scaled).
Keep a per-plugin override in settings ("Force bitmap zoom").
- **Settings:** zoom factor (1.25/1.5/2.0/custom), filter (nearest/bilinear/bicubic/sharpen),
fps cap, per-plugin native-vs-bitmap preference.
- **Teardown order:** stop session → revoke FrameArrived → close frame pool → uncloak (or
just destroy) host window → unregister. Hooks stay installed for the process lifetime
(they're pass-through when inactive).
---
## 12. Test Matrix (budget real time here)
| Area | Cases |
|---|---|
| Frameworks | JUCE (single HWND), VSTGUI, iPlug2, custom (u-he, FabFilter, Eventide) |
| Drag | knob fine-drag with Ctrl/Shift, drag past magnifier edge, endless-rotary (`SetCursorPos` recenter), `ClipCursor` users |
| Wheel | over knobs, over scrollable preset lists (screen-coord lParam) |
| Popups | right-click context menus, combo dropdowns, tooltip hover, preset browser popups |
| Resize | expand/collapse panels, plugin-initiated resize, user zoom change mid-drag |
| Text | double-click value entry, type + Enter/Escape, focus loss |
| Perf | meter-heavy plugin @ 30/60 fps cap, 4 plugins magnified simultaneously, audio headroom before/after at 64-sample buffers |
| Lifecycle | open/close editor 100×, cloak/uncloak, monitor DPI change, magnifier minimized |
| Repaint-while-cloaked | verify each framework keeps painting; add `RedrawWindow` nudge where needed |
---
## 13. Known Sharp Edges (honest list)
1. **Modal popup loops + virtual cursor** — the interaction in Section 8 (deactivate during
popup, reactivate after) must be airtight or menus mis-track. This is the #1 debugging sink.
2. **Double-click synthesis** — easy to forget; value-entry on knobs silently breaks without it.
3. **Plugins that throttle painting when cloaked** — per-plugin nudge list.
4. **`GetMessagePos`** — returns queue-time cursor pos; JUCE uses it in places. Hook it or
accept rare off-by-a-frame hover artifacts.
5. **Per-Monitor DPI** — keep the cloaked window on a known monitor; if the magnifier moves to
a different-DPI monitor, only your scale factor changes, but re-check wheel screen-coord math.
6. **`IsBorderRequired` availability** — cosmetic only (border is on the cloaked window), but
verify no border artifacts leak into captured frames on Win10 (they don't — the border is a
compositor adornment, excluded from `CreateForWindow` content).
7. **Touch/pen** — out of scope above; `WM_POINTER` forwarding is a separate (similar) effort.
8. **Focus stealing** — `AttachThreadInput` bracketing for `SetFocus`; detach immediately after.
---
## 14. Phased Implementation Plan
1. **Phase 0 (1–2 days):** WGC capture of a cloaked JUCE plugin → scaled display, no input.
Validates capture-while-cloaked + repaint behavior + perf on your machines.
2. **Phase 1:** mouse forwarding without hooks. Clicks/hover work; drags glitch on polling
GUIs — expected. Establishes coordinate mapping + hit-testing.
3. **Phase 2:** MinHook virtualization (`GetCursorPos`, `GetAsyncKeyState`, `GetKeyState`,
`SetCursorPos`, `ClipCursor`, `WindowFromPoint`, `GetMessagePos`). Drags become solid.
Run the drag test matrix.
4. **Phase 3:** PopupManager (reposition strategy) + virtualization pause/resume around popups.
5. **Phase 4:** keyboard/focus, double-click synthesis, cursor-shape mirroring, wheel polish.
6. **Phase 5:** multi-plugin registry, session pause on hidden magnifier, settings UI,
`PrintWindow` fallback, native `setContentScaleFactor` preference path.
