yeah thats it for me as well. ive also always played instruments way before i ever got into electronic music so that probably has something to do with it
i very rarely feel inspired when im sitting in front of a VST window twiddling virtual dials around or clicking in notes with my mouse pointer. but working on a track with some keys to bang, knobs to turn, and faders to slap around in front of me is an entirely different experience. the latter actually makes me feel like im doing something musical. i dont have to think about what i'm doing, i just do it.
sure you can use midi controllers, but by the time i have everything mapped, whatever vibe there was is gone anyways. also hardware just gives me the results i want faster. yeah i could probably painstakingly reproduce the tone and character to 80-90% precision with some plugins, but why?