Goof Off is really a solvent, not a cleaner. By that, I mean that it's more of a stain and glue remover (great at removing price tags and their adhesive backings from gifts) and not something I'd readily recommend for vinyl siding and such. I guess it depends on what is on your vinyl, under the soffits, and on the gutters. If they're just dirty from regular mud and wind blown dust and dirt, as well as the occasional bird with aim, I'd say something more specific to that application will be better. (A pressure washer, which can be rented or purchased, will definitely do the job 99% of the time.)
The ZEP products are highly rated, and if you search the HD website for "vinyl siding cleaner", you'll find a wide variety of products more designed for your application than Goof Off. Many of the products you'll see will hook up directly to a standard garden hose -- no pressure washer required. If you spilled paint or glue or something else on the house somehow, then Goof Off may be the product, but for general purpose vinyl siding and gutter cleaning, I'd have to recommend something far more specific to the task. Goof Off may actually work (haven't tried it, so I don't know), but I do know that, over the years, I have used products designed for cleaning what you're asking about with success.
My main concern with Goof Off on siding and soffits and gutters is what it will or won't do to the paint. I don't know how it'll react with that where the surface is metal, vinyl, and so on, so if you do try it, go in an inconspicuous corner and test it first to be safe. I'd really just go with the more specific product though.