Yep. And you'd have your spider-tenders, the military to protect yourself and your products, people supplying the city with all the other resources it needs, the people responsible for enchanting properties into the silk, tailors/designers, the standard builders, entertainers, weaponsmiths, everything else... Most drow in the city would have other careers, because a city can't survive on just silk. It'd get the reputation of silk central, people might imagine countless drow draped in silk sheets, those responsible for the business would be ludicrously, opulently wealthy... but the average drow would not be churning out silk lingerie for sale.
Places do become known for their primary export, even if not everybody there makes it.