HOUSTON —  U.S. Housing Secretary Julián Castro took a pass Friday on addressing growing speculation that he is a top prospect to be Hillary Clinton's running mate.

Speaking with reporters here after a speech, Castro was specifically asked whether he had enough experience to be vice president if the Democratic front-runner picked him. It's a point largely raised by Clinton's critics, though an increasingly salient one as the buzz surrounding Castro and his vice presidential timber intensifies. 

"I'm here today to talk about HUD and the good work that we're doing, and so I'll let other folks make it about 2016," Castro told reporters after addressing the Texas Black Expo. 

Castro, whose backers see him as a shoo-in for a Clinton ticket, has nonetheless faced some questions about whether he is qualified enough for the job. He has served in Obama's Cabinet for less than a year and was mayor of San Antonio for five years before that.

The Texas Tribune thanks its sponsors. Become one.

Castro's trip here came a day before voters headed to the polls in the Alamo City to pick his permanent replacement. On the ballot in the runoff are interim Mayor Ivy Taylor, who won appointment to the office after Castro left last year for Washington, and former state Sen. Leticia Van de Putte

Reiterating his neutrality in the race, Castro wished both candidates well and said he will have a chance to vote this weekend in San Antonio.

"Whoever wins, I look forward to working with the next mayor and helping to ensure that that city has the affordable housing that it needs," he said. Reminded that one of the candidates — Taylor — has been branded the "anti-Castro," Castro repeated that he looks forward to working with whomever gets the job. 

In his speech at the expo, Castro tied his department's efforts to improve housing to a broader mission of expanding economic opportunity for all Americans, something he called "inclusive growth." And he praised President Obama for setting the country on the right path to that goal after inheriting an ailing economy.

"Despite what the critics and the cynics out there sometimes say, the fact is our president has done remarkably well," Castro said. "History will show that he rose to the challenge."