Editor's note: This story has been updated throughout
The three men are joining dozens of fellow House Democrats in this decision, a number that continued to climb on Tuesday. The movement picked up speed over the weekend when Trump engaged in a war of words against U.S. Rep. John Lewis of Georgia, a civil rights icon and beloved figure within the House Democratic caucus.
Castro cited Trump's "utter lack of respect for so many Americans. His attack on civil rights hero John Lewis is the latest example." He added that he will be in Washington to greet constituents in town for the event.
Castro's twin brother, U.S. Housing and Urban Development Secretary Julián Castro, has also said he won't be attending the inauguration.
Green, who represents U.S. House District 9, announced overnight that he will not attend Trump's swearing-in. The seven-term Democrat said his decision was a matter of conscience.
"For some time, I have been hearing from many constituents with varying points of view on the question of whether I will attend the inauguration," Green said in a statement. "My response is Dr. King was right when he indicated: On some issues ‘cowardice asks the question, is it safe? Expediency asks the question, is it politic? Vanity asks the question, is it popular? But, conscience asks the question, is it right? And there comes a time when we must take a position that is neither safe, nor politic, nor popular, but one must take it because it is right.’"
“I will not attend the inauguration because conscience says it is the right thing to do," he added.
Doggett announced his intention not to attend Tuesday afternoon.
"We are sending a message to Mr. Trump," Doggett said in a statement. "Respect, like Pennsylvania Avenue, is a two-way street. Instead of uniting our country, his continued, unprecedented, unpresidential actions are further dividing it."
"I am attending," Gonzalez said in a Monday statement to the Tribune. "And I have accepted the reality that Donald Trump will be the 45th President of the United States. The rhetoric needs to stop. And at this point I believe we have the duty to get back to work."
Trump’s transition team brushed off the inaugural no-shows later Tuesday morning, saying the festivities will still show that Trump plans to be a president for all Americans.
“Obviously we’d love for every member of Congress to attend, but that’s some great seats that hopefully other people can partake in,” Trump spokesman Sean Spicer told reporters.
Patrick Svitek contributed to this report.