Let's just get rid of all invocations at presidential inaugurals. After all, the "tradition" was first introduced by Franklin Delano Roosevelt at his Second Inaugural in 1937. If George Washington, Thomas Jefferson, Abraham Lincoln and the 31 presidents before FDR could get along without the benefit of an inaugural prayer, so can the 44th.
Disclosure: Yes, I am an atheist, just the same way I'm an a-unicornist. Show me a unicorn and I'll believe in them. Same thing goes for gods.