Is this expected behavior? Or is there a setting I'm missing or different
workflow I should follow in order to get textmate to remember what should be
visible in my project?

Yes, this is expected behavior.  It rescans the directories and adds files whenever you open the project or command+tab back to TextMate*.

The way to change which files are discovered is to right click on the root folder and choose "Show Information..." (or select the folder and hit command+i).  Where it says "Recursively Include Contents Matching..." you can define "regular expressions" to include/exclude files and folders.  I'm not sure which syntax that box is using for the patterns but exclamation point (!) acts as negation so you can easily exclude stuff from your directories.

*You can use the plugin ReMate to disable this behavior if you don't want to exclude the files but rescanning them is slow.

Hope that helps!  Enjoy using TextMate! :)

Brandon