There is at least two ways to override template file and customize the html wich is rendered by Pushword : First is App Way (app per app), second is the bundle way (global).
Simplest way is to override it (partially or completly) by create a new file in ./templates/{$host}/page naming it like the default one
Eg: Overriding the default navbar can be done creating a file ./templates/{$host}/page/_navbar.html.twig.
You can see how it's handle for the documentation.
Every package (even core) is built like a symfony bundle, so you can override template file the bundle way.
For this, you just need to find the template file you want to override and create a copy inside your templates/bundles folder like.
Example, to override /page/page_default.html.twig :
./templates/bundles/PushwordCoreBundle/page/page_default.html.twig
Create a page with the slug 404 in the admin to customize your error page. Pushword will automatically render it when a visitor hits a missing URL.
If no 404 page exists, Pushword falls back to a generic error template displaying translated messages (errorTitle, errorDescription).
For static sites (using static-generator), a 404.html file is generated automatically from the same mechanism — one per configured locale.