As we plan to move Infrastructure services to a new Hosting provider its a fair time to re-evaluate the design. Currently these are only ideas, tbd.
Requirements
Deficiencies with current design
- multiple ssh hops causing latency and configuration pain for a few admins
 - firewall configuration out of control of local admins
 - lack of centralised account management making it a bit labouring to remove/add admins
 - lack of centralised configuration making standard configurations on ssh/web service/logging options difficult to acheive/maintain
 - lack of configuration history on services/machines
 - lack of a clear tested backup that is available to local admins
 - logging checked for abnormal events
 - monitoring of systems for expected statue
 - email standards - defined email address, dkim signing
 - bandwidth accounting /montoring isn't done - needs to be kept in check
 
Security Requirements
Be guided by SM
- indications when package updates are required
 - restrictive firewall rules in and out
 - strong authentication for admins to access services)
 - strong identity associated with logins (no shared or root accounts)
 - logging with strong integrity
 - file modification detection
 
Easy of Maintainence Requirements
- preference for distribution maintained packages
 - non-distribution packages need some approval process
 - formation of templates for commonly deployed tasks
 - easier integration of application maintainers
 
Design
Proposed Solutions
There are multiple ways to achieve some of these requirements - lets look at them.
Deficiencies
- multiple ssh hops: 
- direct ssh access - security requirement mitigated by: standardised config, monitoring
 - hopper with more open input firewall rules
 - Single Packet Authentication / Port knocking
 
 - firewall config: 
- locally controlled at VM with alerts on rule changes just to be sure they are authorized/consistant with security requirements
 
 - account management: 
- LDAP server with shared /home directory exported across all guest VMs (Samba?)
 - puppet account management
 
 - standardization: 
- puppet to manage config in OS independent manner
 
 - config history: 
- puppet supports config history?
 - RCS on critical files
 - other VC?
 
 - backup/restoration: 
- daily lvm snapshots of running hosts
 - individual backup daemons on hosts - preferences?
 - better SQL backups
 
 - logging: 
- admins commit rules for their system to categorize log messages
 - centralised logging for integrity. logs readable by local admins
 - common alert rules for hits on prohibited firewall rules (outbound)
 
 - monitoring: 
- local admins place nagios rules to validate system functionality
 - common rules for monitoring to check only designed listening services are running
 
 - email standards: 
- puppet managed
 - dedicate local VM for emailout
 
 - bandwidth accounting /monitoring 
- iptables accounting by IP
 - specialised log monitoring / webanalyiser/ mailgraph
 
 
Security Requirements
- package updates 
- cron-apt or similar
 - monitoring agent detects this
 
 - restrictive firewall 
- rules validated by monitoring
 - rules occasionally tested
 
 - strong authentication for admins 
- ssh key access only
 
 - strong identity associated with logins (no shared or root accounts) 
- no root logins, sudo access for root access
 
 - logging with strong integrity 
- remote log with log directory exported back to machine readonly
 
 - file modification detection 
- lvm snapshots are examined for modified files. Excludes distribution updates and logging/database files
 
 
Maintainence Requirements
Design
Host OS
- ssh access to infrastructure sysadmin managers
 
Common Services
- puppet
 - ldap/samba share/ kerberos(?)
 - logging alert rules repository
 - monitoring alert rules repository
 - dns cache
 - emailout
 
