Field report: web and CRM platform implementation

Onsite design and implementation of a replacement public website, customer relationship management application and supporting virtualised infrastructure for a professional services client. Delivery included engineering-practice reform, identity integration and migration from an undocumented live-server development model.

Assignment record

14 May — Initial onsite discovery

Half Assed engineers attended the client’s head office to review the existing public website and an internal contact database used by sales and account-management teams. The intended engagement was a conventional redesign with a replacement CRM backend and controlled migration of existing customer records.

The client’s engineers demonstrated the live environment by opening files directly over SSH and making a small copy change on the production server. There was no source-control repository, release package, staging environment or reproducible build. Each engineer retained a different local folder containing selected site files.

File names such as header-new.php, header-new-final.php and header-new-final2.php provided the only visible change history. Resolving which copy was current depended on timestamps and the recollection of whoever had last edited the server.

14 May — Backup position

The production server contained the public website, uploaded documents, a MySQL database and the internal customer application. A local backup script existed in the administrator’s home directory but referred to a tape device removed during an office move. Its most recent log entry was eighteen months old.

No current database dump, file archive, virtual-machine snapshot or offsite copy could be located. Engineers believed the hosting provider retained backups, while the provider’s service schedule explicitly excluded customer data. The client had never requested a test restoration.

Before substantive work began, Half Assed took a read-only copy of the live document root, exported the databases and recorded scheduled jobs, packages, users and service configuration. The captured state was stored on two separate encrypted project drives. This became the first verified recovery point for the existing service.

21 May — Delivery-control design

The remediation programme ran alongside product design. A central Git repository was established with named SSH access, protected release branches and tagged deployments. The imported live files formed the initial production baseline, including their inconsistencies, so later changes could be measured rather than reconstructed from memory.

Development, integration, acceptance and production environments were separated. Engineers worked on local checkouts, submitted changes for review and deployed immutable release archives through a scripted process. Direct production editing was removed from ordinary accounts.

Redmine recorded requirements and defects, while Jenkins executed syntax checks, unit tests, asset compilation and packaging. Database migrations travelled with application releases and carried an explicit reverse procedure where rollback was technically possible.

29 May — Security and access model

Shared root and database credentials were replaced with named accounts and service-specific secrets. Administrative SSH required keys and was limited to the management network. Web processes received only the filesystem and database permissions needed by their application role.

Access was divided among content editors, sales users, account managers, compliance staff, application administrators and infrastructure administrators. No ordinary CRM role could change server configuration, and no web-deployment account could administer customer identities.

Production database access required an approved support task and was logged. Test environments used masked customer records rather than copies of current names, contact details and contract information.

05 June — Executive design workshops

The chief executive, commercial director, operations director and marketing lead joined a series of frontend workshops. They reviewed the existing brand, customer enquiries, service structure, account onboarding and the information required before a prospect could be assigned to a sales team.

Initial page designs covered the home page, service pages, sector landing pages, insight articles, contact routes, customer sign-in and account dashboard. The executive group preferred a conservative editorial presentation with strong evidence, clear calls to action and reduced dependence on promotional animation.

Clickable HTML proofs were used instead of static artwork for navigation and form review. Executive comments were recorded against named components and templates, preventing contradictory amendments from being applied directly to isolated pages.

18 June — Engineering capability programme

The client team had maintained PHP applications but had limited experience with contemporary framework, asset and typed-JavaScript workflows. Half Assed delivered paired implementation sessions while the new platform was being built.

Symfony 2.1 introduced routing, controllers, dependency injection, Doctrine entities, form handling and environment-specific configuration. Existing procedural PHP was not copied into controllers unchanged; business rules were separated into services with unit coverage around pricing, enquiry assignment and account state.

Sass and Compass replaced manually duplicated stylesheet values and browser-specific fragments. Variables, mixins and partials represented colour, typography, grids and shared components. Compiled CSS remained the production artefact, so the live servers did not require development tooling.

TypeScript 0.8 was introduced selectively for the CRM’s more stateful interface modules. The early compiler provided enough type checking to document customer records, search results and asynchronous responses while emitting JavaScript compatible with the client’s supported browsers. Ordinary content pages remained progressively enhanced and usable without the application modules.

02 July — Virtual infrastructure specification

The target estate used three VMware ESXi hosts managed through vSphere 5.1. Shared storage provided separate production, database, management and backup volumes. Application services could restart on another host following ordinary host failure.

Two edge virtual machines acted as ingress controllers using HAProxy and Nginx with VRRP failover. They terminated TLS, applied request limits, normalised forwarding headers and routed public website, portal and CRM traffic to separate application pools. Administrative paths were accepted only from the management network.

The term ingress controller referred to the managed reverse-proxy tier at the service boundary. It did not depend on a container scheduler. Health checks removed failed application nodes, and a maintenance backend could be selected without changing public DNS.

16 July — Encryption design

The client required customer and contract data to remain encrypted at rest. Because the selected VMware release did not provide native policy-based VM encryption, the design used encrypted storage containers around the sensitive VM image groups and encrypted guest data volumes.

Keys were to be separated from the datastores, released to authenticated hosts at startup and retained only in memory while systems were running. Recovery material was meant to be escrowed offline for a complete cold start. The signed design stated explicitly that no key-release dependency could reside solely inside the encrypted estate it unlocked.

Storage snapshots and backup exports preserved encrypted blocks. This protected copied media but made independent key custody essential. The acceptance schedule included restart of an application VM and loss of one host; a simultaneous power-off of every host and management service was listed for the later resilience phase.

30 July — Database provisioning

PostgreSQL 9.2 was provisioned with separate databases and roles for public enquiries, CRM records, identity mapping, reporting and deployment metadata. The primary database ran on the production storage tier, with write-ahead logs streamed to a warm standby.

Schema migrations created organisations, contacts, opportunities, activities, documents, consent records, assignments and audit events. Foreign keys and explicit transaction boundaries replaced assumptions enforced only by the former application screens.

The legacy MySQL data was profiled for duplicates, incomplete addresses, invalid owners and inconsistent status values. Transformation rules were reviewed by commercial and compliance representatives before three rehearsal migrations.

13 August — SSO and two-factor authentication

The CRM was integrated with the client directory through a redundant SAML 2.0 identity service. Directory groups mapped users to application roles, while leavers and disabled accounts lost access without a separate CRM administration step.

Infrastructure administrators and users handling exported contract data were required to complete a second factor. The implementation supported time-based one-time-password tokens and retained a small number of sealed recovery codes under the client’s access procedure.

Service-to-service authentication used dedicated identities rather than employee accounts. Assertions were audience-restricted, time-limited and logged at the identity service and receiving application. The public website and ordinary customer enquiry forms did not depend on staff SSO availability.

27 August — CRM implementation

The CRM combined customer history, opportunity stages, scheduled actions, document references and responsibility assignment. Search covered organisation, person, telephone, email and contract reference. Every material state change produced an audit event containing actor, time and previous value.

Dashboards showed unassigned enquiries, expiring actions, stalled opportunities and account-management workload. CSV export required an authorised role and a second-factor confirmation. Large exports ran as background jobs and expired after download.

The public website submitted structured enquiries through a queued integration. If the CRM was unavailable, forms retained their accepted messages and displayed a reference rather than asking customers to submit again.

10 September — Frontend construction

Approved executive proofs were converted into Symfony templates and the shared Sass component set. Content layouts allowed editors to publish service evidence, biographies, case studies and articles without constructing arbitrary page structures.

Responsive behaviour was tested across current desktop browsers, tablets and the growing range of smartphone widths. Images received server-generated derivatives, while the interface avoided JavaScript dependence for primary navigation and enquiry submission.

TypeScript modules supported CRM search suggestions, activity entry, dashboard refresh and document-upload progress. Compiled JavaScript was versioned with each release and served through the edge cache.

01 October — Deployment rehearsals

Three full releases were deployed through development, integration and acceptance without direct server editing. Each build identified its source tag, database migration and compiled assets. Failed acceptance could restore the preceding application package and database snapshot.

Host-failure tests restarted application VMs successfully on the remaining VMware nodes. Individual encrypted image groups unlocked through the key-release service while the rest of the estate stayed online. Edge-controller failover moved the service address without interrupting established test sessions.

The full cold-start test remained outstanding because the client did not approve complete interruption of the new shared platform before launch. The project record carried it into the post-launch resilience schedule.

05 November — Data migration

The final legacy extract was taken after duplicate and ownership reports had been resolved. Customers, contacts, opportunities, activities and current document references loaded into PostgreSQL through versioned migration jobs. Counts and financial totals reconciled to the approved source reports.

Documents were copied into the encrypted storage tier and associated by checksum. Records without a reliable owner entered a controlled exception queue rather than being silently assigned. Compliance staff sampled current, closed and restricted accounts.

The old customer application was placed into read-only mode. The public website continued operating until the final DNS and edge change.

03 December — Production launch

The ingress service address became authoritative for the public website and customer portal. CRM users authenticated through SSO and completed two-factor enrolment where required. Enquiries entered the queue, created CRM records and reached the correct commercial teams.

Database replication, encrypted storage, backup snapshots, reporting jobs and monitoring completed their launch checks. The client engineering team deployed a small content correction through Git, review and the release pipeline without opening a production shell.

Executives approved the new frontend and commercial dashboards. The old web application remained isolated and read-only for the agreed retention period.

First eleven months of operation

The platform operated without a major outage. Application releases moved through source control and acceptance, and the client team continued using Symfony, Sass and TypeScript practices introduced during delivery. Routine host maintenance moved VMs between nodes without a complete estate shutdown.

Nightly snapshots and repository exports reported success. Both retained encrypted VM blocks and application data. Restore tests recovered individual files and VMs while the encryption and identity services were already running.

The deferred full cold-start exercise was rescheduled twice to avoid business disruption. At no point during ordinary maintenance were all VMware hosts, identity nodes and key services offline together.

02 November 2013, 03:14 — UPS failure

An internal UPS supporting the virtualisation hosts reported a battery and inverter fault during a building-supply interruption. Its bypass transfer did not complete, and all three VMware hosts, the shared-storage controllers and the management network lost power within the same interval.

The shutdown was not coordinated. Some guests stopped immediately while databases and application VMs remained active until their hosts lost input. Building power returned later that morning, but the estate now required its first complete cold start since implementation.

Physical storage checks completed and the datastores presented their encrypted VM-image containers. No unrecoverable disk or array fault was detected.

05:06 — Initial restart failure

ESXi management interfaces and the storage array came online, but sensitive VM groups remained locked. The standard startup account did not possess an offline decryption key. Host logs showed repeated calls to an executable named hawd-unseal, followed by authentication-endpoint timeouts.

The documented recovery procedure referred to an escrow package in the client safe. The safe contained sealed two-factor recovery codes and printed host credentials but no VM-image master key. A second envelope held an obsolete key from the pre-production acceptance estate.

Application, database, identity, backup and management VMs could not be started. The edge controllers presented only their static maintenance response.

06:22 — Circular dependency identified

Engineers examined the host-side unlock package from its unencrypted boot partition. Instead of reading an offline escrow key, the routine attempted to authenticate a service identity against the SAML platform, require a token-service approval and retrieve the current image key from a secrets database.

The SAML identity nodes, token service and secrets database were themselves stored inside the encrypted VM group. Starting them required the same key the routine expected them to release. During previous host or VM restarts, at least one identity and secrets node had always remained online, concealing the dependency.

The backup repository used the same envelope keys and identity-mediated release process. Its encrypted exports were present and internally consistent, but their keys could not be retrieved without first starting the unavailable identity estate.

07:10 — Undocumented package review

No version of hawd-unseal appeared in the project Git repository, deployment manifests, signed encryption design or acceptance evidence. Filesystem timestamps placed installation during the original infrastructure build, before production launch.

The console screen displayed an ASCII rendering of the Half Assed half-donkey logo above the text “SECURE IMAGE CONTROLLER.” Package metadata identified Half Assed Web Design as vendor but contained no individual author, source location or recovery instruction.

Client engineers stated that they had not created the routine. Half Assed project engineers stated that it contradicted the approved offline-escrow design and was not part of their recorded deliverables. No change ticket or email authorised its installation.

08:35 — Installation-team review

The original implementation roster was reconstructed from access logs and building records. Several engineers from both organisations had administrative access during the relevant period. Available logs recorded package deployment under a shared commissioning account that had been retired after launch.

One member of the Half Assed installation team had since left the company and emigrated to China. Former colleagues remembered that engineer working on startup automation and console branding, but no surviving record attributed the key-release routine to them. They could not be reached during the incident.

The person’s relocation was treated as contextual information rather than proof of authorship. Every interviewed participant declined responsibility for designing, approving or installing the circular unlock mechanism.

10:12 — Recovery attempts

Engineers searched project drives, password managers, safes, email archives, retired build servers and administrator home directories for an exported master key. They found test keys, certificates and recovery material for the old website, but nothing capable of opening the production image group.

Cloning the SAML or secrets-service VMs did not help because their virtual disks remained encrypted. Recreating the service hostnames and issuing replacement certificates could reproduce network endpoints but not the secrets database containing the image key.

Individual datastore blocks and backup objects were readable as ciphertext. Their headers and checksums indicated intact encrypted containers. Brute-force recovery was not credible: keys were randomly generated at full length, and no weak password-derived protector was present.

Following week — Data assessment

Some developers retained old local source checkouts, but none represented the complete production release or contained current customer data, uploaded documents, database migrations and configuration together. The original 2012 pre-migration capture was available but predated eleven months of CRM operation.

Database storage could not be distinguished from free space without decryption. The primary, standby and backups all depended on the same unavailable production key. No partial customer record or current document could be extracted.

Attribution review

The approved design required offline escrow and explicitly prohibited an unlock dependency contained inside the protected estate. The installed implementation did the opposite. It functioned during rolling maintenance because a surviving identity node could always authorise the next unlock, but it had no valid first step after complete power loss.

The Half Assed branding on the routine established apparent organisational origin but not authorship. Shared commissioning credentials prevented individual attribution. The engineer who later emigrated to China had worked near the relevant component, but no evidence demonstrated that they wrote or installed it.

The client, Half Assed project team and former infrastructure participants each denied responsibility. The review could not identify who replaced the approved escrow model, why it was done or how the full cold-start acceptance test remained deferred.

Final data-loss conclusion

The UPS failure did not materially damage the storage hardware. It exposed a complete dependency cycle that had remained hidden for eleven months: encrypted VM images required a key from SSO and secrets services, while those services could not start until the same images were decrypted.

The production encryption key was not escrowed offline. Every current database replica, source repository, document store, identity service and backup copy was cryptographically dependent on infrastructure contained within the locked image group.

With no usable key, running identity service or independent decrypted backup, the entire website and CRM estate was declared unrecoverable. All production data accumulated since migration was lost.