Tag: IT

  • Zen and the Art of Solution Architecture

    Solution Architecture begins with a simple question:

    “What are we actually trying to do?”

    This question is rarely welcomed.

    The project has already been named.

    The budget has been estimated.

    The vendor has been selected.

    The steering committee has approved a roadmap.

    A programme manager has produced a slide containing six coloured arrows moving confidently toward TARGET STATE.

    Everyone is therefore extremely busy.

    Your question is considered disruptive.

    This is your first lesson.

    1. The Architecture Is Not the Diagram

    The diagram is evidence that architecture may have occurred.

    It is not architecture.

    A rectangle labelled API GATEWAY connected to a rectangle labelled CLOUD by a tasteful blue arrow does not constitute a design.

    Nor does adding:

    ZERO TRUST

    in the corner.

    The architecture is the set of decisions, constraints, interfaces, assumptions, failure modes, operational consequences and compromises represented imperfectly by that diagram.

    Unfortunately, nobody wants to read those.

    They want the diagram.

    Make the diagram.

    Then keep the important material somewhere adults can find it.

    2. Begin With the Problem

    Projects rarely begin with problems.

    They begin with solutions.

    “We need Salesforce.”

    “We need Kubernetes.”

    “We need AI.”

    “We need a data lake.”

    “We need to move to Azure.”

    “We need microservices.”

    “We need Zero Trust.”

    “We need blockchain.”

    The architect’s first duty is to ask:

    “Why?”

    Do not say it aggressively.

    Say it gently.

    Like a therapist.

    “What outcome are we trying to achieve?”

    There may be a silence.

    Someone will eventually say:

    “Modernisation.”

    This is not an outcome.

    Try again.

    “What becomes better?”

    Another pause.

    “User experience.”

    Still not an outcome.

    Eventually, after enough patient excavation, someone may admit:

    “Our order system takes four days to update stock.”

    Excellent.

    Now you have something.

    You may discover that the £14 million cloud transformation can be replaced by fixing three SQL queries.

    Do not expect gratitude.

    3. Requirements Are Things People Remember Later

    At the beginning of a project, requirements are vague.

    “We need it secure.”

    “It needs to be fast.”

    “It must be resilient.”

    “It should scale.”

    Users will insist there are no further requirements.

    This is because the real requirements are hiding.

    They emerge after design approval.

    “Oh, by the way, users in Singapore need access.”

    “We forgot to mention the classified network.”

    “It has to work offline.”

    “There are 40,000 users.”

    “The database is 18 terabytes.”

    “We can’t change the client.”

    “We can’t change the server.”

    “We can’t change the network.”

    “We can’t install software.”

    “We need it by Christmas.”

    “What year?”

    “This year.”

    A mature architect assumes hidden requirements exist.

    A wise architect goes hunting for them.

    4. Functional Requirements Are the Easy Ones

    Functional requirement:

    “The user shall submit an expense claim.”

    Non-functional requirement:

    “The service must remain available during payroll processing, survive loss of a data centre, return results within two seconds, support 12,000 concurrent users, comply with retention policy, integrate with legacy identity, operate through the corporate proxy and cost less than the existing service.”

    Everyone will discuss the expense form.

    You should worry about everything after it.

    Systems rarely fail because nobody knew the button should say Submit.

    They fail because nobody asked what happens when 9,000 people press it at 16:55 on Friday.

    5. Constraints Are Architecture

    A blank sheet of paper is not architecture.

    It is fantasy.

    Real architecture happens because something unpleasant is true.

    The WAN is slow.

    The database cannot be changed.

    The vendor only supports Windows.

    The security team prohibits inbound connections.

    The site has intermittent power.

    The budget is fixed.

    The deadline is ridiculous.

    The application is twenty years old.

    These constraints are not inconveniences around the design.

    They are the design.

    Anyone can architect a perfect system with infinite money, infinite time and no legacy estate.

    This person is called a conference speaker.

    6. The Existing Estate Is Not a Mistake

    The phrase legacy system is often pronounced with disgust.

    Be careful.

    Legacy means:

    “It has been useful long enough to become inconvenient.”

    That Unix server may be ugly.

    It may use an authentication protocol that predates several members of the project team.

    But it has processed every transaction correctly since 2003.

    Your shiny replacement has been alive for six weeks and already requires a hotfix.

    Show respect.

    The old system knows things.

    7. Never Assume the Network

    Application architects sometimes draw:

    USER → APPLICATION

    Between these objects lies:

    Wi-Fi,

    LAN,

    WAN,

    firewalls,

    proxies,

    load balancers,

    NAT,

    DNS,

    VPN,

    TLS inspection,

    SD-WAN,

    identity controls,

    routing policy,

    and occasionally a satellite link nobody mentioned.

    The arrow is doing considerable emotional labour.

    Talk to Network.

    Early.

    8. Identity Is Not “SSO”

    A requirement will say:

    “Must support SSO.”

    This sounds simple.

    It is not.

    Ask:

    Which identity provider?

    Which user populations?

    Employees?

    Contractors?

    Partners?

    Customers?

    Devices?

    Service accounts?

    Privileged administrators?

    Which protocol?

    SAML?

    OIDC?

    Kerberos?

    LDAP?

    Something proprietary invented in 2009?

    What happens when identity is unavailable?

    What about break-glass access?

    Who owns lifecycle?

    Who removes access when Bob leaves?

    Identity architecture begins where the box labelled SSO becomes embarrassing.

    9. Security Is a Design Property

    Security added at the end is usually a firewall rule and some regret.

    Bring security into the design early.

    Not because security teams are always right.

    They are not.

    But because discovering in week forty-two that the proposed service cannot legally transmit data to its chosen cloud region is professionally tiring.

    Also, never accept:

    “Security says no.”

    Ask:

    “What threat or control requirement are we addressing?”

    This transforms theatre into engineering.

    Sometimes.

    10. Availability Has Arithmetic

    The business will ask for:

    “Five nines.”

    Ask why.

    They may not know what it means.

    99.999% availability allows only a few minutes of downtime per year.

    That is expensive.

    It implies engineering.

    It implies operational maturity.

    It implies redundancy.

    It implies maintenance design.

    It implies monitoring.

    It implies people answering phones at unpleasant hours.

    Then ask:

    “How much revenue do we lose during one hour of outage?”

    If the answer is £400, perhaps five nines is excessive.

    Architecture includes knowing when reliability is worth buying.

    11. Disaster Recovery Is Not a Second Data Centre

    A second copy of broken infrastructure is not resilience.

    Ask:

    What is the RTO?

    What is the RPO?

    Who declares disaster?

    How is failover initiated?

    How is data reconciled?

    How do users reconnect?

    What happens to DNS?

    What happens to authentication?

    How do you fail back?

    Has anyone tested it?

    If the answer to the last question is no, you do not have disaster recovery.

    You have disaster optimism.

    12. Integration Is Where Systems Go to Die

    Every project says integration will be simple.

    “It’s just an API.”

    This sentence has killed millions of project hours.

    Ask:

    Who owns the API?

    Is it documented?

    Is it synchronous?

    What is the timeout?

    What is the retry policy?

    What is the rate limit?

    What happens if the downstream system is unavailable?

    How are messages deduplicated?

    What happens when schemas change?

    How are errors reconciled?

    Someone will eventually say:

    “We can just use CSV.”

    Do not laugh.

    CSV has outlived technologies that mocked it.

    13. Data Has Owners Until You Ask Them to Make a Decision

    Every organisation claims data ownership.

    Then you ask:

    “Who defines the authoritative customer address?”

    Silence.

    CRM says it owns customer data.

    Finance says billing is authoritative.

    Sales has another address.

    The warehouse has a spreadsheet.

    Marketing bought a list.

    A regional office maintains its own database because “the central one is always wrong.”

    Architecture reveals political geography.

    The data model is often just the map.

    14. Cloud Is Not an Architecture

    Cloud is a hosting model plus several thousand services and an invoice.

    “We’re cloud-first.”

    Fine.

    Which cloud pattern?

    Managed service?

    Containers?

    Virtual machines?

    Serverless?

    SaaS?

    Hybrid?

    Private connectivity?

    Public endpoints?

    Data residency?

    Identity federation?

    Landing zone?

    Logging?

    Key management?

    Backup?

    FinOps?

    Saying “cloud” does not answer these questions.

    It merely provides more expensive ways to avoid them.

    15. Microservices Are Not Small Services

    A monolith is not automatically bad.

    Microservices are not automatically modern.

    Microservices introduce:

    distributed transactions,

    network failure,

    service discovery,

    versioning,

    observability,

    deployment orchestration,

    eventual consistency,

    and several additional ways for developers to blame each other.

    Use them when organisational and technical boundaries justify them.

    Do not use them because somebody saw Netflix architecture slides.

    You are not Netflix.

    Your organisation sells insurance in Coventry.

    16. Kubernetes Is Not a Business Requirement

    Nobody wakes at 03:00 and thinks:

    “I wish my council tax portal had more container orchestration.”

    Kubernetes is useful.

    It is also operationally substantial.

    If your entire application consists of three services used by 400 people, ask whether you need:

    clusters,

    operators,

    service meshes,

    ingress controllers,

    Helm charts,

    and six engineers who now describe themselves as platform specialists.

    Sometimes a virtual machine is fine.

    This statement may cause offence.

    Proceed.

    17. Buy Versus Build Is Mostly About Regret

    Build:

    Total control.

    Total responsibility.

    Buy:

    Less control.

    Different responsibility.

    SaaS:

    Minimal control.

    Subscription regret.

    There is no universally correct answer.

    Ask:

    Is this capability differentiating?

    Do we have engineering capability?

    How long will we own it?

    What is the exit strategy?

    How portable is the data?

    What happens when the vendor doubles the price?

    What happens when they discontinue the product?

    Architecture must include how you leave.

    Nobody wants to discuss divorce during the wedding.

    Discuss it anyway.

    18. Vendor Diagrams Are Aspirational Literature

    Vendor architecture diagrams have several common features:

    everything is blue,

    everything is secure,

    nothing fails,

    and every arrow leads toward their product.

    Their solution is always:

    scalable,

    resilient,

    AI-enabled,

    enterprise-grade,

    zero-trust,

    cloud-native,

    and transformative.

    Ask difficult questions.

    Where is state stored?

    What are the limits?

    What fails closed?

    What fails open?

    How are upgrades handled?

    What is excluded from the licence?

    What requires professional services?

    The account manager will stop inviting you to lunch.

    This is acceptable.

    19. Licensing Is Architecture

    An architect who ignores licensing can design a technically elegant financial disaster.

    A four-node cluster may require licensing all physical cores.

    A passive DR site may not be passive according to the contract.

    Virtual mobility may widen the licensed estate.

    A “free” feature may require an enterprise tier.

    Ask early.

    Licensing constraints can change topology.

    This is deeply annoying.

    It is still architecture.

    20. Cost Is a Technical Requirement

    If the system works beautifully but nobody can afford to run it, it does not work.

    Include:

    compute,

    storage,

    network,

    support,

    licensing,

    backup,

    monitoring,

    operations,

    people,

    DR,

    growth,

    and exit costs.

    Cloud solutions especially need cost modelling under load.

    A technically perfect design that generates £80,000 per month of unexpected egress is simply a sophisticated billing incident.

    21. Operations Begins Before Go-Live

    Ask who will operate the system.

    Someone will say:

    “BAU.”

    BAU is not a team.

    It is a mystical destination where projects send responsibilities they no longer wish to discuss.

    Who monitors it?

    Who patches it?

    Who restores it?

    Who owns certificates?

    Who handles alerts?

    Who handles capacity?

    Who talks to the vendor?

    Who renews support?

    Who knows when the licence expires?

    If nobody has names, you have not finished the architecture.

    You have merely moved the problem forward in time.

    22. Supportability Beats Cleverness

    Architects enjoy elegance.

    Operations enjoys sleeping.

    Choose accordingly.

    A clever design requiring rare expertise may be technically superior and operationally catastrophic.

    Ask:

    Can the organisation support this at 02:00?

    Can new staff understand it?

    Can it be diagnosed?

    Can it be patched?

    Can it be recovered?

    Can a supplier support it?

    If the answer is no, simplify.

    Complexity is a debt instrument.

    Interest is payable during incidents.

    23. Every Exception Becomes Permanent

    “Temporary firewall rule.”

    “Temporary admin account.”

    “Temporary bypass.”

    “Temporary integration.”

    “Temporary manual process.”

    There is no temporary.

    There is only:

    not yet documented as permanent.

    If an exception is genuinely necessary, give it:

    an owner,

    an expiry date,

    a review point,

    and a removal plan.

    Otherwise it will still exist in twelve years.

    Someone will call it heritage.

    24. Architecture Principles Are Useful Until They Collide

    Typical principles:

    Cloud first.

    Reuse before buy.

    Buy before build.

    Secure by design.

    API first.

    Data is an asset.

    Automation first.

    Open standards.

    User centred.

    Minimise technical debt.

    All excellent.

    Then reality arrives.

    The legacy vendor has no API.

    The approved cloud cannot host the workload.

    The budget does not fund replacement.

    Security requires an appliance.

    The deadline is six weeks.

    Architecture is the practice of resolving contradictions among desirable principles.

    The principle that always wins is:

    “The service must still work.”

    25. Standards Are Guardrails, Not Holy Scripture

    Standards reduce chaos.

    They improve supportability.

    They prevent every project inventing its own authentication system.

    Good.

    But standards also age.

    A standard that exists only because nobody has reviewed it since 2016 is not governance.

    It is sediment.

    Architects should know when to comply.

    They should also know when to request an exception.

    The important word is request.

    Do not simply ignore standards.

    That creates archaeology.

    26. Technical Debt Is Sometimes Rational

    Not every shortcut is stupid.

    Sometimes the correct decision is:

    “We will tolerate this ugly workaround for eighteen months because replacing the underlying platform now costs £2 million.”

    That is not failure.

    That is a conscious trade-off.

    Technical debt becomes dangerous when:

    nobody records it,

    nobody owns it,

    nobody prices it,

    and everyone assumes someone else will repay it.

    Record the debt.

    Record the interest.

    Record the exit.

    Then make the decision visible.

    27. Decision Records Are More Valuable Than Beautiful Documents

    Six months after implementation, nobody remembers why a design decision was made.

    They remember opinions.

    “That was Security.”

    “No, Network insisted.”

    “The vendor said we had to.”

    “I thought Architecture chose it.”

    Use Architecture Decision Records.

    Short ones.

    Decision.

    Context.

    Options.

    Rationale.

    Consequences.

    Date.

    Owner.

    Future architects will bless you.

    Or at least swear at you less.

    28. Never Confuse Consensus With Correctness

    Architecture boards sometimes attempt to reach consensus.

    This is admirable.

    It can also produce grotesque systems designed to offend nobody.

    The network team wants one thing.

    Security wants another.

    Applications wants another.

    Operations wants another.

    The project wants all of them satisfied.

    The resulting solution uses:

    two identity systems,

    three integration methods,

    four hosting patterns,

    and a special exception for Finance.

    Everyone approves.

    Nobody is happy.

    Good architecture sometimes requires a decision.

    Make it.

    Document it.

    Own it.

    29. Governance Should Reduce Risk, Not Generate Theatre

    Good governance asks:

    Is the problem understood?

    Are requirements credible?

    Are risks visible?

    Are decisions justified?

    Is the solution supportable?

    Bad governance asks:

    Has slide 14 been updated to the approved template?

    Architecture assurance is not a ritual blessing.

    Do not become the priest who stamps diagrams.

    Ask questions that can still change something.

    If all decisions have already been made, you are not governing architecture.

    You are conducting an autopsy.

    30. The Architecture Review Board Is Not a Court

    Do not arrive intending to defeat the project.

    Projects are not criminals.

    Usually.

    Your role is to improve the probability of success.

    Ask hard questions.

    But explain why.

    “Where is the session state?”

    is useful.

    “This is rubbish.”

    is not.

    Architects who gain a reputation for obstruction stop being invited early.

    Then they complain architecture is engaged too late.

    This is not Zen.

    This is self-harm.

    31. Never Say “Best Practice” Without Context

    Best practice for whom?

    A global bank?

    A ten-person charity?

    An aircraft manufacturer?

    A hospital?

    A startup?

    A submarine?

    Architecture is contextual.

    A solution appropriate for one environment may be absurd in another.

    Prefer:

    “Given these requirements and constraints, this pattern reduces these risks.”

    It is longer.

    It also means something.

    32. The Target Architecture Is a Direction, Not a Destination

    Target architectures often contain an enchanted future in which:

    all applications use APIs,

    identity is unified,

    data is governed,

    technical debt is gone,

    everything is automated,

    legacy systems are retired,

    and users are delighted.

    This world does not exist.

    Before you reach it, the organisation will:

    merge,

    restructure,

    buy another company,

    change strategy,

    replace the CIO,

    and purchase a large SaaS platform nobody told Architecture about.

    Target architecture is a compass.

    Not a railway timetable.

    33. Roadmaps Are Negotiations With Entropy

    A roadmap should show dependencies, transition states and sequencing.

    It should not simply contain:

    2026 — TRANSFORM
    2027 — OPTIMISE
    2028 — INNOVATE

    That is astrology.

    A useful roadmap tells you:

    what changes,

    in what order,

    why,

    what enables what,

    what can coexist,

    and where risk reduces.

    It should also acknowledge that Year Three is approximately fictional.

    34. Sometimes the Correct Architecture Is “Do Nothing”

    This is rarely popular.

    Projects exist to change things.

    Architects are paid to design things.

    Vendors are paid to sell things.

    But sometimes:

    the system is stable,

    the risk is understood,

    the replacement cost is unjustified,

    and there is no meaningful business benefit.

    “Do nothing for two years while reducing operational risk” can be excellent architecture.

    Do not confuse activity with progress.

    35. Proof of Concept Does Not Mean Production

    A developer demonstrates the technology on a laptop.

    It works.

    Management becomes excited.

    “Can we go live next month?”

    No.

    The proof of concept has:

    one user,

    no monitoring,

    no backup,

    no security model,

    no support model,

    no DR,

    no performance testing,

    no audit logging,

    and credentials stored in the source code.

    The purpose of the proof of concept was to establish feasibility.

    It has done so.

    Do not punish it by promoting it into production.

    36. “Scalable” Is Not a Number

    Every solution is described as scalable.

    Ask:

    From what to what?

    100 users to 1,000?

    10,000 to 1 million?

    Ten transactions per second to 20?

    What dimension scales?

    Compute?

    Storage?

    Connections?

    Tenants?

    Geographies?

    Staff?

    If nobody knows the expected load, scalability is decorative language.

    37. Latency Is Geography Collecting Rent

    You cannot architecture-diagram your way around the speed of light.

    If users are in Australia and the application is in London, something will take time.

    If the application makes twenty sequential database calls per transaction, it will take more time.

    If each call crosses an inspected VPN tunnel twice, congratulations: you have invented interactive archaeology.

    Put workloads near users and data where possible.

    Reduce chatty protocols.

    Measure.

    Physics is an unusually stubborn stakeholder.

    38. Logs Are Part of the Product

    When designing systems, architects often draw happy paths.

    User authenticates.

    Request processed.

    Response returned.

    Also design:

    failure,

    timeout,

    retry,

    rejection,

    partial completion,

    and investigation.

    Can Operations determine what happened?

    Can Security reconstruct an event?

    Can Support correlate a user complaint?

    Can you trace a transaction across services?

    If not, the system will eventually fail invisibly.

    Invisible failures are especially popular with executives.

    39. Time Is Infrastructure

    Clock synchronisation matters.

    Certificates care about time.

    Kerberos cares about time.

    Distributed logs care about time.

    Databases care about time.

    Auditors care intensely about time.

    When two systems disagree by seven minutes, debugging becomes metaphysics.

    Architect time.

    Nobody will thank you.

    This is normal.

    40. The Most Dangerous Box Is “Other”

    Whenever a diagram contains:

    OTHER SYSTEMS

    ask what they are.

    Likewise:

    External Users.

    Third Parties.

    Legacy Interfaces.

    Partner Network.

    Shared Services.

    Miscellaneous Data Sources.

    Every vague box contains future incidents.

    Ambiguity is where dependencies breed.

    41. Architecture Is Mostly Asking Embarrassing Questions Early

    Who owns this?

    How many users?

    Where is the data?

    What happens if it fails?

    Who supports it?

    What does the licence permit?

    Why are we doing this?

    What happens when the contract ends?

    What happens if the supplier disappears?

    How do we recover?

    Has anyone tested that?

    Who pays?

    What does “real time” mean?

    Who approved the risk?

    These are not glamorous questions.

    They are extremely valuable.

    42. You Will Be Asked to Approve Things You Did Not Design

    A project will arrive three days before go-live.

    They will say:

    “We just need Architecture sign-off.”

    Do not sign.

    Review it.

    If it is acceptable, say so.

    If risks exist, state them.

    If information is missing, state that.

    Never allow architectural approval to mean:

    “An architect was present near the end.”

    Your name will remain attached to the decision long after everyone else has moved on.

    43. Never Become the Diagram Monkey

    You are not there merely to make Visio attractive.

    Although attractive diagrams help.

    You are there to:

    clarify,

    structure,

    challenge,

    model,

    analyse,

    trade off,

    communicate,

    and decide.

    If every meeting ends with:

    “Can you update the diagram?”

    ask whether you are performing architecture or desktop publishing.

    Then update the diagram anyway.

    Because apparently the arrows are the wrong colour.

    44. Architecture Is Social Engineering Without the Phishing

    Technical decisions happen through people.

    You will need to persuade:

    developers,

    security,

    operations,

    programme managers,

    vendors,

    finance,

    procurement,

    and executives.

    Being technically correct is insufficient.

    You must explain consequences in language each audience understands.

    To engineers:

    failure modes.

    To finance:

    cost.

    To executives:

    risk and outcome.

    To operations:

    supportability.

    To security:

    control.

    To programme managers:

    dependency and schedule.

    The architecture does not exist until enough people understand it to build and operate it.

    45. Do Not Fall in Love With Your Design

    You will create something elegant.

    Then a requirement will appear that ruins it.

    This is painful.

    Do not defend the design because it is yours.

    Architecture is not sculpture.

    If the constraints change, change the solution.

    The best architects abandon their favourite ideas faster than mediocre architects defend theirs.

    46. Simplicity Must Be Defended

    Complexity arrives automatically.

    Every stakeholder adds one requirement.

    Every vendor adds one component.

    Every risk adds one control.

    Every integration adds one interface.

    Nobody owns total complexity except Architecture.

    Therefore say:

    “No, we don’t need another platform.”

    “We can reuse this service.”

    “This component adds no value.”

    “Remove that hop.”

    “Why are there two databases?”

    Simplicity is rarely created.

    It is excavated.

    47. There Is No Perfect Architecture

    There are only trade-offs.

    Availability versus cost.

    Security versus usability.

    Consistency versus latency.

    Speed of delivery versus technical debt.

    Standardisation versus flexibility.

    Build versus buy.

    Centralisation versus autonomy.

    The architect who claims to have eliminated trade-offs has usually hidden them.

    Find them.

    Write them down.

    Make the organisation choose consciously.

    That is much of the job.

    48. The Best Architecture Document Is the One Someone Uses

    A 180-page solution design nobody reads is less valuable than five pages everybody understands.

    Documentation should answer questions.

    What are we building?

    Why?

    How does it work?

    What depends on what?

    How is it secured?

    How does it fail?

    How is it operated?

    What decisions were made?

    Where are the risks?

    Write enough.

    Not everything.

    Nobody has ever been saved during a Severity One incident because the architecture document had an excellent glossary.

    49. Eventually You Become the Person People Ask

    Years pass.

    You learn the estate.

    You learn the politics.

    You know which standards matter.

    You know which vendor diagrams lie.

    You know which legacy systems genuinely cannot be touched.

    A project manager will eventually enter a meeting and say:

    “We need Peter because he knows how all this joins together.”

    This is flattering.

    It is also a warning.

    Write things down.

    Teach other architects.

    Do not become another undocumented dependency.

    The enterprise already has enough of those.

    50. The Final Zen

    Solution Architecture is not the art of designing perfect systems.

    It is the practice of making imperfect decisions under incomplete information while twenty-seven people have different definitions of success.

    You will rarely have enough time.

    You will never have complete requirements.

    The technology will change.

    The organisation will change.

    The budget will change.

    Someone will acquire another company during implementation.

    A vendor will rename the product halfway through your document.

    Yet the architect continues.

    Ask the awkward question.

    Draw the useful diagram.

    Find the hidden dependency.

    Expose the assumption.

    Quantify the risk.

    Simplify the design.

    Record the decision.

    And when somebody finally asks:

    “So, is this architecture future-proof?”

    Do not laugh.

    Look thoughtful.

    Then say:

    “It gives us a controlled path for future change.”

    This sounds wise.

    More importantly, it does not promise anything impossible.

    You have achieved architectural enlightenment.

  • The Tao of Solution Assurance

    Solution Assurance is the ancient corporate discipline of examining a design after everybody important has already committed to it, identifying several serious risks, documenting them carefully, and then watching the programme proceed exactly as before.

    It is sometimes confused with governance.

    This is unfair to governance.

    Governance occasionally stops things.

    Solution Assurance exists in the delicate philosophical territory between architecture, risk management, quality control and ritual sacrifice.

    Its purpose is simple:

    To provide confidence.

    Not necessarily correctness.

    Not necessarily safety.

    Certainly not certainty.

    Confidence.

    Confidence is extremely important because executives become nervous when presented with reality.

    A red RAG status creates anxiety.

    A green RAG status creates confidence.

    An amber RAG status creates meetings.

    Therefore the experienced Solution Assurance practitioner understands the first teaching:

    The colour is not the risk.

    The colour is what management can emotionally tolerate this week.

    1. Assurance Begins After the Decision

    In theory, assurance should begin early.

    In practice, the sequence is:

    1. Vendor selected.
    2. Contract negotiated.
    3. Budget announced.
    4. Programme mobilised.
    5. Launch date communicated.
    6. Solution designed.
    7. Assurance invited.

    This is efficient because it prevents inconvenient technical facts from interfering with commercial momentum.

    You will receive an email.

    Subject: Architecture Assurance – Urgent

    It will say:

    Hi,

    Could you please provide assurance against the attached design? We are seeking approval at Thursday’s board.

    This should be straightforward as the solution has already been reviewed extensively by the supplier.

    Thanks.

    It is Wednesday afternoon.

    The attachment is 184 pages.

    Half the diagrams are unreadable.

    The security section says:

    Security requirements will be confirmed during implementation.

    The implementation began three months ago.

    Welcome to Assurance.

    2. “Assured” Does Not Mean “Good”

    A solution can be:

    technically questionable,

    operationally immature,

    poorly documented,

    expensive,

    dependent on unsupported technology,

    and nevertheless assured.

    This is because assurance is rarely a binary judgement.

    It is more sophisticated.

    You can say:

    Assured subject to conditions.

    This is one of the great phrases of corporate civilisation.

    It means:

    We have identified several reasons why this may go badly wrong, but everyone has a steering committee in ten minutes.

    The conditions are then recorded.

    The programme acknowledges them.

    The programme continues.

    Six months later, during the incident review, someone asks:

    “Why wasn’t this risk identified?”

    You produce the assurance report.

    Page 17.

    Risk 4.

    Highlighted.

    Bold.

    Rated Red.

    The room becomes quiet.

    This is the closest Solution Assurance gets to physical pleasure.

    3. Never Ask “Is It Secure?”

    This is not a useful question.

    Everything is secure in PowerPoint.

    Ask:

    How is authentication implemented?

    How are privileged accounts controlled?

    Where are secrets stored?

    What is logged?

    Who can access the logs?

    How is encryption implemented?

    Who owns the keys?

    How are certificates rotated?

    What happens if identity is unavailable?

    How are vulnerabilities patched?

    What is exposed externally?

    How is compromise detected?

    The supplier will respond:

    “We follow industry best practice.”

    Ask which one.

    They will become irritated.

    This is progress.

    4. The Supplier Has Assured Itself

    One of the more delightful developments in enterprise technology is supplier-provided assurance.

    The supplier has reviewed the supplier’s design of the supplier’s product and concluded that the supplier recommends it.

    Excellent.

    A presentation will contain:

    PROVEN ARCHITECTURE

    ENTERPRISE GRADE

    SECURE BY DESIGN

    HIGHLY AVAILABLE

    SCALABLE

    There may be a Gartner logo.

    There will certainly be clouds.

    Your job is to ask:

    “Where is customer data stored?”

    The account manager will say:

    “In our secure cloud.”

    You ask:

    “Which country?”

    There will be a pause.

    “Within our global infrastructure.”

    This is not a country.

    Continue.

    5. Evidence Is Better Than Reassurance

    Programmes enjoy reassurance.

    “We’ve tested it.”

    “The supplier is confident.”

    “Security has been involved.”

    “Operations are comfortable.”

    “The business is happy.”

    None of these are evidence.

    Ask:

    Where are the test results?

    Where is the threat model?

    Where is the support model?

    Where is the capacity forecast?

    Where is the recovery test?

    Where is the data-flow diagram?

    Where is the licensing assessment?

    Who signed off the operational acceptance?

    At this point someone will accuse you of being “very detailed.”

    That is because evidence is offensive when reassurance was expected.

    6. The Red Flag Is Usually in the Footnote

    Executive summaries are optimistic.

    Detailed sections are cautious.

    Footnotes are where truth goes to hide.

    The first page may say:

    The proposed solution meets all strategic requirements.

    Page 73 may say:

    Note: current design does not provide automatic failover between sites.

    Page 106:

    Backup integration is outside current scope.

    Page 129:

    Existing identity platform is not formally supported.

    Page 151:

    Performance testing has not yet been scheduled.

    Page 162:

    Licensing position remains subject to vendor clarification.

    The executive summary will remain green.

    Your job is to make page 162 somebody’s problem.

    7. “Out of Scope” Is a Magical Phrase

    If something difficult cannot be solved, it can often be moved out of scope.

    Disaster recovery?

    Out of scope.

    Operational monitoring?

    Out of scope.

    Data migration reconciliation?

    Out of scope.

    Decommissioning?

    Out of scope.

    Security hardening?

    Phase Two.

    When enough things are out of scope, the project becomes very simple.

    It may no longer deliver a usable service.

    But the project is beautifully controlled.

    Solution Assurance should always ask:

    “Out of scope for whom?”

    If the answer is:

    “Operations will pick it up later.”

    You have discovered a landfill site.

    8. Phase Two Does Not Exist

    There is Phase One.

    Then there is production.

    Phase Two is a spiritual concept.

    It contains:

    technical debt,

    nice-to-have security controls,

    automation,

    performance optimisation,

    proper monitoring,

    full documentation,

    removal of temporary accounts,

    legacy decommissioning,

    and everything everyone promised would happen after go-live.

    Phase Two is funded from next year’s budget.

    Next year arrives.

    The programme is closed.

    A new transformation initiative begins.

    Phase Two becomes “legacy remediation.”

    Eventually it becomes somebody’s audit finding.

    9. RAG Status Is Applied Psychology

    Red means:

    Something is wrong.

    Amber means:

    Something is wrong but we are still discussing it.

    Green means:

    Nobody senior has asked the right question yet.

    There is also:

    Amber-Green.

    This is corporate synaesthesia.

    Amber-Green means:

    There are substantial concerns but the programme director has a board meeting.

    There is also:

    Green with commentary.

    This means:

    Please read the commentary.

    Nobody reads the commentary.

    10. Assurance Meetings Are Linguistic Combat

    A typical assurance meeting contains:

    the architect,

    the programme manager,

    the delivery lead,

    the supplier,

    security,

    operations,

    and somebody from PMO who has never spoken but is taking frighteningly good notes.

    You ask:

    “What happens if the database becomes unavailable?”

    Supplier:

    “The platform is highly resilient.”

    You:

    “How?”

    Supplier:

    “It uses a clustered architecture.”

    You:

    “What is the failover time?”

    Supplier:

    “It is designed for rapid recovery.”

    You:

    “What is the tested failover time?”

    Supplier:

    “We haven’t tested that scenario yet.”

    Programme Manager:

    “Is this really necessary for this stage?”

    You:

    “Yes.”

    Programme Manager:

    “Can we record it as an action?”

    This is how architecture becomes archaeology.

    11. The Action Log Is Where Risks Go to Hibernate

    Actions are useful.

    Until there are 147 of them.

    Every uncomfortable issue can be transformed into an action.

    Action 37: Confirm DR capability.

    Owner: Supplier.

    Due date: Friday.

    Friday arrives.

    Status:

    Open – awaiting supplier input.

    Next week:

    Open – supplier investigating.

    Next month:

    Open – to be addressed post go-live.

    Three months later:

    Closed – transferred to BAU.

    Nothing has actually happened.

    But the action is closed.

    Governance has achieved transcendence.

    12. “Accepted Risk” Requires Someone to Accept It

    Teams sometimes say:

    “The business has accepted the risk.”

    Ask:

    “Who?”

    A silence follows.

    “The business.”

    The business is not a person.

    The business does not have an email address.

    The business cannot attend court.

    The business cannot explain itself to an auditor.

    Risk acceptance requires an accountable individual with authority to accept the consequence.

    Find that person.

    Make them understand the risk.

    Get the acceptance recorded.

    You will be accused of bureaucracy.

    Ignore this.

    The same people will become intensely interested in documentation after the failure.

    13. Risk Language Must Describe Consequences

    Bad risk:

    There is a risk that the solution may not be resilient.

    Excellent.

    Meaningless.

    Better:

    Failure of the primary database node may cause complete service outage because automatic failover has not been implemented or tested. Recovery is dependent on manual intervention by the supplier. Estimated recovery time is unknown.

    Now management becomes interested.

    Especially the word unknown.

    Executives dislike unknown.

    This is useful.

    14. Assurance Is Not About Catching People Out

    Mostly.

    The objective is to expose uncertainty before uncertainty becomes outage.

    That requires uncomfortable questions.

    Not theatrical aggression.

    Do not enter a review saying:

    “This design is rubbish.”

    Ask:

    “What requirement led to this pattern?”

    Sometimes there is a good answer.

    Sometimes the answer is:

    “The vendor said so.”

    Sometimes:

    “We’ve always done it this way.”

    Sometimes:

    “We copied another project.”

    Sometimes nobody knows.

    The last one is surprisingly common.

    15. Architecture Debt Must Be Visible

    Every programme accumulates compromises.

    Temporary integration.

    Manual failover.

    Unsupported browser.

    Shared service account.

    Single region deployment.

    Missing monitoring.

    One firewall exception.

    One more firewall exception.

    A third firewall exception to make the first two work.

    Each seems reasonable alone.

    Together they form a service held together by hope.

    Assurance should aggregate these.

    A solution with twenty individually tolerable risks may be collectively intolerable.

    Programmes dislike this idea.

    They prefer risks in separate rows.

    Separate rows look smaller.

    16. Never Allow “Known Limitation” to Become “Normal”

    Known limitation is often corporate language for:

    We know it is broken.

    Examples:

    “The service requires restart every Sunday.”

    “The interface occasionally duplicates messages.”

    “Users must clear browser cache after upgrades.”

    “Failover can cause data inconsistency.”

    “Support requires local administrator access.”

    These may be genuine limitations.

    But they must have consequences, ownership and remediation.

    Otherwise three years later someone will say:

    “That’s just how it works.”

    This is how defects become culture.

    17. Test Evidence Is More Valuable Than Test Plans

    A test plan says:

    “We intend to test resilience.”

    Test evidence says:

    “We unplugged it and watched what happened.”

    Prefer the second.

    Ask for:

    load test results,

    failover evidence,

    restore evidence,

    penetration test findings,

    security scans,

    integration results,

    user acceptance outcomes.

    Never be impressed by:

    TESTING COMPLETE

    Ask:

    “What failed?”

    If the answer is:

    “Nothing.”

    Be suspicious.

    Either the system is extraordinary or the testing was decorative.

    18. If Nobody Tested Failure, Nobody Tested the System

    Successful transactions are pleasant.

    Failures are architecture.

    Disconnect the network.

    Kill the service.

    Expire the token.

    Remove the DNS record.

    Fill the disk.

    Lose the node.

    Corrupt the message.

    Throttle the API.

    Break authentication.

    Then observe.

    Systems reveal their real architecture when something goes wrong.

    19. Operational Acceptance Is Not “Ops Were Invited”

    Programs sometimes claim Operations has accepted a service because someone from Operations attended a meeting.

    That is not acceptance.

    Ask:

    Do they have monitoring?

    Runbooks?

    Access?

    Training?

    Escalation paths?

    Support contracts?

    Backup procedures?

    Recovery procedures?

    Capacity information?

    Known-error records?

    Service ownership?

    If not, Operations has not accepted the service.

    They have merely witnessed its birth.

    20. The Handover Document Is Usually Fiction

    A project handover document says:

    “BAU support will be provided by Infrastructure Services.”

    Infrastructure Services says:

    “Never heard of it.”

    Project:

    “They were on the distribution list.”

    Infrastructure:

    “So was Catering.”

    Project:

    “We assumed they were aware.”

    Operations:

    “We assumed you had a support contract.”

    Supplier:

    “Support contract?”

    Silence.

    The solution is now live.

    Congratulations.

    21. Security Exceptions Breed

    One exception is temporary.

    Two exceptions are pragmatic.

    Three exceptions are architecture.

    Assurance must track:

    what control is bypassed,

    why,

    who approved it,

    what compensating controls exist,

    when the exception expires.

    Otherwise the exception survives.

    After several years it becomes:

    legacy security model.

    People will then be afraid to remove it.

    22. Data Sovereignty Is Not Where the Salesman Lives

    Ask where data resides.

    The vendor says:

    “UK hosted.”

    Ask:

    Backups?

    Logs?

    Telemetry?

    Support access?

    Disaster recovery?

    Sub-processors?

    AI services?

    Analytics?

    Suddenly the United Kingdom becomes geographically flexible.

    Assurance exists partly to continue asking after the first comforting answer.

    23. “Encrypted” Is the Beginning of the Question

    Encrypted where?

    At rest?

    In transit?

    Client-side?

    Server-side?

    Which algorithm?

    Who holds the keys?

    Can the provider decrypt it?

    Can administrators?

    How are keys rotated?

    What happens during recovery?

    If the vendor says:

    “AES-256.”

    Do not applaud.

    AES-256 is not an architecture.

    It is a cipher.

    24. Backup Is Not Resilience

    A backup protects data.

    It does not automatically protect:

    availability,

    configuration,

    identity,

    network connectivity,

    integration state,

    DNS,

    certificates,

    secrets,

    or the ability of anyone to remember how to restore the bloody thing.

    Assurance should ask for recovery, not backup.

    “How quickly can you restore the service from nothing?”

    Watch confidence decrease.

    This is healthy.

    25. DR Documentation Is Often Fantasy Literature

    The DR plan may say:

    In the event of primary site loss, service will fail over to secondary site.

    Ask:

    Who performs the failover?

    “How?”

    “Using the DR process.”

    “Where is that?”

    “In the DR document.”

    You are already reading the DR document.

    This is recursive resilience.

    Continue until someone admits Gary knows how.

    26. Capacity Is Not “Scalable”

    Supplier:

    “The platform scales automatically.”

    You:

    “To what?”

    Supplier:

    “As demand increases.”

    You:

    “What is the tested maximum?”

    Supplier:

    “That depends on configuration.”

    You:

    “What configuration are we buying?”

    Supplier:

    “We can confirm that during implementation.”

    Programme:

    “Can we move on?”

    No.

    We cannot.

    27. Performance Requirements Need Numbers

    “Fast.”

    No.

    “Responsive.”

    No.

    “Near real time.”

    Absolutely not.

    Use:

    95th percentile response under two seconds.

    10,000 concurrent users.

    500 transactions per second.

    Batch completion before 06:00.

    Data propagation within 30 seconds.

    Numbers can be tested.

    Adjectives can only be discussed.

    28. Monitoring Is Not a Dashboard Nobody Watches

    A colourful dashboard is not monitoring.

    Ask:

    Who receives alerts?

    What thresholds exist?

    What constitutes service degradation?

    Who responds?

    How quickly?

    Are alerts tested?

    Are dependencies monitored?

    Can users be affected while every infrastructure metric remains green?

    The answer to the last question is usually yes.

    Infrastructure can be perfectly healthy while the application is utterly fucked.

    This is why service monitoring exists.

    In theory.

    29. Observability Is Not Logging Everything

    Modern systems can produce terrifying quantities of logs.

    This is not observability.

    Observability means being able to answer:

    What happened?

    Where?

    When?

    To whom?

    Why?

    Across which components?

    A petabyte of JSON nobody can correlate is merely expensive confusion.

    30. Compliance Is Not Security

    A system may pass an audit and still be insecure.

    A system may be secure and still fail compliance.

    These overlap.

    They are not identical.

    Checkbox security produces magnificent evidence packs.

    Attackers do not generally read them.

    31. The Penetration Test Is Not an Exorcism

    A penetration test does not bless the system.

    It tests a defined scope at a point in time.

    Ask:

    What was excluded?

    Was authentication tested?

    APIs?

    Internal interfaces?

    Cloud configuration?

    Privilege escalation?

    Mobile clients?

    Infrastructure?

    Was the production configuration actually tested?

    The executive summary will say:

    No critical findings.

    Page twelve may contain eight High findings.

    Read page twelve.

    32. “Low Risk” Findings Can Combine Into a High Risk System

    Weak password policy.

    Verbose error messages.

    Excessive permissions.

    Unrestricted outbound connectivity.

    Poor logging.

    Individually low or medium.

    Together:

    Excellent afternoon for an attacker.

    Assurance must think in systems.

    Risk registers often do not.

    33. Dependencies Are Where Assurance Earns Its Keep

    The application may be resilient.

    But it depends on:

    DNS,

    identity,

    network,

    API gateway,

    certificate authority,

    message broker,

    database,

    storage,

    third-party payment gateway,

    and an ancient file transfer server in Swindon.

    Ask what happens when each fails.

    Someone will say:

    “That is outside our solution boundary.”

    Failure does not respect solution boundaries.

    34. The Boundary Diagram Is a Negotiation

    Projects draw solution boundaries partly to define ownership.

    Unfortunately, the customer experience does not care.

    If your beautifully assured application cannot function because the corporate proxy is unavailable, the service is unavailable.

    Users will not say:

    “Fortunately, the application component remained compliant with its architecture.”

    They will say:

    “It doesn’t fucking work.”

    Assure the service.

    Not just the boxes.

    35. Third Parties Are First-Class Risks

    Vendor:

    “We use a specialist third party for that.”

    Assurance:

    “Who?”

    Vendor:

    “That information is commercially sensitive.”

    Assurance:

    “They process our data.”

    Vendor:

    “We can provide details under NDA.”

    Good.

    Continue.

    Subcontracting does not outsource accountability.

    It merely lengthens the incident bridge.

    36. Exit Strategy Is Architecture

    Every supplier relationship ends.

    Eventually.

    Ask:

    How do we retrieve data?

    In what format?

    How long does extraction take?

    What does it cost?

    Can another provider consume it?

    What happens to backups?

    When is data deleted?

    How do we verify deletion?

    What happens if the supplier becomes insolvent?

    Procurement may consider these gloomy questions.

    They are.

    So is divorce law.

    Still useful.

    37. Licensing Assurance Exists Because Lawyers Enjoy Ambiguity

    Technical teams think software licensing is about software.

    It is actually about contractual nouns.

    Installed.

    Used.

    Accessed.

    Processor.

    Core.

    Named user.

    Authorised user.

    Indirect use.

    Backup.

    Failover.

    Test.

    Development.

    Virtualisation.

    Cloud mobility.

    Multiplexing.

    Ask whether the architecture changes licence exposure.

    If nobody knows, record the uncertainty.

    Do not allow:

    “The account manager said it was fine.”

    The account manager will not attend the audit.

    38. Cost Assurance Must Include Success

    Projects estimate cost at average load.

    Success changes this.

    More users.

    More storage.

    More API traffic.

    More logs.

    More backups.

    More egress.

    More licences.

    Ask:

    “What does this cost if adoption is twice forecast?”

    If the answer is:

    “That would be a good problem to have.”

    You have found someone who does not pay cloud bills.

    39. Assumptions Are Risks Wearing Fake Moustaches

    Designs contain assumptions.

    “Existing WAN has sufficient capacity.”

    “Users have modern browsers.”

    “Partner API supports required volumes.”

    “Directory contains accurate attributes.”

    “Legacy system will remain available.”

    “We assume 20% annual growth.”

    Assumptions should be validated.

    Otherwise they are risks disguised grammatically.

    40. “To Be Confirmed” Has an Expiry Date

    TBC is acceptable early.

    Later, it becomes dangerous.

    At design review:

    TBC.

    At build:

    TBC.

    At test:

    TBC.

    At go-live:

    TBC.

    In the incident report:

    Root cause.

    Every TBC should have:

    an owner,

    a due date,

    and consequences if unresolved.

    Otherwise you are manufacturing uncertainty professionally.

    41. Decision Ownership Matters

    Who decided to accept single-region hosting?

    Who decided not to implement automated recovery?

    Who approved the unsupported integration?

    Who accepted the licence risk?

    The answer cannot be:

    “The programme.”

    Programs do not go to disciplinary hearings.

    People do.

    Architecture decisions require named ownership.

    This tends to improve decision quality dramatically.

    42. Escalation Is Not Failure

    Assurance practitioners sometimes avoid escalation because they do not want to appear obstructive.

    This is cowardice wearing stakeholder-management clothing.

    If a material risk exceeds your authority, escalate it.

    Calmly.

    With evidence.

    Without drama.

    Then let the accountable person decide.

    Your job is not to win.

    Your job is to ensure the decision is conscious.

    43. A Waiver Is Not a Magic Spell

    Sometimes a programme requests an architectural waiver.

    Fine.

    A waiver should state:

    what standard is being waived,

    why,

    risk created,

    compensating controls,

    owner,

    expiry.

    A permanent waiver is not a waiver.

    It is a new standard nobody has admitted exists.

    44. Mature Assurance Knows When to Stop

    Not every system requires military-grade resilience.

    Not every application needs active-active deployment across continents.

    Not every dataset needs hardware-backed encryption keys rotated hourly.

    Assurance must be proportionate.

    Risk depends on consequence.

    The lunch-menu application can occasionally fail.

    The air-traffic control system should perhaps have stronger aspirations.

    Apply judgement.

    Otherwise assurance itself becomes the risk.

    45. The Assurance Practitioner Must Understand Delivery

    A reviewer who has never built anything is dangerous.

    They may demand:

    perfect documentation,

    zero technical debt,

    complete automation,

    full resilience,

    maximum security,

    infinite scalability,

    and delivery by Friday.

    Architecture is trade-off.

    Assurance must understand trade-off.

    Ask whether risk is conscious and proportionate.

    Do not demand utopia.

    Utopia is not supportable.

    46. “Industry Best Practice” Is Often Consultancy Incense

    Whenever someone invokes best practice, ask:

    For this context?

    For this scale?

    For this threat model?

    For this regulatory environment?

    For this operating model?

    A multinational bank and a village museum do not necessarily require identical controls.

    If your assurance framework says they do, the framework is the thing requiring assurance.

    47. Templates Are Useful Until They Replace Thinking

    Assurance templates provide consistency.

    Good.

    But the template does not know what is important.

    A reviewer may spend twenty minutes checking whether every heading is populated while missing the fact that the system has no backup.

    This is called compliance theatre.

    Never confuse completeness of form with completeness of thought.

    48. Architecture Boards Attract PowerPoint

    A board pack may contain:

    executive summary,

    strategic alignment,

    business outcomes,

    capability mapping,

    technology principles,

    risk summary,

    implementation roadmap.

    Excellent.

    Ask:

    “What port does it use?”

    Nobody knows.

    This is not because ports are strategically important.

    It is because detail reveals whether anybody has actually designed anything.

    Move between levels.

    That is the job.

    49. The Best Assurance Question Is Often “Show Me”

    “We have backups.”

    Show me a restore.

    “We monitor it.”

    Show me an alert.

    “We tested failover.”

    Show me the evidence.

    “Operations accepted it.”

    Show me the acceptance.

    “The vendor supports this.”

    Show me where.

    “Security approved it.”

    Show me the decision.

    “Licensing is covered.”

    Show me the entitlement.

    This phrase eliminates approximately seventy percent of enterprise bullshit.

    Use responsibly.

    50. Assurance Must Survive Executive Pressure

    Someone important will eventually say:

    “Can you just sign this off?”

    No.

    You can review it.

    You can assure it.

    You can identify conditions.

    You can record risks.

    You cannot transform uncertainty into certainty because a meeting starts at 14:00.

    If necessary say:

    “I can provide assurance based on the evidence available.”

    This is polite.

    It also creates a clear boundary around reality.

    51. Beware the Urgent Executive Exception

    There is always one.

    “We need to bypass the normal process.”

    Why?

    “Business-critical.”

    Everything is business-critical shortly before a board meeting.

    Urgency may justify accelerated assurance.

    It does not justify no assurance.

    Fast decisions need clearer risk statements, not fewer.

    52. The Incident Will Reopen Every Argument

    After an outage, people become historians.

    Someone will say:

    “Nobody could have predicted this.”

    Check your assurance report.

    Often, someone did.

    Another will say:

    “This was an unforeseeable dependency.”

    Check the architecture review.

    It may be listed.

    A third will say:

    “We understood the risk.”

    Ask for acceptance.

    Silence.

    This is why documentation matters.

    Not for blame.

    For organisational memory.

    Blame is merely a side effect.

    53. Lessons Learned Are Usually Lessons Observed

    The post-incident review produces:

    Improve documentation.

    Engage stakeholders earlier.

    Strengthen testing.

    Clarify ownership.

    Review monitoring.

    These lessons have appeared in every enterprise incident review since approximately 1987.

    A lesson is not learned because it is written.

    It is learned when behaviour changes.

    Otherwise it is merely rediscovered wisdom.

    54. Assurance Findings Need Closure Criteria

    Finding:

    “Improve monitoring.”

    Impossible to close meaningfully.

    Better:

    “Implement synthetic transaction monitoring for customer login and payment workflows, with alerts routed to 24×7 support and tested before production release.”

    Now closure can be evidenced.

    Specificity is the enemy of ceremonial governance.

    55. Do Not Let the Programme Mark Its Own Homework

    Programme:

    “We have resolved Finding 12.”

    Assurance:

    “How?”

    Programme:

    “We discussed it.”

    No.

    Resolution requires evidence.

    Otherwise the student has written:

    Corrected

    in the margin of their own exam paper.

    56. Some Risks Should Stop Go-Live

    This will upset people.

    Good.

    Examples may include:

    known exploitable security defects,

    no viable recovery capability for a critical service,

    unresolved data-loss risk,

    unsupported production configuration,

    absence of required regulatory controls,

    major capacity failure under expected load.

    A go-live date is not a law of physics.

    Sometimes the correct assurance outcome is:

    “No.”

    This is rare.

    It should remain available.

    Otherwise assurance is merely decorative.

    57. “Conditional Go-Live” Means Conditions

    Do not approve go-live subject to ten conditions which cannot realistically be completed after go-live.

    That is not conditional approval.

    That is denial with poor emotional resilience.

    If the condition matters before production, require it before production.

    If it can genuinely follow, assign:

    owner,

    date,

    risk,

    escalation.

    Words must mean things.

    This principle is surprisingly controversial.

    58. Assurance Should Reduce Surprise

    Perfect systems do not exist.

    Incidents will happen.

    The objective is not zero failure.

    It is fewer stupid surprises.

    You should not discover after go-live that:

    the backup never worked,

    the supplier does not provide 24×7 support,

    the application cannot run in DR,

    the licence excludes virtualisation,

    the logs contain personal data,

    the database has a 2TB limit,

    the certificate renewal is manual,

    or the only administrator is on maternity leave.

    These are not black swans.

    They are pigeons standing directly in front of you.

    59. The Highest Form of Assurance Is Boring Production

    No P1 incidents.

    Predictable patching.

    Tested recovery.

    Clear ownership.

    Known capacity.

    Controlled change.

    Understandable costs.

    Useful monitoring.

    Boring.

    Architects sometimes dislike boring systems.

    Operations loves them.

    Customers rarely complain that their transaction completed without architectural excitement.

    Boring is underrated.

    60. The Final Tao

    The novice believes Solution Assurance exists to approve solutions.

    The experienced practitioner knows it exists to expose decisions.

    The master understands that the organisation will sometimes make the wrong decision anyway.

    Your role is therefore not omnipotence.

    It is clarity.

    Make assumptions visible.

    Make dependencies visible.

    Make consequences visible.

    Make ownership visible.

    Ask for evidence.

    Challenge optimism.

    Separate confidence from fact.

    Do not allow green status to erase red engineering.

    Do not allow urgency to repeal physics.

    Do not allow governance to replace judgement.

    And above all, never write:

    “No significant architectural risks identified.”

    unless you have looked very, very hard.

    Because six months later, at 02:17 on a Sunday morning, when production is down, the backup is corrupt, DNS is pointing at the wrong data centre, the supplier’s support desk is closed, the certificate expired yesterday and nobody can remember who owns the service, somebody will find your assurance report.

    They will scroll to the final page.

    They will read your name.

    And they will ask the oldest question in enterprise architecture:

    “Who the fuck signed this off?”

    At that moment, enlightenment is achieved.

    Usually by someone else.

  • The IT Department Survival Guide for New Starters

    Welcome to IT.

    You have been recruited because the organisation believes you possess valuable technical skills, sound judgement and the ability to remain calm under pressure.

    Within three weeks you will discover that your actual role is to explain why a printer cannot be fixed by changing somebody’s password.

    This guide exists to help.

    1. Learn the First Law of IT

    Everything is your fault.

    The payroll system is slow.

    IT.

    The meeting room is cold.

    IT.

    A customer cannot remember their username.

    IT.

    The coffee machine says DESCALE.

    IT.

    Karen has deleted an Excel workbook containing the organisation’s entire procurement strategy.

    Definitely IT.

    You may occasionally attempt to explain that Information Technology does not control plumbing, building access, furniture, catering or the weather.

    This is a beginner’s mistake.

    The user does not care which department owns the problem.

    They have found somebody wearing a headset.

    That somebody is you.

    Accept this.

    It will save time.

    2. Never Say “That Should Work”

    The gods hear this.

    You may test a system for six months.

    You may perform penetration testing, regression testing, failover testing, disaster recovery testing and a full dress rehearsal involving nineteen engineers and a conference bridge.

    The moment you tell management:

    “That should work.”

    A certificate will expire.

    Prefer:

    “We have not identified any current impediment to successful operation.”

    This means the same thing but allows considerably more room for professional retreat.

    Other useful phrases include:

    “That’s interesting.”

    Meaning:

    That is absolutely fucked.

    “I haven’t seen that before.”

    Meaning:

    I have seen this six times and none ended well.

    “Let me check the logs.”

    Meaning:

    Please stop talking while I think.

    “There may be a dependency.”

    Meaning:

    Nobody documented this bastard thing.

    “We need to understand the business impact.”

    Meaning:

    Is anyone actually using it?

    3. The Service Desk Knows Everything

    Treat the Service Desk well.

    Senior architects may understand strategy.

    Network engineers may understand routing.

    Security may understand certificates.

    Database administrators may understand things spoken of only in whispers.

    But the Service Desk knows that Finance cannot print on Thursdays because Derek installed a label printer driver in 2019.

    This is real knowledge.

    The CMDB will tell you:

    FIN-PRINT-04 — HP LaserJet — ACTIVE

    The Service Desk will tell you:

    “That’s actually the tea-room printer. FIN-PRINT-04 fell down the stairs during the office move. The one Finance uses is called Susan.”

    Believe the Service Desk.

    Buy them biscuits.

    4. Do Not Insult Legacy Systems

    You will encounter systems older than some employees.

    Do not laugh.

    A Windows Server 2008 machine under someone’s desk may turn out to process £80 million a year in direct debits.

    An Access 2003 database called:

    MASTER_FINAL_USE_THIS_ONE_v7.mdb

    may contain the only authoritative record of something legally significant.

    A beige PC in Facilities may control every door in the building.

    You will ask:

    “Why hasn’t this been replaced?”

    Everyone will look at the floor.

    You will eventually learn that replacement was proposed in:

    and 2024.

    Each programme produced a strategy.

    The old system continued running.

    Do not mock it.

    It has survived more transformation programmes than you have.

    Show respect.

    5. Never Reboot Anything Without Witnesses

    Rebooting a laptop is harmless.

    Rebooting a server is theology.

    Before restarting infrastructure, obtain:

    a ticket,

    an approved change,

    a backup,

    a rollback plan,

    a witness,

    and preferably a small priest.

    The application owner will insist that the system can be restarted at any time.

    Do not believe them.

    The moment it goes down, seventeen unidentified business processes will emerge screaming from the darkness.

    One of them will be “month end.”

    It is always month end.

    Nobody knows when month end begins.

    It appears to last approximately thirty-one days.

    6. Production Is Different

    Development works.

    Test mostly works.

    Pre-production is theoretically identical to production.

    It is not.

    Production contains:

    three undocumented firewall rules,

    a certificate installed by somebody who left in 2018,

    a manual DNS entry,

    a service account called temp_admin,

    and one scheduled task created by Keith.

    Never delete Keith’s scheduled task.

    Nobody knows what it does.

    Keith is unreachable.

    But whenever the task is disabled, Belgium stops invoicing.

    7. Learn the Hierarchy of Passwords

    There are passwords.

    There are admin passwords.

    There are service accounts.

    There are break-glass accounts.

    There are credentials stored in approved privileged-access systems.

    And there is a text file called:

    passwords.txt

    on an old shared drive.

    Security will insist this does not exist.

    Operations will know exactly where it is.

    Your objective is not to become comfortable with this.

    Your objective is to survive long enough to remove it without bringing down payroll.

    8. DNS Is Probably Involved

    When an application behaves inexplicably, someone will eventually say:

    “Could be DNS.”

    This will be offered as either wisdom or sarcasm.

    Do not dismiss it.

    DNS has caused enough damage to earn its reputation.

    Other usual suspects include:

    certificates,

    time synchronisation,

    firewalls,

    proxies,

    permissions,

    storage,

    load balancers,

    and that one forgotten NAT rule in the disaster recovery site.

    Eventually somebody will discover the actual cause was a typo.

    This does not invalidate the investigation.

    It merely completes it.

    9. Certificates Expire Only on Weekends

    Certificate expiry dates are visible months in advance.

    Monitoring systems can alert on them.

    Renewal processes can be automated.

    Owners can be assigned.

    None of this matters.

    The certificate will expire at 02:13 on a Sunday.

    A senior manager will call.

    They will say:

    “The website is down.”

    You will ask:

    “Which website?”

    They will reply:

    “The website.”

    This is all the information you are getting.

    10. Change Management Is a Ritual, Not a Guarantee

    The change form exists to answer several important questions:

    What are you changing?

    Why?

    When?

    How?

    What happens if it goes wrong?

    Who approved this madness?

    You will spend forty minutes completing it.

    The Change Advisory Board will spend four minutes discussing it.

    Someone will ask:

    “Has the business approved this?”

    You will say:

    “Yes.”

    Someone else will ask:

    “What’s the rollback?”

    You will repeat the paragraph already on screen.

    The change will be approved.

    Then, two hours before implementation, an executive will request an “urgent small amendment.”

    The small amendment will fundamentally alter the architecture.

    You will be asked whether it can be included under the existing change.

    It cannot.

    It will be.

    11. Incidents Have Gravity

    A Priority 4 incident is ignored.

    A Priority 3 gets a ticket.

    A Priority 2 gets a Teams call.

    A Priority 1 bends spacetime.

    People who have never previously shown interest in the system will materialise.

    Directors will join the bridge.

    Suppliers will join.

    Cyber will join.

    Communications will join.

    Someone from Risk will ask whether the incident is “contained.”

    Nobody knows what that means yet.

    The technical team will be trying to fix the problem while twenty-three people ask them for updates.

    Eventually somebody sensible will create two calls:

    Technical Bridge.

    Management Bridge.

    This is one of civilisation’s greatest inventions.

    On the Management Bridge, executives can ask:

    “When will it be fixed?”

    On the Technical Bridge, engineers can answer:

    “When you stop fucking asking.”

    12. Never Give a Recovery Time Unless You Mean It

    Management will request an ETA.

    They do not actually want an estimate.

    They want certainty disguised as an estimate.

    If you say:

    “Thirty minutes.”

    At twenty-nine minutes someone will ask:

    “Are we still on track?”

    At thirty-one minutes your estimate will be treated as a failed contractual commitment.

    Prefer:

    “We are working through the recovery sequence. I’ll update when we have a validated restoration point.”

    This is IT language for:

    We have no bloody idea, but Gary has found something promising.

    13. Gary Is Important

    Every department has a Gary.

    Gary may not actually be called Gary.

    He may be called Steve, Anita, Mo, Raj, Susan or Dave.

    Gary has worked there for twenty-seven years.

    Gary knows:

    why server names begin with Z,

    which fibre pair is actually live,

    why Warehouse Three must never be rebooted remotely,

    which database column is lying,

    and why the chief executive’s laptop cannot be replaced before the board meeting.

    Gary’s knowledge is undocumented because nobody has ever given Gary enough time to document it.

    Management describes this as a key-person risk.

    Then gives Gary more work.

    Identify Gary.

    Protect Gary.

    Learn from Gary.

    If Gary says:

    “Don’t touch that.”

    Do not touch that.

    14. Architecture Diagrams Are Historical Fiction

    The diagram you receive on your first day will contain:

    two firewalls,

    three servers,

    a database,

    and a cloud.

    The actual environment will contain:

    six firewalls,

    forty-seven servers,

    three clouds,

    a forgotten MPLS circuit,

    two appliances nobody owns,

    and something labelled “temporary gateway” installed eleven years ago.

    Treat architecture diagrams as archaeological evidence.

    Useful.

    Interesting.

    Not necessarily current.

    If somebody says:

    “The diagram is accurate.”

    Ask:

    “As of when?”

    This question will make you unpopular but powerful.

    15. The CMDB Is Aspirational

    Configuration Management Databases contain valuable information about assets, dependencies and ownership.

    In theory.

    In practice, you may find:

    three entries for the same server,

    an application owner who retired,

    a laptop listed as a critical production dependency,

    and a database marked “decommissioned” which is currently processing customer transactions.

    Never assume the CMDB is wrong.

    Never assume it is right.

    Think of it as a witness with a complicated relationship with truth.

    16. The Cloud Is Someone Else’s Computer, Plus Billing

    At some point someone will say:

    “We should move this to the cloud.”

    This may be correct.

    It may also mean:

    We would like the same mess, but billed monthly.

    Cloud platforms provide extraordinary capabilities.

    They also allow an enthusiastic developer to create £18,000 of infrastructure before lunch.

    Learn tagging.

    Learn budgets.

    Learn identity.

    Learn networking.

    Learn how egress charging works before somebody creates an exciting multi-cloud architecture.

    Most importantly, never accept the phrase:

    “It’ll be cheaper.”

    Ask:

    “Compared with what?”

    Watch the room become philosophical.

    17. Vendors Are Your Friends Until Renewal

    Suppliers will use phrases such as:

    strategic partnership,

    customer success,

    digital journey,

    co-innovation,

    and trusted advisor.

    These expressions mean:

    We would like another purchase order.

    A vendor account manager will remember your birthday if the contract is large enough.

    Three months before renewal, they will become intensely interested in your roadmap.

    One month after renewal, support will ask you to reproduce the problem on the latest version.

    The latest version will not support your operating system.

    This is enterprise software.

    18. Licensing Is Dark Magic

    Nobody fully understands enterprise licensing.

    Not Sales.

    Not Procurement.

    Not Legal.

    Not the vendor.

    Certainly not the auditor.

    You will encounter concepts such as:

    named user,

    concurrent user,

    processor,

    core,

    socket,

    virtual core,

    installed instance,

    running instance,

    minimum quantities,

    indirect access,

    multiplexing,

    and “authorised environment.”

    At some point you will ask:

    “How many licences do we actually need?”

    The room will go quiet.

    A consultant will be hired.

    Three months later you will receive a spreadsheet containing thirty-seven tabs and the phrase:

    Subject to contractual interpretation.

    Keep it.

    It cost £90,000.

    19. Security Will Say No

    This is partly their job.

    Do not become angry.

    Instead ask:

    “What control objective are we trying to satisfy?”

    This transforms an argument into architecture.

    Sometimes.

    Security may still say no.

    If they do, ask for the requirement in writing.

    Not because you intend to fight them.

    Because six months later somebody will ask why the project is late.

    Documentation is not bureaucracy.

    Documentation is armour.

    20. Users Lie, But Usually Innocently

    “The computer just deleted my file.”

    No, it didn’t.

    “I haven’t changed anything.”

    They have.

    “It worked yesterday.”

    Possibly.

    “I’ve restarted it.”

    They logged off.

    “The internet is down.”

    One website is unavailable.

    “My password definitely works.”

    It does not.

    Do not accuse users of lying.

    Users report their model of reality.

    Your job is to identify the gap between their model and the logs.

    Be polite.

    You may need these people later.

    Especially Payroll.

    Never antagonise Payroll.

    21. Screenshots Are Evidence

    Ask for a screenshot.

    Not:

    “What did the error say?”

    Users will paraphrase:

    “It said access or something.”

    The actual message will say:

    SQLSTATE 28000: Login failed for user svc_finance_prod.

    This distinction matters.

    Screenshots also reveal:

    the URL,

    time,

    username,

    browser,

    environment,

    and seventeen browser tabs containing information you did not ask to know.

    Be professional.

    22. Never Trust “Quick Question”

    A colleague approaching your desk with:

    “Quick question…”

    is carrying at least forty-five minutes of work.

    Common variants include:

    “Can I pick your brain?”

    “Just while you’re here…”

    “You know about networks, right?”

    “This’ll only take a second.”

    The correct response is not hostility.

    The correct response is:

    “Sure. What’s the ticket number?”

    Watch nature take its course.

    23. Projects End. Applications Do Not.

    Projects have budgets.

    Governance.

    Steering committees.

    Milestones.

    Celebrations.

    Applications have Tuesday mornings.

    The project team will deliver a shiny new system.

    Photographs will be taken.

    Cake may appear.

    Then the project closes.

    Six months later Operations asks:

    “Who supports this?”

    Silence.

    The project manager has moved to another transformation programme.

    The architect is consulting in Dubai.

    The supplier says support was not included.

    The business says IT owns it.

    IT says the business owns it.

    The application continues running.

    This is how legacy begins.

    24. Backups Are Not the Same as Recovery

    Someone will proudly tell you:

    “We back everything up.”

    Ask:

    “Have we restored it?”

    A backup that has never been restored is a theory.

    A disaster recovery plan that has never been tested is literature.

    A failover process dependent on one person remembering a password is folklore.

    Test recovery.

    Document recovery.

    Then test the document.

    Otherwise, during an incident, somebody will discover that the backup server depends on the system you are trying to restore.

    This is called enterprise architecture.

    25. Monitoring Produces Two States

    No alerts.

    Too many alerts.

    In the first state, management asks whether monitoring works.

    In the second, everyone ignores it.

    Your mission is to reach the mythical third state:

    Useful alerts.

    This involves deleting hundreds of alarms that effectively mean:

    “CPU exists.”

    If every event is critical, nothing is critical.

    This principle also applies to email marked HIGH IMPORTANCE.

    26. Meetings Reproduce

    IT meetings reproduce by mitosis.

    A project meeting identifies a technical issue.

    A technical meeting is created.

    The technical meeting identifies a security concern.

    A security workshop is created.

    The security workshop identifies a dependency.

    A dependency call is created.

    Eventually eight people attend meetings all day discussing work none of them now has time to perform.

    Protect blocks of actual working time.

    Do not apologise for this.

    Someone has to configure the thing.

    27. Teams Status Is Political

    Green means available.

    Yellow means possibly alive.

    Red means either extremely busy or eating lunch.

    Do Not Disturb means senior architect attempting to produce something before another meeting begins.

    Offline means nothing.

    Some people have been “Offline” since 2022 while responding instantly to every message.

    Do not infer reality from Teams presence.

    It is less reliable than the CMDB.

    28. Document Everything Important

    Especially decisions.

    After a meeting, write:

    “To confirm our agreed position…”

    This sentence has prevented more professional disasters than most cybersecurity products.

    Record:

    what was decided,

    who decided it,

    what assumptions were made,

    what risks were accepted,

    and who owns the next action.

    Six months later, when someone says:

    “IT recommended this architecture.”

    You can produce the email showing that IT recommended the opposite.

    Do not wave it triumphantly.

    Simply attach it.

    The effect is stronger.

    29. Never Become the Only Person Who Knows

    Being indispensable feels good.

    Until you want a holiday.

    Document your work.

    Cross-train colleagues.

    Share passwords through proper systems.

    Automate repetitive tasks.

    The goal is not to become the hero who receives calls at 03:00.

    The goal is to build systems that do not require heroes.

    Heroic IT is usually failed engineering wearing a cape.

    30. Finally: Find the People Who Actually Make Things Work

    Every IT department has formal structures.

    Architecture.

    Infrastructure.

    Applications.

    Service Management.

    Security.

    PMO.

    Data.

    Cloud.

    Workplace.

    Networks.

    Then there is the real structure.

    The network engineer who answers the phone.

    The DBA who knows the ancient application.

    The Service Desk analyst who notices patterns.

    The project manager who writes things down.

    The security architect who explains rather than obstructs.

    The desktop engineer who knows the executives.

    The developer who admits when something is broken.

    The procurement person who understands the licence.

    The administrator who knows where the contract lives.

    Find these people.

    Be useful to them.

    Do not waste their time.

    Share credit.

    Bring biscuits occasionally.

    And remember the final rule.

    One day, perhaps years from now, a nervous new starter will approach your desk.

    They will say:

    “Sorry, quick question. Everyone says you know how this works.”

    You will look at the undocumented system.

    You will look at the obsolete server.

    You will remember Gary.

    Then you will hear yourself say:

    “Right. Whatever you do, don’t reboot it.”

    And at that moment, your induction will finally be complete.

  • Problem Solving

    Problem Solving

    Simple Problems

    lets start with a game.

    Here is a 2D6 table of typical computer failures. The operator is, for whatever reason, isolated from a professional support team and the problem need to be resolved by the computer operator on their own. Either roll dice, or just pick one. Read it, have a quick think and state or write your proposed way to a resolution.

    What is your suggested fix ?

    NumberIssueDescription
    2Hard drive failureThe computer’s hard drive has failed and needs to be replaced.
    3Blue screen of deathThe computer has crashed and is displaying a blue screen with an error message.
    4Power supply failureThe computer is not turning on and the power supply needs to be replaced.
    5Virus infectionThe computer has been infected with a virus and needs to be cleaned and secured.
    6Corrupt system filesThe computer’s operating system has corrupt files that need to be repaired or reinstalled.
    7OverheatingThe computer is overheating and needs better ventilation or additional cooling.
    8Driver issuesThe computer’s drivers are outdated or missing and need to be updated or installed.
    9Network connectivity issuesThe computer is having trouble connecting to the network and needs troubleshooting.
    10Peripheral failureA peripheral device such as a printer, scanner or external hard drive is not working properly and needs to be fixed or replaced.
    11Software conflictsTwo or more pieces of software are conflicting with each other and causing issues that need to be resolved.
    12RAM failureThe computer’s RAM has failed and needs to be replaced.
    13BIOS errorThe computer’s BIOS is not working properly and needs to be reset or updated.

    Note: This is just a starting point, and you can modify the computer failures to fit your your audiences preferences. Additionally, you can use these computer failures as the basis for larger through experiments, or combine them to create more complex problem with technological challenges.

    Here are some possible resolution states for the 2d6 computer failures table I provided above.

    Most of these are obvious, so likely you nailed it ! or came real close ?

    NumberIssueDescription
    2Hard drive failureThe operator needs to replace the failed hard drive with a new one and reinstall the operating system and software.
    3Blue screen of deathThe operator needs to diagnose the error message and troubleshoot the issue, which could involve updating drivers, repairing corrupt files, or resetting the system.
    4Power supply failureThe operator needs to replace the failed power supply with a new one.
    5Virus infectionThe operator needs to run anti-virus software to detect and remove the virus, and also implement better security measures to prevent future infections.
    6Corrupt system filesThe operator needs to repair or reinstall the operating system and software.
    7OverheatingThe operator needs to improve the computer’s ventilation and cooling systems, or replace malfunctioning fans or heatsinks.
    8Driver issuesThe operator needs to download and install the latest drivers for the affected hardware, or uninstall and reinstall the drivers to resolve conflicts.
    9Network connectivity issuesThe operator needs to diagnose the connection issues and troubleshoot the network, which could involve resetting the router, checking network settings, or repairing cables.
    10Peripheral failureThe operator needs to troubleshoot the peripheral device, which could involve updating drivers, replacing cables, or resetting the device.
    11Software conflictsThe operator needs to identify the conflicting software and uninstall one of them, or update them to compatible versions.
    12RAM failureThe operator needs to identify and replace the failed RAM with new modules.
    13BIOS errorThe operator needs to diagnose the error message and either reset or update the BIOS settings.

    Note: These resolutions are just general guidelines and may vary depending on the specific scenario in the real world. As always, it’s up to the operator to decide what resolution is appropriate for their problem. In this scenario, is going to need to read the manual or go online and look up how to do thing. The operator is likely going to have to buy or get locally stocked spare parts, or will probably end up download items from the internet. Life is really much easier if your have an IT department…

    Just for completeness, lets look the computer failures table, and provide a list of incorrect solutions.

    NumberIssueDescription
    2Hard drive failureThe operator tries to recover the data from the failed hard drive, but accidentally causes further damage and data loss.
    3Blue screen of deathThe operator attempts to fix the issue without proper knowledge or tools, causing further system instability.
    4Power supply failureThe operator tries to use a power supply with incorrect specifications, leading to further hardware damage.
    5Virus infectionThe operator ignores the virus and continues to use the computer, which spreads the infection to other devices on the network.
    6Corrupt system filesThe operator tries to repair the corrupt files without proper knowledge or tools, causing further system instability.
    7OverheatingThe operator tries to cool down the computer with a makeshift solution, causing further damage or fire hazard.
    8Driver issuesThe operator installs incorrect drivers or malware disguised as drivers, leading to further system instability and security risks.
    9Network connectivity issuesThe operator disables security protocols to allow easier network access, exposing the network to security threats.
    10Peripheral failureThe operator tries to fix the peripheral device with makeshift solutions, causing further damage or electrical hazards.
    11Software conflictsThe operator deletes critical system files or registry keys, causing further system instability or even data loss.
    12RAM failureThe operator continues to use the computer with the failed RAM, leading to system crashes and data corruption.
    13BIOS errorThe operator resets the BIOS to default settings without proper knowledge or tools, causing further system instability

    Note: These incorrect solutions are meant to illustrate the potential consequences of incorrect or ill-informed actions, and may vary depending on the specific scenario and the real world. As always, it’s either up to the operator to decide what is the appropriate action and own the consequence, or ask for Support.

    So, since this is still a game, lets look at some possibly weird, bizarre, or genius solutions (depending on your viewpoint) to the computer failure problems:

    NumberIssueDescription
    2Hard drive failureThe operator uses an electromagnet to realign the failed hard drive’s magnetic fields, restoring it to a functional state.
    3Blue screen of deathThe operator realizes the error message is actually a hidden message in a secret code and deciphers it to reveal a clue to a larger conspiracy.
    4Power supply failureThe operator uses a jury-rigged system of batteries and solar panels to power the computer.
    5Virus infectionThe operator creates a “vaccine” program that infects the virus with a counter-virus, effectively neutralizing it.
    6Corrupt system filesThe operator uses a quantum computer to scan and repair the corrupt files, restoring the system to an even better state than before.
    7OverheatingThe operator immerses the computer in a liquid coolant that cools it down without the need for fans or heatsinks.
    8Driver issuesThe operator reprograms the drivers with a custom code that optimizes their performance and resolves the conflicts.
    9Network connectivity issuesThe operator uses a quantum entanglement device to establish a stable, instantaneous network connection that bypasses the need for physical cables or wireless signals.
    10Peripheral failureThe operator invents a new type of peripheral device that integrates with the computer’s neural network, allowing for direct mental control.
    11Software conflictsThe operator creates a custom middleware layer that allows incompatible software to communicate with each other without conflict.
    12RAM failureThe operator develops a new type of non-volatile RAM that never fails or degrades, and integrates it into the computer’s hardware.
    13BIOS errorThe operator uses a time machine to travel back in time and prevent the error from ever happening.

    Note: These solutions are meant to be unlikely, unsafe and may not be suitable (or possible) for your resolution. As always, it’s up to the operator to decide what solutions are appropriate.

    So to summarise here, is the table of computer problems, along with possible correct, incorrect, and the genius solutions:

    NumberComputer ProblemCorrect SolutionIncorrect SolutionGenius Solution
    2Hard drive failureReplace the failed componentsAccidentally causes further damageRealign the magnetic fields using an electromagnet
    3Blue screen of deathUpdate drivers and patchesMake the issue worseDecipher the hidden message to reveal a larger conspiracy
    4Power supply failureReplace the failed componentsUse a power supply with incorrect specificationsUse a jury-rigged system of batteries and solar panels
    5Virus infectionInstall anti-virus softwareIgnore the virusCreate a “vaccine” program that neutralizes the virus
    6Corrupt system filesRestore the system from a backupAttempt to repair corrupt filesUse a quantum computer to scan and repair the files
    7OverheatingReplace the cooling systemUse a makeshift solutionImmerse the computer in a liquid coolant
    8Driver issuesInstall correct driversInstall incorrect drivers or malwareReprogram the drivers with a custom code
    9Network connectivity issuesCheck network settings and cablesDisable security protocolsUse a quantum entanglement device to establish a stable network
    10Peripheral failureReplace the failed componentsUse makeshift solutionsCreate a new type of neural peripheral device
    11Software conflictsUninstall conflicting softwareDelete critical system filesCreate a custom middleware layer to allow compatibility
    12RAM failureReplace the failed componentsContinue to use the failed RAMDevelop a new type of non-volatile RAM
    13BIOS errorReset BIOS to default settingsReset BIOS without proper knowledgeUse a time machine to prevent the error from ever happening

    Note: Genius resolution items 5, 6, 8 are interesting, 13 is absurd ?

    Probability

    Now lets consider results in terms of the probability of success and failure of each resolution strategy.

    NumberComputer ProblemCorrect SolutionIncorrect SolutionGenius SolutionProbability of Success for Correct SolutionProbability of Failure for Incorrect SolutionProbability of Success for Genius Solution
    2Hard drive failureReplace the failed components (90%)Accidentally causes further damage (25%)Realign the magnetic fields using an electromagnet (60%)90%25%60%
    3Blue screen of deathUpdate drivers and patches (80%)Make the issue worse (20%)Decipher the hidden message to reveal a larger conspiracy (50%)80%20%50%
    4Power supply failureReplace the failed components (90%)Use a power supply with incorrect specifications (30%)Use a jury-rigged system of batteries and solar panels (70%)90%30%70%
    5Virus infectionInstall anti-virus software (95%)Ignore the virus (10%)Create a “vaccine” program that neutralizes the virus (75%)95%10%75%
    6Corrupt system filesRestore the system from a backup (90%)Attempt to repair corrupt files (40%)Use a quantum computer to scan and repair the files (70%)90%40%70%
    7OverheatingReplace the cooling system (85%)Use a makeshift solution (30%)Immerse the computer in a liquid coolant (60%)85%30%60%
    8Driver issuesInstall correct drivers (85%)Install incorrect drivers or malware (20%)Reprogram the drivers with a custom code (60%)85%20%60%
    9Network connectivity issuesCheck network settings and cables (90%)Disable security protocols (30%)Use a quantum entanglement device to establish a stable network (80%)90%30%80%
    10Peripheral failureReplace the failed components (90%)Use makeshift solutions (35%)Create a new type of neural peripheral device (65%)90%35%65%
    11Software conflictsUninstall conflicting software (85%)Delete critical system files (25%)Create a custom middleware layer to allow compatibility (70%)85%25%70%
    12RAM failureReplace the failed components (90%)Continue to use the failed RAM (20%)Develop a new type of non-volatile RAM (60%)90%20%60%
    BIOS errorReset BIOS to default settings (80%)Reset BIOS without proper knowledge (15%)Use a time machine to prevent the error from ever happening (50%)80%15%50%

    Balancing level of effort and common sense seem to be the pragmatic win scenario wit the highest probability of success. In each case there is a much lower probability of fixing the solution with the wrong solution. The Trail and Error, minimum effort or juts bodge it can still get you positive result. Interestingly, then there is a slightly lower probability of fixing with the genius solution.

    In the context of the computer problems table, the “genius” category refers to a highly unconventional and creative solution that may not be immediately apparent, but ultimately proves to be highly effective in resolving the issue at hand. These solutions are often unexpected and require out-of-the-box thinking, lateral problem solving, and a deep understanding of the underlying technical and operational principles involved. While they may be risky and unorthodox, they have a high probability of success when executed properly, and, as a by-product, can lead to significant innovation and advancement in the field of computer technology.

    The Effects of Role

    The solution givens so far are from the perspective of an operator, who in this scenario has to think for themselves out of their own experience, resulting to business process to achieve the best outcome.

    Lets Look at the typical resolution strategies for solutions for computer problems, from the perspective of different professional roles in the tech industry. It becomes apparent (and be no real surprise) that the certain roles are better able to fix certain types of problem. The Right Statements are from Role descriptions.

    Operator:

    • Right: Follows established procedures and protocols to resolve the issue.
    • Wrong: Attempts to fix the problem without proper authorization or knowledge, potentially causing further damage.
    • Genius: Applies creative problem-solving skills to quickly resolve the issue in an unexpected way.

    System Administrator:

    • Right: Uses knowledge of the system’s architecture and operation to identify and fix the problem.
    • Wrong: Attempts to apply a generic solution or workaround without understanding the specific context or technical details.
    • Genius: Develops a custom solution or tool that automates the resolution of the problem, saving time and improving efficiency.

    System Architect:

    • Right: Analyzes the system’s design and identifies the root cause of the problem, then develops a comprehensive solution.
    • Wrong: Attempts to fix the problem without fully understanding the system’s design or dependencies, potentially causing unintended consequences.
    • Genius: Identifies a previously unnoticed flaw in the system’s design, and develops an innovative solution that improves the system’s performance and reliability.

    Software Developer:

    • Right: Analyzes the source code and identifies the bug or error causing the problem, then develops a patch or update to fix it.
    • Wrong: Attempts to fix the problem without fully understanding the code or its implications, potentially introducing new bugs or issues.
    • Genius: Develops a novel algorithm or approach to solve the problem in a way that is more efficient, scalable, or robust than existing solutions.

    More Complex Problems

    Here is another 2×3 grid of unusual and hard-to-fix computer problems, along with potential categories of solutions:

    ProblemSolution – RightSolution – WrongSolution – Genius
    System-wide memory leakIdentify and fix root cause in codeRestart system or close memory-intensive appsDevelop new garbage collection algorithm to prevent future leaks
    Corrupted firmwareReinstall firmware from trusted sourceAttempt to fix code manually, causing further corruptionReverse-engineer firmware to identify and fix underlying issue
    Network-wide DDoS attackBlock attack traffic at network levelShut down network, causing disruption and downtimeDevelop dynamic traffic routing algorithm to divert attack traffic
    Encryption key compromiseGenerate new keys and revoke old onesAttempt to recover old keys from backups or hackersDevelop quantum-resistant encryption algorithm
    Hardware component failureReplace failed component with new oneAttempt to fix component with makeshift solutionDevelop firmware patch to route around failed component
    Catastrophic system meltdownRestore system from comprehensive backupsAttempt to fix individual components without understanding root causeDevelop AI-based predictive maintenance system to prevent future meltdowns

    Note that the “genius” solutions in this table are highly unconventional and may require significant expertise, resources, or innovation to implement. In most cases they are reactive and preventative, using the problem as a catalyst for lateral thinking that looks at resolution to solve the next occurrence of the problem. Needing Investment, the more likely real word scenario is the Right Solution.

    However, they have the potential to significantly improve the system’s performance, reliability, and security, and may lead to breakthroughs in the field of computer technology.

    Deep Dives & RCA

    In problem-solving, a “deep dive” is an approach that involves exploring a problem in great detail, often using a systematic and rigorous process of analysis. The goal of a deep dive is to gain a thorough understanding of the problem, its causes, and its potential solutions. This approach can be particularly useful in complex problem-solving situations, where there may be multiple interacting factors at play.

    In IT and QA, Root cause analysis (RCA) is a specific type of deep dive that focuses on identifying the underlying cause of a problem. RCA involves a systematic and structured process of analysis, often using tools such as flowcharts, fishbone diagrams, or the 5 Whys method, to trace the problem back to its root cause. The goal of RCA is to identify the fundamental reason why the problem occurred, rather than just addressing the symptoms.

    The relationship between a deep dive and root cause analysis is one of formality, the RCA is a specific structured type of deep dive that is focused on identifying the underlying cause of a problem. In other words, RCA is a deep dive that is specifically designed to identify the root cause of a problem, whereas a more general deep dive may explore a problem in greater detail without necessarily focusing on the root cause.

    By using a deep dive approach that includes RCA, problem-solvers can gain a thorough understanding of the problem and its root cause, which can help them develop more effective solutions that address the fundamental issue. This approach can be particularly useful in complex or ambiguous problem-solving situations, where the root cause may not be immediately obvious.

    Let’s use the example of the system-wide memory leak from the advanced problem table. Here is an example of how to perform root cause analysis on this problem:

    1. Define the problem: The problem is a system-wide memory leak, which is causing the system to slow down and potentially crash.
    2. Gather data: Collect data about the problem, such as error messages, log files, and system performance metrics. Identify when the problem first started, which applications or processes are affected, and whether there are any patterns or trends.
    3. Identify potential causes: Based on the data collected, identify potential causes of the problem. Some possible causes of a memory leak could include poorly optimized code, incorrect memory allocation, or an underlying hardware issue.
    4. Test and validate potential causes: Develop hypotheses based on the potential causes identified in step 3, and test them using a variety of methods, such as code analysis tools, memory profiling tools, or hardware diagnostics. Validate the hypotheses by comparing the test results to the data collected in step 2.
    5. Identify the root cause: Based on the results of the tests, identify the underlying root cause of the problem. In the case of a memory leak, the root cause might be a specific section of code that is not releasing memory properly, or a bug in the garbage collection algorithm.
    6. Develop and implement a solution: Based on the root cause analysis, develop and implement a solution that addresses the underlying problem. This might involve rewriting code, changing memory allocation settings, or updating system firmware or drivers.
    7. Monitor and verify the solution: Monitor the system after the solution has been implemented to ensure that the problem has been fully resolved. Verify that the system is running smoothly and that there are no further signs of memory leaks or other related issues.

    By following this cyclic process, experts in the tech industry can identify the root cause of complex problems and develop effective solutions that address the underlying issues, rather than just treating the symptoms.

    Root cause analysis can be a powerful tool for improving system performance, reliability, and security, and is an essential part of any effective problem-solving strategy.

    Lateral Thinking

    Lateral thinking is a problem-solving approach that involves thinking creatively and “outside the box” to find unexpected solutions. Unlike traditional linear thinking, which relies on a step-by-step process to solve problems, lateral thinking encourages a more flexible and open-minded approach that can help identify new perspectives and possibilities.

    In the context of computer problem resolution, lateral thinking can be a valuable tool for finding innovative solutions to complex or unusual problems. Instead of relying solely on established procedures and protocols, lateral thinking encourages operators, administrators, architects, and developers to think creatively and experiment with new ideas and approaches.

    For example, a lateral-thinking approach might involve exploring unusual or unconventional solutions, such as developing custom software tools, using machine learning algorithms to detect patterns or anomalies, or leveraging emerging technologies like blockchain or quantum computing. By thinking laterally, experts in the tech industry can find solutions to problems that might have otherwise been deemed unsolvable or too complex to tackle.

    Of course, lateral thinking is not a substitute for careful analysis, research, and expertise. It should be used in conjunction with other problem-solving techniques, and must be based on a solid foundation of technical knowledge and experience.

    However, when applied judiciously and with creativity, lateral thinking can help unlock new possibilities and improve the overall quality of solutions in the field of computer technology.

    Avoiding ‘Rabbit Holes’

    In logical terms, a rabbit hole refers to a line of reasoning or investigation that leads to a series of increasingly complex or tangential issues, without necessarily contributing to a resolution of the original problem. This can occur when the problem solver becomes fixated on a particular detail or idea, and begins to explore it in great depth, even if it is not directly relevant to the problem at hand. The result is that the problem solver may become lost or distracted, and may end up wasting valuable time and resources on issues that do not ultimately contribute to the solution. Rabbit holes can be particularly challenging to navigate in complex or ambiguous problem-solving situations, as the problem solver may not have a clear sense of what is relevant or important.

    When it comes to solving complex problems, there are different approaches that can be taken, each with its strengths and weaknesses. The lateral approach and the root cause approach are two such approaches, and they can be contrasted as follows:

    • The lateral approach involves exploring multiple possible solutions to a problem, even if they seem unrelated or unconventional. This approach is often used when a problem is particularly complex or when conventional methods have not been successful. The goal of the lateral approach is to come up with creative and innovative solutions that may not have been considered otherwise.
    • The root cause approach, on the other hand, involves identifying the underlying cause of a problem and addressing it directly. This approach is often used when a problem has a clear and identifiable cause, and the goal is to prevent similar issues from occurring in the future. The root cause approach can involve a detailed analysis of the problem, including data collection, hypothesis testing, and problem validation.

    In terms of rabbit holes and deep dives into problems, the lateral approach may be more prone to exploring rabbit holes, as it involves exploring multiple possible solutions, some of which may not ultimately be useful. However, this approach can also lead to unexpected breakthroughs and insights, as well as the discovery of new approaches and solutions.

    The root cause approach, on the other hand, is designed to avoid rabbit holes and deep dives by focusing on the underlying cause of the problem. This approach can be more systematic and targeted, and is often used when time is of the essence or when resources are limited. However, it may also miss opportunities for creative solutions or unexpected insights.

    Ultimately, the choice between the lateral approach and the root cause approach will depend on the specific problem at hand, as well as the goals and resources of the problem solver. In some cases, a combination of both approaches may be appropriate, as each approach can complement the other in different ways.

    Structured and Unstructured Problem Solving

    Consider efficiency.

    Structured Method

    One structured method for complex problem solving that is efficient is the following:

    1. Define the problem: Clearly define the problem you are trying to solve. Identify the symptoms and the root cause, and set a clear objective for the problem-solving process.
    2. Gather information: Gather all the necessary information about the problem, its causes, and its context. This may involve data collection, stakeholder interviews, literature reviews, or other research methods.
    3. Analyze the information: Use a structured process to analyze the information you have gathered. This may involve techniques such as root cause analysis, SWOT analysis, or other problem-solving tools. Identify the key issues, risks, and opportunities associated with the problem.
    4. Develop and evaluate solutions: Brainstorm potential solutions to the problem, and evaluate them using a structured decision-making process. Consider the feasibility, impact, and risks associated with each option, and select the best solution.
    5. Implement the solution: Develop an action plan to implement the solution, and assign responsibilities and resources. Monitor progress and adjust the plan as necessary.
    6. Evaluate the results: Once the solution has been implemented, evaluate its effectiveness in solving the problem. Identify any unintended consequences or other issues that arose, and make any necessary adjustments.

    By following this structured approach, problem-solvers can efficiently and effectively navigate complex problem-solving situations, and develop solutions that are based on a rigorous analysis of the problem and its underlying causes.

    Lets take the scenario where the company’s network being hacked and sensitive data is being stolen. The structured approach to solving the problem would involve a methodical process that can be broken down into the following steps:

    1. Define the problem: The first step is to clearly define the problem by gathering information and identifying the scope and impact of the security breach. This includes understanding the nature of the data that was stolen, the extent of the damage, and the potential consequences for the company and its clients.
    2. Identify the root cause: Once the problem is defined, the next step is to identify the root cause of the security breach. This involves conducting a thorough investigation to identify the vulnerabilities in the network that were exploited by the attacker. This might include analyzing network logs, conducting forensic analysis of the compromised systems, and interviewing employees who had access to the affected systems.
    3. Develop a solution: Once the root cause has been identified, the next step is to develop a solution to address the problem. This might involve a range of measures, such as upgrading the company’s network security, implementing more robust access controls, or developing new policies and procedures to prevent future breaches.
    4. Implement the solution: The fourth step involves implementing the solution, which might involve a range of technical and organizational changes. This might include updating software and hardware, training employees on new policies and procedures, or engaging third-party security experts to help monitor and protect the network.
    5. Monitor and evaluate: The final step is to monitor and evaluate the effectiveness of the solution over time. This might involve conducting regular security audits, reviewing incident reports, and analyzing data on network traffic and user behavior to identify potential threats and vulnerabilities.

    By following a structured approach to problem-solving, the team can ensure that they have a clear understanding of the problem, identify the root cause, develop an effective solution, and monitor its effectiveness over time. While this approach may not be as flexible or creative as an unstructured approach, it can help ensure that all aspects of the problem are carefully considered and that solutions are implemented in a methodical and rigorous way.

    Unstructured Method

    An unstructured, genius or synergetic approach to complex problem solving may involve the following elements:

    1. Intuition: Rather than relying solely on data and analysis, this approach relies heavily on the intuition and creativity of the problem solver. This can involve thinking outside the box, coming up with unconventional solutions, or tapping into “gut instincts” to guide decision-making.
    2. Collaboration: Collaboration and teamwork are often key components of this approach. Bringing together diverse perspectives and expertise can lead to breakthrough insights and solutions that might not have been possible through individual effort.
    3. Exploration: This approach may involve a willingness to explore different avenues, even if they initially seem unrelated or tangential to the problem at hand. This can involve taking risks, experimenting with new approaches, or exploring unconventional ideas.
    4. Adaptability: Problem-solvers using this approach may be more willing to adapt and change course as new information arises, rather than sticking rigidly to a pre-determined plan or approach.
    5. Learning: Continuous learning and growth are important aspects of this approach. Problem-solvers may seek out new knowledge, insights, and feedback to improve their ability to solve complex problems over time.

    While this approach is less structured than the method outlined earlier, it can be highly effective in achieving the same outcome. By tapping into intuition, collaboration, exploration, adaptability, and learning, problem-solvers can generate creative and innovative solutions to complex problems that may have been difficult to address using a more traditional, structured approach.

    Let’s again consider the complex computer problem where a company’s network has been hacked, and sensitive data has been stolen. An unstructured, genius or synergetic approach to solving this problem might involve the following steps:

    1. Intuition: The problem solver might start by tapping into their intuition and creative thinking to come up with unconventional solutions. They might consider unconventional approaches such as using social engineering techniques to identify the attacker or tracing the stolen data using blockchain technology.
    2. Collaboration: The problem solver might bring together a diverse team of experts, including IT security specialists, data analysts, and even social scientists or hackers, to pool their expertise and generate innovative ideas.
    3. Exploration: The team might explore a variety of different approaches to solving the problem, even if they initially seem unrelated or tangential. For example, they might explore the possibility of using AI algorithms to identify patterns in the stolen data or analyze social media posts to track the attacker.
    4. Adaptability: The team might be willing to adapt and change course as new information emerges. For example, they might shift their focus from tracking the attacker to shoring up the company’s network security, or they might adjust their approach to incorporate new data or insights.
    5. Learning: The team might engage in continuous learning and growth to improve their ability to solve complex problems over time. They might seek out new knowledge, insights, and feedback to refine their approach and stay ahead of evolving threats.

    By applying an unstructured approach to this complex computer problem, the team might be able to generate creative and innovative solutions that traditional, structured methods may have overlooked. While this approach may be more time-consuming and unpredictable than a traditional method, it can be highly effective in addressing complex and rapidly-evolving challenges in the fast-paced world of technology.

    Structured vs Unstructured

    The structured approach is designed to provide a systematic and efficient method for problem-solving. It typically involves breaking down the problem into smaller components, analyzing each component, and then synthesizing a solution from the results of the analysis. This approach may be less innovative than the unstructured approach but it can be more reliable, especially when dealing with complex systems.

    The unstructured approach to solving the complex computer problem may involve a more creative and exploratory process, with a focus on brainstorming and testing various hypotheses. This approach can be highly effective in uncovering innovative solutions to difficult problems, but it may also be time-consuming and require a higher level of expertise.

    When comparing the two approaches in terms of outcome, the unstructured approach may yield more creative and potentially ground breaking solutions, while the structured approach may produce more reliable and tested solutions.

    However, the unstructured approach may be more costly in terms of time and effort, as it requires more exploration and experimentation.

    In summary, both the unstructured and structured approaches to complex problem-solving have their strengths and weaknesses.

    • The unstructured approach can be highly creative and effective in generating innovative solutions but may require more time and effort.
    • The structured approach, on the other hand, can provide a reliable and efficient method for problem-solving but may be less innovative.

    The choice of approach ultimately depends on the nature of the problem, the available resources, and the desired outcome. In a cost or time sensitive environment it is likely that a structured process will be preferred route to solving a problem, usually at the expense of innovation.