Tag: Survival

  • The IT Department Survival Guide for New Starters

    Welcome to IT.

    You have been recruited because the organisation believes you possess valuable technical skills, sound judgement and the ability to remain calm under pressure.

    Within three weeks you will discover that your actual role is to explain why a printer cannot be fixed by changing somebody’s password.

    This guide exists to help.

    1. Learn the First Law of IT

    Everything is your fault.

    The payroll system is slow.

    IT.

    The meeting room is cold.

    IT.

    A customer cannot remember their username.

    IT.

    The coffee machine says DESCALE.

    IT.

    Karen has deleted an Excel workbook containing the organisation’s entire procurement strategy.

    Definitely IT.

    You may occasionally attempt to explain that Information Technology does not control plumbing, building access, furniture, catering or the weather.

    This is a beginner’s mistake.

    The user does not care which department owns the problem.

    They have found somebody wearing a headset.

    That somebody is you.

    Accept this.

    It will save time.

    2. Never Say “That Should Work”

    The gods hear this.

    You may test a system for six months.

    You may perform penetration testing, regression testing, failover testing, disaster recovery testing and a full dress rehearsal involving nineteen engineers and a conference bridge.

    The moment you tell management:

    “That should work.”

    A certificate will expire.

    Prefer:

    “We have not identified any current impediment to successful operation.”

    This means the same thing but allows considerably more room for professional retreat.

    Other useful phrases include:

    “That’s interesting.”

    Meaning:

    That is absolutely fucked.

    “I haven’t seen that before.”

    Meaning:

    I have seen this six times and none ended well.

    “Let me check the logs.”

    Meaning:

    Please stop talking while I think.

    “There may be a dependency.”

    Meaning:

    Nobody documented this bastard thing.

    “We need to understand the business impact.”

    Meaning:

    Is anyone actually using it?

    3. The Service Desk Knows Everything

    Treat the Service Desk well.

    Senior architects may understand strategy.

    Network engineers may understand routing.

    Security may understand certificates.

    Database administrators may understand things spoken of only in whispers.

    But the Service Desk knows that Finance cannot print on Thursdays because Derek installed a label printer driver in 2019.

    This is real knowledge.

    The CMDB will tell you:

    FIN-PRINT-04 — HP LaserJet — ACTIVE

    The Service Desk will tell you:

    “That’s actually the tea-room printer. FIN-PRINT-04 fell down the stairs during the office move. The one Finance uses is called Susan.”

    Believe the Service Desk.

    Buy them biscuits.

    4. Do Not Insult Legacy Systems

    You will encounter systems older than some employees.

    Do not laugh.

    A Windows Server 2008 machine under someone’s desk may turn out to process £80 million a year in direct debits.

    An Access 2003 database called:

    MASTER_FINAL_USE_THIS_ONE_v7.mdb

    may contain the only authoritative record of something legally significant.

    A beige PC in Facilities may control every door in the building.

    You will ask:

    “Why hasn’t this been replaced?”

    Everyone will look at the floor.

    You will eventually learn that replacement was proposed in:

    and 2024.

    Each programme produced a strategy.

    The old system continued running.

    Do not mock it.

    It has survived more transformation programmes than you have.

    Show respect.

    5. Never Reboot Anything Without Witnesses

    Rebooting a laptop is harmless.

    Rebooting a server is theology.

    Before restarting infrastructure, obtain:

    a ticket,

    an approved change,

    a backup,

    a rollback plan,

    a witness,

    and preferably a small priest.

    The application owner will insist that the system can be restarted at any time.

    Do not believe them.

    The moment it goes down, seventeen unidentified business processes will emerge screaming from the darkness.

    One of them will be “month end.”

    It is always month end.

    Nobody knows when month end begins.

    It appears to last approximately thirty-one days.

    6. Production Is Different

    Development works.

    Test mostly works.

    Pre-production is theoretically identical to production.

    It is not.

    Production contains:

    three undocumented firewall rules,

    a certificate installed by somebody who left in 2018,

    a manual DNS entry,

    a service account called temp_admin,

    and one scheduled task created by Keith.

    Never delete Keith’s scheduled task.

    Nobody knows what it does.

    Keith is unreachable.

    But whenever the task is disabled, Belgium stops invoicing.

    7. Learn the Hierarchy of Passwords

    There are passwords.

    There are admin passwords.

    There are service accounts.

    There are break-glass accounts.

    There are credentials stored in approved privileged-access systems.

    And there is a text file called:

    passwords.txt

    on an old shared drive.

    Security will insist this does not exist.

    Operations will know exactly where it is.

    Your objective is not to become comfortable with this.

    Your objective is to survive long enough to remove it without bringing down payroll.

    8. DNS Is Probably Involved

    When an application behaves inexplicably, someone will eventually say:

    “Could be DNS.”

    This will be offered as either wisdom or sarcasm.

    Do not dismiss it.

    DNS has caused enough damage to earn its reputation.

    Other usual suspects include:

    certificates,

    time synchronisation,

    firewalls,

    proxies,

    permissions,

    storage,

    load balancers,

    and that one forgotten NAT rule in the disaster recovery site.

    Eventually somebody will discover the actual cause was a typo.

    This does not invalidate the investigation.

    It merely completes it.

    9. Certificates Expire Only on Weekends

    Certificate expiry dates are visible months in advance.

    Monitoring systems can alert on them.

    Renewal processes can be automated.

    Owners can be assigned.

    None of this matters.

    The certificate will expire at 02:13 on a Sunday.

    A senior manager will call.

    They will say:

    “The website is down.”

    You will ask:

    “Which website?”

    They will reply:

    “The website.”

    This is all the information you are getting.

    10. Change Management Is a Ritual, Not a Guarantee

    The change form exists to answer several important questions:

    What are you changing?

    Why?

    When?

    How?

    What happens if it goes wrong?

    Who approved this madness?

    You will spend forty minutes completing it.

    The Change Advisory Board will spend four minutes discussing it.

    Someone will ask:

    “Has the business approved this?”

    You will say:

    “Yes.”

    Someone else will ask:

    “What’s the rollback?”

    You will repeat the paragraph already on screen.

    The change will be approved.

    Then, two hours before implementation, an executive will request an “urgent small amendment.”

    The small amendment will fundamentally alter the architecture.

    You will be asked whether it can be included under the existing change.

    It cannot.

    It will be.

    11. Incidents Have Gravity

    A Priority 4 incident is ignored.

    A Priority 3 gets a ticket.

    A Priority 2 gets a Teams call.

    A Priority 1 bends spacetime.

    People who have never previously shown interest in the system will materialise.

    Directors will join the bridge.

    Suppliers will join.

    Cyber will join.

    Communications will join.

    Someone from Risk will ask whether the incident is “contained.”

    Nobody knows what that means yet.

    The technical team will be trying to fix the problem while twenty-three people ask them for updates.

    Eventually somebody sensible will create two calls:

    Technical Bridge.

    Management Bridge.

    This is one of civilisation’s greatest inventions.

    On the Management Bridge, executives can ask:

    “When will it be fixed?”

    On the Technical Bridge, engineers can answer:

    “When you stop fucking asking.”

    12. Never Give a Recovery Time Unless You Mean It

    Management will request an ETA.

    They do not actually want an estimate.

    They want certainty disguised as an estimate.

    If you say:

    “Thirty minutes.”

    At twenty-nine minutes someone will ask:

    “Are we still on track?”

    At thirty-one minutes your estimate will be treated as a failed contractual commitment.

    Prefer:

    “We are working through the recovery sequence. I’ll update when we have a validated restoration point.”

    This is IT language for:

    We have no bloody idea, but Gary has found something promising.

    13. Gary Is Important

    Every department has a Gary.

    Gary may not actually be called Gary.

    He may be called Steve, Anita, Mo, Raj, Susan or Dave.

    Gary has worked there for twenty-seven years.

    Gary knows:

    why server names begin with Z,

    which fibre pair is actually live,

    why Warehouse Three must never be rebooted remotely,

    which database column is lying,

    and why the chief executive’s laptop cannot be replaced before the board meeting.

    Gary’s knowledge is undocumented because nobody has ever given Gary enough time to document it.

    Management describes this as a key-person risk.

    Then gives Gary more work.

    Identify Gary.

    Protect Gary.

    Learn from Gary.

    If Gary says:

    “Don’t touch that.”

    Do not touch that.

    14. Architecture Diagrams Are Historical Fiction

    The diagram you receive on your first day will contain:

    two firewalls,

    three servers,

    a database,

    and a cloud.

    The actual environment will contain:

    six firewalls,

    forty-seven servers,

    three clouds,

    a forgotten MPLS circuit,

    two appliances nobody owns,

    and something labelled “temporary gateway” installed eleven years ago.

    Treat architecture diagrams as archaeological evidence.

    Useful.

    Interesting.

    Not necessarily current.

    If somebody says:

    “The diagram is accurate.”

    Ask:

    “As of when?”

    This question will make you unpopular but powerful.

    15. The CMDB Is Aspirational

    Configuration Management Databases contain valuable information about assets, dependencies and ownership.

    In theory.

    In practice, you may find:

    three entries for the same server,

    an application owner who retired,

    a laptop listed as a critical production dependency,

    and a database marked “decommissioned” which is currently processing customer transactions.

    Never assume the CMDB is wrong.

    Never assume it is right.

    Think of it as a witness with a complicated relationship with truth.

    16. The Cloud Is Someone Else’s Computer, Plus Billing

    At some point someone will say:

    “We should move this to the cloud.”

    This may be correct.

    It may also mean:

    We would like the same mess, but billed monthly.

    Cloud platforms provide extraordinary capabilities.

    They also allow an enthusiastic developer to create £18,000 of infrastructure before lunch.

    Learn tagging.

    Learn budgets.

    Learn identity.

    Learn networking.

    Learn how egress charging works before somebody creates an exciting multi-cloud architecture.

    Most importantly, never accept the phrase:

    “It’ll be cheaper.”

    Ask:

    “Compared with what?”

    Watch the room become philosophical.

    17. Vendors Are Your Friends Until Renewal

    Suppliers will use phrases such as:

    strategic partnership,

    customer success,

    digital journey,

    co-innovation,

    and trusted advisor.

    These expressions mean:

    We would like another purchase order.

    A vendor account manager will remember your birthday if the contract is large enough.

    Three months before renewal, they will become intensely interested in your roadmap.

    One month after renewal, support will ask you to reproduce the problem on the latest version.

    The latest version will not support your operating system.

    This is enterprise software.

    18. Licensing Is Dark Magic

    Nobody fully understands enterprise licensing.

    Not Sales.

    Not Procurement.

    Not Legal.

    Not the vendor.

    Certainly not the auditor.

    You will encounter concepts such as:

    named user,

    concurrent user,

    processor,

    core,

    socket,

    virtual core,

    installed instance,

    running instance,

    minimum quantities,

    indirect access,

    multiplexing,

    and “authorised environment.”

    At some point you will ask:

    “How many licences do we actually need?”

    The room will go quiet.

    A consultant will be hired.

    Three months later you will receive a spreadsheet containing thirty-seven tabs and the phrase:

    Subject to contractual interpretation.

    Keep it.

    It cost £90,000.

    19. Security Will Say No

    This is partly their job.

    Do not become angry.

    Instead ask:

    “What control objective are we trying to satisfy?”

    This transforms an argument into architecture.

    Sometimes.

    Security may still say no.

    If they do, ask for the requirement in writing.

    Not because you intend to fight them.

    Because six months later somebody will ask why the project is late.

    Documentation is not bureaucracy.

    Documentation is armour.

    20. Users Lie, But Usually Innocently

    “The computer just deleted my file.”

    No, it didn’t.

    “I haven’t changed anything.”

    They have.

    “It worked yesterday.”

    Possibly.

    “I’ve restarted it.”

    They logged off.

    “The internet is down.”

    One website is unavailable.

    “My password definitely works.”

    It does not.

    Do not accuse users of lying.

    Users report their model of reality.

    Your job is to identify the gap between their model and the logs.

    Be polite.

    You may need these people later.

    Especially Payroll.

    Never antagonise Payroll.

    21. Screenshots Are Evidence

    Ask for a screenshot.

    Not:

    “What did the error say?”

    Users will paraphrase:

    “It said access or something.”

    The actual message will say:

    SQLSTATE 28000: Login failed for user svc_finance_prod.

    This distinction matters.

    Screenshots also reveal:

    the URL,

    time,

    username,

    browser,

    environment,

    and seventeen browser tabs containing information you did not ask to know.

    Be professional.

    22. Never Trust “Quick Question”

    A colleague approaching your desk with:

    “Quick question…”

    is carrying at least forty-five minutes of work.

    Common variants include:

    “Can I pick your brain?”

    “Just while you’re here…”

    “You know about networks, right?”

    “This’ll only take a second.”

    The correct response is not hostility.

    The correct response is:

    “Sure. What’s the ticket number?”

    Watch nature take its course.

    23. Projects End. Applications Do Not.

    Projects have budgets.

    Governance.

    Steering committees.

    Milestones.

    Celebrations.

    Applications have Tuesday mornings.

    The project team will deliver a shiny new system.

    Photographs will be taken.

    Cake may appear.

    Then the project closes.

    Six months later Operations asks:

    “Who supports this?”

    Silence.

    The project manager has moved to another transformation programme.

    The architect is consulting in Dubai.

    The supplier says support was not included.

    The business says IT owns it.

    IT says the business owns it.

    The application continues running.

    This is how legacy begins.

    24. Backups Are Not the Same as Recovery

    Someone will proudly tell you:

    “We back everything up.”

    Ask:

    “Have we restored it?”

    A backup that has never been restored is a theory.

    A disaster recovery plan that has never been tested is literature.

    A failover process dependent on one person remembering a password is folklore.

    Test recovery.

    Document recovery.

    Then test the document.

    Otherwise, during an incident, somebody will discover that the backup server depends on the system you are trying to restore.

    This is called enterprise architecture.

    25. Monitoring Produces Two States

    No alerts.

    Too many alerts.

    In the first state, management asks whether monitoring works.

    In the second, everyone ignores it.

    Your mission is to reach the mythical third state:

    Useful alerts.

    This involves deleting hundreds of alarms that effectively mean:

    “CPU exists.”

    If every event is critical, nothing is critical.

    This principle also applies to email marked HIGH IMPORTANCE.

    26. Meetings Reproduce

    IT meetings reproduce by mitosis.

    A project meeting identifies a technical issue.

    A technical meeting is created.

    The technical meeting identifies a security concern.

    A security workshop is created.

    The security workshop identifies a dependency.

    A dependency call is created.

    Eventually eight people attend meetings all day discussing work none of them now has time to perform.

    Protect blocks of actual working time.

    Do not apologise for this.

    Someone has to configure the thing.

    27. Teams Status Is Political

    Green means available.

    Yellow means possibly alive.

    Red means either extremely busy or eating lunch.

    Do Not Disturb means senior architect attempting to produce something before another meeting begins.

    Offline means nothing.

    Some people have been “Offline” since 2022 while responding instantly to every message.

    Do not infer reality from Teams presence.

    It is less reliable than the CMDB.

    28. Document Everything Important

    Especially decisions.

    After a meeting, write:

    “To confirm our agreed position…”

    This sentence has prevented more professional disasters than most cybersecurity products.

    Record:

    what was decided,

    who decided it,

    what assumptions were made,

    what risks were accepted,

    and who owns the next action.

    Six months later, when someone says:

    “IT recommended this architecture.”

    You can produce the email showing that IT recommended the opposite.

    Do not wave it triumphantly.

    Simply attach it.

    The effect is stronger.

    29. Never Become the Only Person Who Knows

    Being indispensable feels good.

    Until you want a holiday.

    Document your work.

    Cross-train colleagues.

    Share passwords through proper systems.

    Automate repetitive tasks.

    The goal is not to become the hero who receives calls at 03:00.

    The goal is to build systems that do not require heroes.

    Heroic IT is usually failed engineering wearing a cape.

    30. Finally: Find the People Who Actually Make Things Work

    Every IT department has formal structures.

    Architecture.

    Infrastructure.

    Applications.

    Service Management.

    Security.

    PMO.

    Data.

    Cloud.

    Workplace.

    Networks.

    Then there is the real structure.

    The network engineer who answers the phone.

    The DBA who knows the ancient application.

    The Service Desk analyst who notices patterns.

    The project manager who writes things down.

    The security architect who explains rather than obstructs.

    The desktop engineer who knows the executives.

    The developer who admits when something is broken.

    The procurement person who understands the licence.

    The administrator who knows where the contract lives.

    Find these people.

    Be useful to them.

    Do not waste their time.

    Share credit.

    Bring biscuits occasionally.

    And remember the final rule.

    One day, perhaps years from now, a nervous new starter will approach your desk.

    They will say:

    “Sorry, quick question. Everyone says you know how this works.”

    You will look at the undocumented system.

    You will look at the obsolete server.

    You will remember Gary.

    Then you will hear yourself say:

    “Right. Whatever you do, don’t reboot it.”

    And at that moment, your induction will finally be complete.