[{"body":"","link":"https://systemsgolive.com/consulting/","section":"","tags":null,"title":"Cloud \u0026 DevOps Consulting"},{"body":"","link":"https://systemsgolive.com/tags/ansible/","section":"tags","tags":null,"title":"Ansible"},{"body":"","link":"https://systemsgolive.com/tags/automation/","section":"tags","tags":null,"title":"Automation"},{"body":"","link":"https://systemsgolive.com/tags/azure/","section":"tags","tags":null,"title":"Azure"},{"body":"","link":"https://systemsgolive.com/categories/azure/","section":"categories","tags":null,"title":"Azure"},{"body":"","link":"https://systemsgolive.com/tags/bicep/","section":"tags","tags":null,"title":"Bicep"},{"body":"","link":"https://systemsgolive.com/tags/case-study/","section":"tags","tags":null,"title":"Case Study"},{"body":"Problem As part of a live infrastructure migration for a healthtech platform (moving production services from an existing UK cloud hosting provider to Microsoft Azure), every application server was being built by hand: OS hardening, user creation, SSH configuration, Tomcat and OpenJDK installation, file-sync setup, TLS certificates, and LDAP, all configured manually, server by server. Each new application server took roughly 4 hours of hands-on-keyboard work.\nThe platform is a Digital Social Care Record (DSCR) system operating in a highly regulated environment: care providers using it must be registered with the Care Quality Commission (CQC) and meet the requirements of the NHS Data Security and Protection (DSP) Toolkit. In that environment, manual builds carry real risk: undocumented configuration drift between nodes, inconsistent hardening, and no repeatable record of what was actually done to a server, all things an auditor (and an incident) will eventually surface.\nApproach Working within a larger Azure migration project, I split infrastructure delivery into two layers:\nProvisioning (Bicep + Azure Deployment Stacks): VMs are provisioned directly into pre-existing, pre-approved subnets. Network and VNet boundaries are handled separately and were not part of this automation. Configuration (Ansible): Everything on top of the VM, OS hardening, user creation, SSH setup, Tomcat, OpenJDK, file synchronization, Let's Encrypt TLS certificates issued and auto-renewed via Certbot, and LDAP, is deployed through templated, validated playbooks. Configuration changes are checked (dry-run/diff) before being applied, and validation gates prevent broken configuration from reaching production silently. The new Azure environment is being built in parallel with the existing production environment, not as a big-bang cutover. A site-to-site VPN connects the two environments; migration traffic between them uses private IP addressing over that encrypted VPN tunnel, and services are not directly exposed to the public internet. That private connectivity is used to copy application directories and content across ahead of time, and at cutover, rsync is used to pull across only the incremental changes rather than a full transfer, keeping the final cutover window short.\nTo control cost while the two environments run side by side, the new Azure infrastructure is deliberately built at a lower compute tier than production load requires. The plan is to scale out compute at the point of cutover, when traffic is redirected to Azure via a DNS (FQDN) change.\nThe result is a repeatable pipeline: provisioning and full service configuration for a node now happen the same way, every time, instead of depending on whoever is building it that day and what they remember to do.\nOutcome Build time: ~4 hours per application server manually → ~30 minutes of automated pipeline time (Bicep provisioning + Ansible configuration) to fully configure the entire cluster (OS, security, services, TLS, LDAP). Risk reduction: Removed the main source of human error and configuration drift during an actively audited migration. Configuration is now defined in code and validated before deployment, not typed by hand under time pressure. Recovery speed: A failed or replaced node can be rebuilt to the same known-good state in minutes rather than being manually reconstructed from memory or scattered notes. Time reallocation: Removed repetitive manual build work from the migration timeline, freeing time for higher-value work on the migration itself (directory services, file-sync architecture, and cutover planning). This work sits inside a larger, still-ongoing cloud migration for the platform; the outcomes above reflect the provisioning and configuration automation specifically, not the full migration end-to-end.\nScope note: this covers the web/application infrastructure layer: VM provisioning (Bicep), OS/service configuration (Ansible), and the private connectivity (site-to-site VPN, firewall rules, rsync-based content sync) used to build it out. The database layer is a separate workstream and is not covered in this document. Final production cutover (DNS flip and compute scale-out) had not yet occurred at time of writing. Example of infrastructure automation work delivered as part of an employed role, not an independent client engagement.\n","link":"https://systemsgolive.com/post/infrastructure-automation-healthtech-azure-migration-case-study/","section":"post","tags":["Case Study","Azure","Bicep","Ansible","Infrastructure as Code","Cloud Migration","Automation"],"title":"Case Study: Infrastructure Automation Cuts Server Build Time from 4 Hours to 30 Minutes on a Regulated Healthtech Platform"},{"body":"","link":"https://systemsgolive.com/categories/","section":"categories","tags":null,"title":"Categories"},{"body":"","link":"https://systemsgolive.com/tags/cloud-migration/","section":"tags","tags":null,"title":"Cloud Migration"},{"body":"","link":"https://systemsgolive.com/categories/devops/","section":"categories","tags":null,"title":"DevOps"},{"body":"","link":"https://systemsgolive.com/tags/infrastructure-as-code/","section":"tags","tags":null,"title":"Infrastructure as Code"},{"body":"","link":"https://systemsgolive.com/tags/","section":"tags","tags":null,"title":"Tags"},{"body":"","link":"https://systemsgolive.com/tags/build/","section":"tags","tags":null,"title":"Build"},{"body":"","link":"https://systemsgolive.com/tags/buildkit/","section":"tags","tags":null,"title":"Buildkit"},{"body":"","link":"https://systemsgolive.com/tags/compiler/","section":"tags","tags":null,"title":"Compiler"},{"body":"","link":"https://systemsgolive.com/categories/docker/","section":"categories","tags":null,"title":"Docker"},{"body":"","link":"https://systemsgolive.com/tags/dockerfile/","section":"tags","tags":null,"title":"Dockerfile"},{"body":"","link":"https://systemsgolive.com/tags/multi-stage/","section":"tags","tags":null,"title":"Multi-Stage"},{"body":"The problem I had a TypeScript app in a container. The image was 289MB. Most of that was stuff the app never touches at runtime – the TypeScript compiler, webpack, jest, eslint. Build tools. They have no business being in a production image.\nMulti-stage builds fix this. Same app, same code, same behaviour. The image dropped to 167MB. But size is just one of four things multi-stage builds improve. Here is what they actually do, and why each one matters.\nWhy a TypeScript app even has this problem Node runs JavaScript. It does not run TypeScript. So a TypeScript app has a build step: tsc compiles .ts into .js.\nThat compiler – and every dev tool around it – is needed to build the app. None of it is needed to run it. That single distinction is the whole game.\nMy package.json had two kinds of dependencies:\ndependencies: express. Needed at runtime. devDependencies: typescript, webpack, jest, eslint, prettier, and their @types. Needed only to build. On disk that split is brutal:\nAll deps installed: 128MB Production deps only: 4.9MB So roughly 123MB of my image was build tooling I would never call in production.\nSingle-stage: ships everything 1FROM node:22-alpine 2WORKDIR /app 3COPY package*.json ./ 4RUN npm ci # installs ALL deps, dev tooling included 5COPY . . # copies the .ts source 6RUN npm run build # tsc compiles src -\u0026gt; dist 7EXPOSE 3000 8CMD [\u0026#34;node\u0026#34;, \u0026#34;dist/server.js\u0026#34;] This works. But look at what lands in the image. npm ci installs all 128MB of dependencies. COPY . . brings in the .ts source. npm run build compiles it, but the compiler stays. Source, compiler, bundler, test runner – all shipped. None of them run in production.\nFinal image: 289MB.\nMulti-stage: leave the build behind 1# ---- build stage (throwaway) ---- 2FROM node:22-alpine AS build 3WORKDIR /app 4COPY package*.json ./ 5RUN npm ci # ALL deps, incl. the compiler 6COPY . . 7RUN npm run build # tsc -\u0026gt; dist 8 9# ---- runtime stage (what ships) ---- 10FROM node:22-alpine AS runtime 11WORKDIR /app 12ENV NODE_ENV=production 13COPY package*.json ./ 14RUN npm ci --omit=dev # production deps only – no compiler 15COPY --from=build /app/dist ./dist # copy only the compiled output 16USER node 17EXPOSE 3000 18CMD [\u0026#34;node\u0026#34;, \u0026#34;dist/server.js\u0026#34;] Two FROM lines means two separate images. When the build finishes, the build stage is discarded. Everything in it – compiler, dev deps, .ts source – is gone. The final image is the runtime stage only.\nNow for the four reasons this matters.\n1. Reduced image size The key instruction is in the runtime stage:\n1RUN npm ci --omit=dev --omit=dev tells npm to skip everything in devDependencies. In this project that means typescript, webpack, jest, eslint, prettier, and all their @types packages. None of them are installed in the final image.\nThe other instruction that keeps the image lean:\n1COPY --from=build /app/dist ./dist This copies only the compiled output from the build stage. Not the .ts source files. Not node_modules from the build stage. Just the .js that the runtime actually needs.\nImage Size single-stage 289MB multi-stage 167MB 122MB gone – about 42% – purely from leaving build tooling behind. The base image (node:22-alpine, ~163MB) is identical in both, so nearly all of the savings is exactly the dev tooling and source the runtime stage never installs.\n2. Improved security The runtime stage never installs dev dependencies. That is not just a size win.\nWhen I ran npm install, npm flagged 19 moderate vulnerabilities – almost all in dev tooling (webpack and jest dependency chains). Because the runtime stage uses RUN npm ci --omit=dev, those packages are never installed in what ships. Those CVEs are not present in the final image. You cannot be exploited through a compiler that is not there.\nThe same applies to source code. In a single-stage build, COPY . . puts your .ts source into the image. In the multi-stage build, source never touches the runtime stage:\n1# build stage: source lands here, gets compiled, then this stage is discarded 2COPY . . 3RUN npm run build 4 5# runtime stage: source is never copied here 6# only the compiled output crosses the boundary 7COPY --from=build /app/dist ./dist No source code in the final image means no accidental exposure of business logic, credentials left in source comments, or internal module paths. Not present beats present but unused.\n3. Enhanced maintainability Named stages make each stage's purpose explicit in the Dockerfile itself:\n1FROM node:22-alpine AS build # intent: compile the app, will be discarded 2... 3FROM node:22-alpine AS runtime # intent: this is what ships to production Without named stages, a single-stage Dockerfile relies entirely on comments to explain why certain commands appear in a certain order. With named stages, the structure makes the intent clear.\nNamed stages also let you target a specific stage at build time:\n1docker build --target build -t myapp:build . This builds only up to the build stage – useful when debugging a compilation error or inspecting what the compiler produced, without running the full multi-stage build. You cannot do this with a single-stage Dockerfile.\nAs Dockerfiles grow – test stages, separate stages for different build targets – named stages keep the file readable by grouping related instructions under a clear label rather than scattering them through a single linear script.\n4. Faster builds Two things make multi-stage builds faster: layer caching and stage parallelisation.\nLayer caching Docker caches each instruction as a layer and reuses it on subsequent builds if nothing above it changed. The order of instructions matters enormously.\nIn the build stage:\n1COPY package*.json ./ # cached until package.json changes 2RUN npm ci # cached until package.json changes – 128MB install skipped on most builds 3 4COPY . . # invalidated on every source change 5RUN npm run build # reruns only when source changes package.json changes rarely. COPY package*.json ./ and RUN npm ci are served from cache on almost every build. Only COPY . . and RUN npm run build rerun when you edit source code. If you reversed the order – copying all source before running npm ci – a single file change would force a full npm ci on every build.\nThe same pattern applies in the runtime stage:\n1COPY package*.json ./ 2RUN npm ci --omit=dev # cached as long as package.json is unchanged 3COPY --from=build /app/dist ./dist Stage parallelisation BuildKit (Docker's build engine) can run independent stages in parallel. If you add a test stage that branches off the same deps as build, BuildKit runs them simultaneously:\n1FROM node:22-alpine AS deps 2WORKDIR /app 3COPY package*.json ./ 4RUN npm ci 5 6FROM deps AS test # runs in parallel with \u0026#39;build\u0026#39; 7COPY . . 8RUN npm test 9 10FROM deps AS build # runs in parallel with \u0026#39;test\u0026#39; 11COPY . . 12RUN npm run build 13 14FROM node:22-alpine AS runtime 15WORKDIR /app 16ENV NODE_ENV=production 17COPY package*.json ./ 18RUN npm ci --omit=dev 19COPY --from=build /app/dist ./dist 20USER node 21EXPOSE 3000 22CMD [\u0026#34;node\u0026#34;, \u0026#34;dist/server.js\u0026#34;] test and build both depend on deps, but not on each other. BuildKit detects that and runs them at the same time. On a CI machine with multiple cores this meaningfully cuts total build time compared to running each stage sequentially.\nFor the two-stage example in this post the gain is modest (16.2s → 14.8s). With a heavier toolchain or a parallel test stage, the difference is far larger.\nWhen to use it Use multi-stage when your app has a build step whose tools you do not want in production:\nTypeScript (tsc) React / Vite / webpack bundles Go (compile to a binary) Java (build the jar) The rule: if you need something to build but not to run, it belongs in a build stage you throw away.\nWhen not to bother: a plain JavaScript app with no build step. Node runs the .js directly. There is nothing to compile, nothing to leave behind. Multi-stage just adds complexity for no gain. Single-stage with an alpine base is the right call there.\nTakeaway Multi-stage builds give you four things, and size is just the entry point:\nSmaller images – RUN npm ci --omit=dev and COPY --from=build keep only what the app needs to run. Fewer CVEs – dev tooling and source code never land in the runtime stage, so their vulnerabilities are not in what you ship. Readable Dockerfiles – AS build and AS runtime make intent explicit and unlock --target for debugging individual stages. Faster builds – instruction ordering maximises cache hits; BuildKit parallelises independent stages. For this app: 289MB down to 167MB, 19 CVEs gone, build time cut. On a heavier toolchain all four numbers are bigger. The pattern is always the same – a throwaway build stage, a clean runtime stage, and COPY --from=build to carry only the output across the boundary.\n","link":"https://systemsgolive.com/post/multi-stage-docker-builds-smaller-safer-faster/","section":"post","tags":["dockerfile","multi-stage","build","runtime","compiler","typescript","nodejs","buildkit"],"title":"Multi-Stage Docker Builds: Smaller, Safer, and Faster"},{"body":"","link":"https://systemsgolive.com/tags/nodejs/","section":"tags","tags":null,"title":"Nodejs"},{"body":"","link":"https://systemsgolive.com/categories/optimisation/","section":"categories","tags":null,"title":"Optimisation"},{"body":"","link":"https://systemsgolive.com/tags/runtime/","section":"tags","tags":null,"title":"Runtime"},{"body":"","link":"https://systemsgolive.com/categories/security/","section":"categories","tags":null,"title":"Security"},{"body":"","link":"https://systemsgolive.com/tags/typescript/","section":"tags","tags":null,"title":"Typescript"},{"body":"","link":"https://systemsgolive.com/categories/architecture/","section":"categories","tags":null,"title":"Architecture"},{"body":"","link":"https://systemsgolive.com/tags/aws/","section":"tags","tags":null,"title":"AWS"},{"body":"","link":"https://systemsgolive.com/categories/aws/","section":"categories","tags":null,"title":"AWS"},{"body":"","link":"https://systemsgolive.com/tags/ec2/","section":"tags","tags":null,"title":"EC2"},{"body":"","link":"https://systemsgolive.com/tags/efs/","section":"tags","tags":null,"title":"EFS"},{"body":"","link":"https://systemsgolive.com/tags/high-availability/","section":"tags","tags":null,"title":"High Availability"},{"body":"","link":"https://systemsgolive.com/tags/nfs/","section":"tags","tags":null,"title":"NFS"},{"body":"Most engineers reach for EBS without thinking twice. It works, it's fast, and it's familiar. But EBS is attached to one instance in one AZ. The moment you need more than one instance to read and write the same data, EBS falls short.\nThat's where EFS comes in, and understanding when to use it is the difference between a working architecture and one that creates problems later.\nThe Problem EBS is block storage. One volume, one instance, one AZ. If you're running a single EC2 instance, it's fine. But if you're running a distributed application across multiple instances or multiple AZs, and those instances need access to a shared filesystem, EBS won't cut it.\nCommon scenarios where this matters:\nA web application with shared uploads or static assets A distributed application where multiple nodes need to read config or write logs to a common location Any workload requiring shared persistent storage across an Auto Scaling Group You could work around it with S3, but S3 is object storage and it doesn't behave like a filesystem. For workloads that expect POSIX filesystem semantics, EFS is the right tool.\nWhat EFS Actually Is EFS is a managed NFS filesystem. You mount it on EC2 instances just like any other filesystem. Multiple instances can mount it simultaneously, read and write concurrently, and see each other's changes in real time.\nWhat makes it resilient is the architecture. EFS is regional. AWS manages mount targets in each AZ automatically. Your instances mount via their local AZ mount target, which means traffic stays within the AZ and doesn't cross AZ boundaries unnecessarily.\nWhat We Built Three EC2 instances (web-01, web-02, web-03) across three availability zones (eu-west-2a, eu-west-2b, eu-west-2c), all mounting a single EFS filesystem (lab-efs).\nThe goal was simple: write from one instance, read from another. Prove shared concurrent access across AZs.\nSecurity Group Design This is where most people get it wrong on first setup.\nEFS mount targets sit behind a security group. If your EC2 instances can't reach port 2049 (NFS) on the mount targets, the mount times out with no useful error message. I spent time on this.\nThe cleanest pattern for a lab is a self-referencing security group rule:\nCreate one SG (ec2-efs-sg) Attach it to your EC2 instances and to the EFS mount targets Add an inbound rule: TCP 2049, source = the SG itself Any resource in the group can reach port 2049 on any other resource in the group. Simple, no CIDR blocks, no hardcoded IPs.\nIn production, I'd go further: a dedicated sg-efs on the mount targets with a single inbound rule sourced from the EC2 SG only. Principle of least privilege. The EFS SG has no reason to allow SSH or HTTP traffic.\nMounting with TLS Use the EFS mount helper with TLS enabled. The commands below are for Amazon Linux 2023, which was used for this lab. If you are on a different distribution, the package manager and package name may differ.\n1sudo yum install -y amazon-efs-utils 2sudo mkdir /mnt/efs 3sudo mount -t efs -o tls fs-xxxxxxxx:/ /mnt/efs The -o tls flag encrypts data in transit. It's one flag. There's no good reason to skip it.\nProving It Works Once mounted on all three instances, the test is straightforward:\nOn web-01:\n1echo \u0026#34;written from web-01 in eu-west-2a\u0026#34; | sudo tee /mnt/efs/test.txt On web-02:\n1echo \u0026#34;written from web-02 in eu-west-2b\u0026#34; | sudo tee -a /mnt/efs/test.txt On web-03:\n1cat /mnt/efs/test.txt Output:\n1written from web-01 in eu-west-2a 2written from web-02 in eu-west-2b One file, three instances, three AZs, real-time visibility. That's shared persistent storage working as expected.\nOne thing worth noting: tee without -a overwrites the file. Always use -a when appending. It's an easy mistake that makes the output look wrong when the filesystem is actually working fine.\nOne Caveat: Mounts Don't Survive Reboots The manual mount works, but it doesn't persist. Reboot web-01 and the EFS mount is gone. Running cat /mnt/efs/test.txt returns No such file or directory because /mnt/efs is now an empty local directory.\nTo make the mount persistent, add an entry to /etc/fstab on each instance:\n1echo \u0026#34;fs-xxxxxxxx:/ /mnt/efs efs _netdev,tls 0 0\u0026#34; | sudo tee -a /etc/fstab The _netdev flag is important. It tells the OS to wait for the network to be available before attempting the mount. Without it, the instance can fail to boot cleanly if EFS isn't reachable during startup.\nDry-run the fstab entry to verify it is correct, without actually mounting:\n1sudo mount -fav Then reboot and verify the mount is back:\n1sudo reboot 2# after reconnecting 3df -h | grep efs 4cat /mnt/efs/test.txt After rebooting web-01, the EFS filesystem mounts automatically and test.txt is immediately readable. The same data written earlier from web-01 and web-02 is still there. The /etc/fstab entry ensures the mount is restored on every boot without any manual intervention, and because the underlying data lives on EFS rather than the instance, nothing is lost when the instance restarts.\nWhen to Use EFS vs EBS vs S3 This comes up a lot so it's worth being direct about it:\nEBS: single instance, high IOPS, databases, OS volumes. Fast, simple, not shareable. EFS: shared filesystem across multiple instances or AZs. POSIX semantics, concurrent access, higher latency than EBS. S3: object storage, not a filesystem. Great for backups, static assets, data lakes. Doesn't behave like a mounted drive. If your application writes to a path like /var/uploads and you need multiple instances to see those files, EFS is the answer. Don't force S3 into a filesystem role. It creates complexity at the application layer that EFS handles natively.\nWhat's Next EFS Access Points for per-application directory isolation, and performance mode selection (General Purpose vs Max I/O) for higher-throughput workloads.\nEFS is one of those services that engineers underuse because they default to EBS. Once you've seen it work across AZs in real time, the use cases become obvious.\n","link":"https://systemsgolive.com/post/amazon-efs-shared-storage-across-availability-zones/","section":"post","tags":["AWS","EFS","EC2","Storage","High Availability","NFS","VPC"],"title":"Shared Storage Across Availability Zones with Amazon EFS"},{"body":"","link":"https://systemsgolive.com/tags/storage/","section":"tags","tags":null,"title":"Storage"},{"body":"","link":"https://systemsgolive.com/categories/storage/","section":"categories","tags":null,"title":"Storage"},{"body":"","link":"https://systemsgolive.com/tags/vpc/","section":"tags","tags":null,"title":"VPC"},{"body":"","link":"https://systemsgolive.com/tags/backup/","section":"tags","tags":null,"title":"Backup"},{"body":"Most teams treat backup as a checkbox. A retention policy gets set, scheduled snapshots run quietly in the background, and nobody thinks about it again until something breaks. The problem is that an untested backup is not a backup strategy - it's a false sense of security.\nThis post covers two things I built and tested in AWS: a centralised backup strategy across EC2 and S3 using AWS Backup, and a formal resilience assessment using AWS Resilience Hub. Neither of these is particularly complex to set up, but the decisions behind them matter more than the configuration itself.\nWhat We Built A centralised AWS Backup vault with a customer-managed KMS key A tiered backup plan covering EC2 (EBS) and S3, with tag-based resource selection A tested restore procedure with a measured EC2 RTO A Resilience Hub application with a defined RTO/RPO policy, an initial assessment, and an actioned recommendation Project 1 - AWS Backup Strategy The Problem Individual AWS services have their own snapshot mechanisms - EBS snapshots, RDS automated backups, S3 versioning. They all work, but they're isolated. There's no unified view of what's protected, no consistent retention policy, and no straightforward way to prove to an auditor that your recovery posture is coherent.\nAWS Backup solves this by acting as an orchestration layer across services. One vault, one plan, one place to look.\nSetup at a Glance KMS key - AWS KMS \u0026gt; Customer-managed keys \u0026gt; Create key. Named backup-dev-key. Symmetric key, Encrypt and Decrypt. Key policy scoped to the AWS Backup service principal and the admin IAM user. Backup vault - AWS Backup \u0026gt; Backup vaults \u0026gt; Create backup vault. Named backup-vault-dev, assigned the customer-managed KMS key above. Backup plan - AWS Backup \u0026gt; Backup plans \u0026gt; Create plan \u0026gt; Build a new plan. Named backup-dev-plan. Two rules added, both targeting backup-vault-dev: Field daily-7-day-retention weekly-30-day-retention Backup frequency Daily Weekly Start within 8 hours 8 hours Complete within 7 days 7 days Total retention 7 days 30 days Resource assignment - Named backup-lab-selection. IAM role: Default role (selected by default). Resource selection: Include all resource types. Under tags, add key Backup / value true, then click Assign resources. IAM - Under the resource assignment, select Default role. If AWSBackupDefaultServiceRole does not yet exist in the account, AWS will create it automatically with the correct permissions. Two managed policies are attached by default: AWSBackupServiceRolePolicyForBackup (creating recovery points) and AWSBackupServiceRolePolicyForRestores (restoring them). For S3, AWSBackupServiceRolePolicyForS3Backup and AWSBackupServiceRolePolicyForS3Restore also need to be attached - this is a common oversight covered in the S3 section below. Vault Design The first decision was the encryption key. AWS Backup defaults to an AWS-managed key, which is fine for many use cases. For this setup, I chose a customer-managed KMS key instead.\nThe reason is control. With a customer-managed key you can define key policies, restrict who can decrypt recovery points, rotate the key on your own schedule, and get a full audit trail via CloudTrail. In a regulated environment - financial services, healthcare - that level of control is often a compliance requirement, not a preference. It costs $1 per month. The trade-off is straightforward.\nBackup Plan: Two Rules, One Decision The backup plan uses two rules rather than one:\nDaily backups with 7-day retention - for operational recovery. If a deployment breaks something or a file gets deleted, you want yesterday's state. Short window, low storage cost, fast to restore. Weekly backups with 30-day retention - for compliance and longer-term recovery. If a problem goes undetected for two weeks - corrupted data, a subtle misconfiguration - you still have a restore point. This also maps to common audit requirements. A single daily backup with a long retention period would technically cover both scenarios, but the cost compounds quickly. Tiering the retention keeps storage costs proportional to the actual recovery value of each point in time.\nTag-Based Resource Selection Rather than hardcoding resource ARNs into the backup plan, I used tag-based selection. Any resource tagged Backup: true is automatically included in the plan.\nThis matters at scale. If you're managing dozens of resources across multiple services, ARN-based selection becomes a maintenance burden - every new resource requires a plan update. Tag-based selection is self-service: the resource is created with the right tag and it's automatically protected. It also makes the backup policy auditable - you can query which resources carry the tag and verify coverage in one pass.\nTesting the Restore The backup jobs for both EC2 (webapp-01) and S3 completed successfully.\nCompleting a backup is the easy part. The only thing that matters is whether you can restore from it under pressure. Before running the test, confirm there is a completed recovery point in backup-vault-dev – if the scheduled job shows as aborted, trigger an on-demand backup from the vault first. Here is how the test was run:\nSimulate failure - EC2 \u0026gt; select web-dev-01 \u0026gt; Instance state \u0026gt; Terminate. Note the exact time – this is T0. Restore - AWS Backup \u0026gt; Backup vaults \u0026gt; backup-vault-dev \u0026gt; find the EC2 recovery point \u0026gt; Restore. Instance type: t2.micro. IAM role: Default role. Note the exact time you click Restore – this is T1. Confirm recovery - Go to EC2 and wait for the new instance to reach running state. Note that time – this is T2. RTO = T2 - T0.\nEvent Time T0 - instance terminated 22:36 T1 - restore initiated 22:38 T2 - instance running 22:39 EC2 RTO: 3 minutes.\nThe restored instance came up with all data intact – confirming the recovery point was consistent and the restore procedure worked as expected.\nThat number is specific to this instance size and snapshot state. In a production context, you'd add application startup time, health check validation, and any DNS or load balancer propagation on top. The infrastructure recovery itself was 3 minutes - what that means for service RTO depends on the application layer above it.\nA Note on S3 Restore S3 restore via AWS Backup works differently from EC2. It restores to a new bucket rather than back into the existing one, which means you need the restore role to have the right permissions to create a bucket in the target account - a common oversight on first setup.\nFor object-level recovery, S3 versioning handles it more cleanly: delete the object, remove the delete marker, file is back. For full bucket recovery AWS Backup is the right tool, but the IAM permissions need to be right upfront.\nProject 2 - AWS Resilience Hub Assessment The Problem Having backups in place answers one question: can we recover data? Resilience Hub answers a different question: does this workload actually meet its defined RTO and RPO targets, and what's missing?\nThe distinction is important. A workload can have backups and still fail a resilience assessment - because it has no alerting, no runbooks, no tested failure scenarios. Resilience Hub surfaces those gaps systematically.\nSetup at a Glance Create application - AWS Resilience Hub \u0026gt; Applications \u0026gt; Add application. Named backup-lab-app. Resources imported using the same Backup: true tag from Project 1, which pulled in the EC2 instance and S3 bucket automatically. Resiliency policy - Created a new policy with RTO 1 hour / RPO 24 hours, tier set to Non-critical. Attached to the application before publishing. IAM - Resilience Hub requires a service role with AWSResilienceHubAsssessmentExecutionPolicy attached. This is created automatically on first use if you allow it, or you can pre-create it manually and pass the role ARN when setting up the application. Initial assessment - Published the application, then ran the assessment from the application dashboard. Score: 40/100. Implement recommendation - CloudWatch \u0026gt; Alarms \u0026gt; Create alarm. Used the exact name from the Resilience Hub recommendation: AWSResilienceHub-Ec2CpuUtilizationAlarm_2020-07-13. Threshold set to 90% CPU utilisation over a 5-minute period. Reassessment - Re-ran the assessment from the Resilience Hub application dashboard. Score updated to 41/100. Defining the Policy Before running an assessment, you define a resiliency policy with explicit RTO and RPO targets. I used:\nRTO: 1 hour RPO: 24 hours These are appropriate for a non-critical workload. In a financial services context you'd tighten this considerably - a trading platform or payment service might require an RTO measured in minutes and an RPO of near-zero. The policy tier drives the strictness of the assessment, so getting it right for the workload type matters.\nInitial Assessment: 40/100 The initial score was 40/100. The RTO/RPO policy showed zero breaches - the workload could theoretically meet the recovery targets. The gaps were operational: 14 recommended CloudWatch alarms not implemented, 4 SOPs missing.\nThis is a realistic result for a single EC2 instance with no auto-recovery configured. The score reflects operational readiness, not just whether the infrastructure can recover. A workload with no alerting has no way of knowing it needs to recover in the first place.\nActioning a Recommendation From the 14 recommended alarms, I implemented the CPU utilisation alarm: AWSResilienceHub-Ec2CpuUtilizationAlarm_2020-07-13.\nThe naming convention matters here. Resilience Hub discovers implemented alarms by exact name match against its recommendation templates. If the name doesn't match precisely, the alarm won't register as implemented and the score won't move. Worth knowing before you spend time creating alarms that don't get picked up.\nReassessment: 41/100 Score moved from 40 to 41. One alarm implemented, 13 remaining. The value here isn't the number - it's the closed loop. Assessment identified a gap, a control was implemented, reassessment confirmed it. That's the workflow.\nIn production you'd work through the full recommendation set systematically: alarms first, then SOPs, then FIS experiments for chaos validation. Each pass raises the score and - more importantly - raises the operational confidence in the workload.\nWhat's Next The logical next steps from here are chaos testing with AWS Fault Injection Service - injecting EC2 failures and AZ-level disruptions to validate that the recovery procedures actually work under fire, not just in theory. Alongside that, RDS backup validation with a tested restore and measured RTO, and Vault Lock configuration for immutable backup retention to satisfy compliance requirements around tamper-proof recovery points. Those will be covered in a follow-up post.\nTakeaways A few things worth carrying from this:\nBackup strategy is about recovery confidence, not backup frequency. The retention tiers, the tag-based selection, the KMS key - those are all in service of being able to answer \u0026quot;can we recover, and how fast?\u0026quot; with a real number rather than a guess.\nResilience Hub is most useful as a gap analysis tool. The score itself is less important than the list of what's missing. Work the list.\nTest your restores. Every time, not just once.\n","link":"https://systemsgolive.com/post/aws-backup-resilience-hub-rto-validation/","section":"post","tags":["AWS","Backup","Resilience","RTO","RPO"],"title":"Building and Validating an AWS Backup Strategy with Resilience Hub"},{"body":"","link":"https://systemsgolive.com/tags/resilience/","section":"tags","tags":null,"title":"Resilience"},{"body":"","link":"https://systemsgolive.com/tags/rpo/","section":"tags","tags":null,"title":"RPO"},{"body":"","link":"https://systemsgolive.com/tags/rto/","section":"tags","tags":null,"title":"RTO"},{"body":"","link":"https://systemsgolive.com/tags/blameless/","section":"tags","tags":null,"title":"Blameless"},{"body":"","link":"https://systemsgolive.com/categories/career/","section":"categories","tags":null,"title":"Career"},{"body":"","link":"https://systemsgolive.com/tags/devops/","section":"tags","tags":null,"title":"Devops"},{"body":"","link":"https://systemsgolive.com/tags/incident-management/","section":"tags","tags":null,"title":"Incident-Management"},{"body":"","link":"https://systemsgolive.com/tags/infrastructure/","section":"tags","tags":null,"title":"Infrastructure"},{"body":"","link":"https://systemsgolive.com/tags/postmortem/","section":"tags","tags":null,"title":"Postmortem"},{"body":"","link":"https://systemsgolive.com/tags/sre/","section":"tags","tags":null,"title":"Sre"},{"body":"","link":"https://systemsgolive.com/categories/sre/","section":"categories","tags":null,"title":"SRE"},{"body":"Most engineers are great at firefighting. The alert fires, you jump in, you find the issue, you push the fix, you close the ticket. Job done. But that's just table stakes. What separates good engineers from truly exceptional ones isn't how fast they fix things. It's what they do after.\nWriting a postmortem is one of those habits that will genuinely make you stand out — not just to your team, but to leadership, to hiring managers, and frankly to yourself.\nWait, What Even Is a Postmortem? A postmortem is a structured written record of what went wrong, why it went wrong, and what you're going to do to make sure it never happens again.\nGoogle's SRE team literally wrote the book on this. A blameless postmortem culture, they argue, is one of the most powerful tools an engineering organisation can have. The goal isn't to find someone to blame. It's to understand the system well enough to make it more resilient.\n\u0026quot;A truly blameless postmortem culture results in more reliable systems.\u0026quot; — Google SRE Workbook, Chapter 10\nIt's a Mindset, Not Just Documentation Anyone can follow a runbook and fix an SSL cert at 2am. What not everyone does is sit down the next morning and ask: why did this happen? Could I have seen it coming? What does this tell me about my system?\nThat's the postmortem mindset. Here's why it sets you apart:\nYou think beyond the fix. Not just patching the symptom — understanding the disease. You demonstrate ownership. You understand how the whole thing fits together, not just your slice of it. You show leadership. Writing something that stops your whole team hitting the same wall? That's a leadership contribution, full stop. You build institutional memory. Future-you will be grateful. So will whoever comes after you. I've worked in infrastructure long enough to know — the engineers who write these things, and write them well, are the ones who get trusted with the bigger problems.\nBlameless. Always. This is the bit most people get wrong. Postmortems are not about naming who broke prod.\nEven if a human made a mistake, the real question is: why was it possible for that mistake to cause this much damage? Could a better alert have caught it earlier? Could a rate limit have slowed the blast radius? Was the documentation actually clear?\nBlame kills learning. Blamelessness creates safety. And safety is what makes people flag near-misses before they become outages.\nA Real-World Example 07:30 on a Monday. Your web server stops serving HTTPS. Blackbox monitoring fires — every check returning 502. On-call gets paged within minutes.\nAfter digging, you find the SSL certificate expired over the weekend. The renewal cron job failed silently — it lost write access to the cert directory after a SELinux policy update three weeks earlier. Nobody noticed. Certificate renewed manually at 08:15. Services restored.\nNow — did you just close the ticket and move on?\nA good engineer writes the postmortem. They capture how it was detected (the alert), how it was resolved (manual renewal), and what actually caused it (SELinux context change silently broke the automation). Then — the part that really matters — they add action items: a Prometheus alert for certs expiring within 14 days, a CI test to dry-run renewal on every deploy, and an audit of other cron jobs that might have the same SELinux problem.\nThat 45-minute fix just became a permanent improvement. And the engineer who documented it? They've shown they understand the system at a level most people never bother with. Notice what's missing from that write-up: names. No one got blamed. The system got fixed.\nThe Template I built this off Google's SRE postmortem principles and made it available so you don't have to start from scratch every time something breaks. These sections cover most incidents. For bigger ones – multi-team outages, revenue impact, prolonged recovery – you'll want more: Background, Team Impact, Revenue Impact, Where We Got Lucky, a Glossary. Use what fits. Skip what doesn't.\n📎 Download the template – grab it here.\nField Description Title Short, descriptive summary Date When did it happen? Authors Who wrote this? Status Draft / In Review / Final Severity SEV1 (critical) → SEV3 (low) Duration Start time → End time 1. Executive Summary Two or three sentences max. What happened, what broke, how was it fixed? If a VP reads only this, do they get it?\n2. Detection How did you find out? Alert, customer complaint, someone noticing in a dashboard? This section tells you how good your visibility actually is — and it's often uncomfortable reading.\nWhat triggered the first alert? How long between the incident starting and the team being paged? 3. Impact What broke and for how long? How many users or requests were affected? Any SLA/SLO breaches? 4. Root Cause Don't stop at the surface. Use the 5 Whys — keep digging until you hit something systemic.\n❌ \u0026quot;The cert expired because nobody renewed it.\u0026quot;\n✅ \u0026quot;The renewal cron job failed silently after a SELinux policy change removed write access to the cert directory. No alerts existed for renewal failures or upcoming expiry.\u0026quot;\n5. Timeline Factual. Chronological. No editorialising. Times matter.\nTime (UTC) Event 07:30 Blackbox alert fires: HTTPS returning 502 07:34 On-call paged, investigation begins 07:52 Root cause identified: expired certificate 08:15 Cert renewed manually, services restored 6. Resolution What did you actually do to fix it? Hotfix, rollback, config change? How long did full recovery take? Be specific — vague resolutions make for useless postmortems.\n7. What Went Well Always include this. It's not fluff — it reinforces what's working and keeps the tone constructive.\nBlackbox monitoring caught the outage within minutes Stakeholders were kept informed throughout 8. What Could Have Gone Better No alerting on certificate expiry No automated test validating the renewal process SELinux change wasn't cross-checked against dependent services 9. Action Items The most important section. Vague action items are useless. Each one needs an owner, a type, and a tracking reference.\nAction Type Priority Owner Prometheus alert: certs expiring within 14 days Detect P1 Mickael CI test: dry-run cert renewal on deploy Prevent P1 Team Audit cron jobs affected by SELinux changes Prevent P2 Mickael Update runbook with SELinux troubleshooting steps Mitigate P3 Team 10. Lessons Learned What did this incident reveal about your system? What assumptions were wrong? What change would prevent a whole class of similar issues — not just this one?\nFinal Thoughts It doesn't have to be 10 pages. A tight one-pager with a solid root cause and three action items is worth more than a war-and-peace document nobody reads.\nThe engineers I respect most aren't the ones who never break things. They're the ones who, when things break, make sure the whole team is smarter for it. That's the difference between being good at your job and making the system better.\nThat's the blameless postmortem mindset. And it will make you stand out.\nFurther Reading: Google SRE Workbook – Chapter 10 — sre.google/workbook/postmortem-culture/\n","link":"https://systemsgolive.com/post/the-blameless-postmortem-mindset-why-engineers-who-write-them-stand-out/","section":"post","tags":["postmortem","incident-management","blameless","sre","devops","infrastructure"],"title":"The Blameless Postmortem Mindset: Why Engineers Who Write Them Stand Out"},{"body":"","link":"https://systemsgolive.com/tags/diagnostics/","section":"tags","tags":null,"title":"Diagnostics"},{"body":"","link":"https://systemsgolive.com/tags/heap-dump/","section":"tags","tags":null,"title":"Heap Dump"},{"body":"The Problem When our care web app started experiencing CPU spikes on the production Tomcat servers, the software engineering team needed answers fast. What was the JVM doing at that exact moment? Was memory the problem? Were threads getting stuck? I needed a repeatable, safe way to capture a snapshot of the JVM mid-spike - without making things worse.\nA Bit of Background - The JVM Our care web app is a Java Spring application running on Apache Tomcat. When Tomcat starts, it runs inside the JVM - the Java Virtual Machine. Think of the JVM as the engine keeping the application alive. It manages memory, runs threads to handle incoming requests, and periodically cleans up unused objects through Garbage Collection (GC).\nThe Heap - Where Memory Lives Inside the JVM, all Java objects are stored in a region of memory called the heap. Two numbers define it:\nHeap capacity - the maximum memory the JVM is allowed to use, set via the -Xmx flag. On our servers this was fixed at 22 GB, reserved from physical RAM at startup. Heap in use - how much is actually occupied by live objects right now. 1Heap capacity: 22 GB (the ceiling - fixed at startup) 2Heap in use: 15 GB (what is actually used right now) 3Usage: 68% When heap in use climbs too close to the capacity ceiling, the Garbage Collector works harder and harder to free space. That GC pressure is one of the most common causes of CPU spikes in Java applications.\nThreads - the Workers Inside the JVM The JVM also manages threads - Tomcat spins up one per incoming HTTP request. Under pressure, threads can end up stuck:\nRUNNABLE - actively executing WAITING - idle, waiting to be woken up BLOCKED - waiting for another thread to release a lock A CPU spike often means a large number of threads in RUNNABLE or BLOCKED state - the JVM thrashing rather than making progress.\nThe Two Diagnostic Captures To understand what the JVM is doing during a spike, there are two standard captures:\nHeap dump - a complete snapshot of everything in heap memory at a single point in time. Every object, its size, and what is holding a reference to it.\nThread dump - a snapshot of every thread and what it is doing at that exact moment.\nTogether they give the engineering team the full picture of JVM state during an incident.\nEstimating the Heap Dump File Size Before capturing anything, it is worth knowing how large the file will be:\nHeap dump size ≈ heap currently in use at time of capture\nThe dump only captures occupied memory - not the full 22 GB capacity. I checked the live heap on each server before doing anything:\n1sudo /usr/lib/jvm/jdk-19.0.1/bin/jcmd \u0026lt;PID\u0026gt; GC.heap_info Which returned:\n1garbage-first heap total 23068672K, used 15966281K That is 22 GB total, 15.2 GB in use - so the dump file would be approximately 15 GB. Knowing this upfront meant I could confirm there was enough disk space before proceeding.\nValidating the JVM Before Capturing Before running the script, I ran these commands manually on each server to confirm everything was in order.\nStep 1 - Confirm Tomcat is running and identify the PID and JVM path\n1ps -ef | grep tomcat Look for the line with org.apache.catalina.startup.Bootstrap - the PID is the second column and the Java binary path is at the start of the command.\nStep 2 - Find the correct JDK tools to use\n1sudo find / -name \u0026#34;jcmd\u0026#34; 2\u0026gt;/dev/null This finds all JDK installations on the server. Cross-reference the path with what you saw in Step 1 - you want the jcmd that matches the Java binary Tomcat is actually using.\nStep 3 - Set the PID\n1PID=\u0026lt;number from Step 1\u0026gt; Explicitly set this so all subsequent commands target the correct Tomcat process.\nStep 4 - Check disk space\n1df -h Get the full picture across all partitions - not just /tmp.\nStep 5 - Check actual heap in use\n1sudo /usr/lib/jvm/jdk-19.0.1/bin/jcmd $PID GC.heap_info Example output:\n1garbage-first heap total 23068672K, used 15966281K 2 region size 16384K, 686 young (11239424K), 5 survivors (81920K) 3Metaspace used 720390K, committed 744512K, reserved 1769472K 4 class space used 68642K, committed 81408K, reserved 1048576K The used value is what matters - that is your estimated heap dump file size. Once all five checks pass, the server is ready for the script.\nManual Capture Steps These are the individual commands the script automates. Useful to know if you ever need to run them manually or want to understand exactly what the script is doing under the hood.\nStep 6 - Take the heap dump\n1sudo /usr/lib/jvm/jdk-19.0.1/bin/jmap -dump:format=b,file=/tmp/heapdump-$(hostname)-$(date +%Y%m%d%H%M%S).hprof $PID This connects to the live JVM and writes a full memory snapshot to disk. The JVM will pause briefly during capture - this is normal. The output file is a .hprof binary file sized roughly equal to the heap in use.\nStep 7 - Take the thread dump (3 snapshots, 30 seconds apart)\n1for i in {1..3}; do 2 echo \u0026#34;--- Dump $i at $(date) ---\u0026#34; \u0026gt;\u0026gt; /tmp/threaddump-$(date +%Y%m%d).txt 3 sudo /usr/lib/jvm/jdk-19.0.1/bin/jstack -l $PID \u0026gt;\u0026gt; /tmp/threaddump-$(date +%Y%m%d).txt 4 sleep 30 5done Three snapshots thirty seconds apart gives a view of thread behaviour over time rather than a single moment. This runs for approximately 90 seconds total.\nStep 8 - Verify both files were created\n1ls -lh /tmp/heapdump-*.hprof 2ls -lh /tmp/threaddump-*.txt Confirm both files exist and the sizes look as expected before doing anything else.\nStep 9 - Quick scan for BLOCKED threads\n1grep -A 5 \u0026#34;BLOCKED\u0026#34; /tmp/threaddump-$(date +%Y%m%d).txt Any threads in BLOCKED state are waiting for a lock held by another thread - a signal of potential deadlock or contention worth flagging to the engineering team immediately.\nStep 10 - Copy files to your local machine\n1scp user@tomcat-01:/tmp/heapdump-*.hprof . 2scp user@tomcat-01:/tmp/threaddump-*.txt . Downloads both dump files for analysis. The .hprof heap dump requires Eclipse MAT to open - it is a binary file and cannot be read in a text editor.\nStep 11 - Clean up the server\n1sudo rm /tmp/heapdump-*.hprof 2rm /tmp/threaddump-*.txt Heap dump files are large - remove them once safely transferred to free up disk space.\nWhere These Tools Come From The tools used to capture dumps - jmap, jstack, and jcmd - are not third-party software. They are bundled inside the JDK (Java Development Kit) installed on the server:\n1/usr/lib/jvm/jdk-19.0.1/bin/jmap \u0026lt;- takes heap dumps 2/usr/lib/jvm/jdk-19.0.1/bin/jstack \u0026lt;- takes thread dumps 3/usr/lib/jvm/jdk-19.0.1/bin/jcmd \u0026lt;- checks heap info, triggers GC One thing worth knowing: the JRE (Java Runtime Environment) is enough to run a Java app, but it does not include these diagnostic tools. Only the full JDK does. Confirming the JDK is present is the first check before any diagnostic work.\nThe Diagnostic Script Rather than running commands manually under pressure during a live incident, I built a shell script to handle the full capture process safely and consistently. The script is available on my GitHub - jvm_diagnostics.sh.\nAt a high level it does the following:\nStep 1 - Verify Tomcat is running. Detects the process, grabs the PID, and locates jmap, jstack, and jcmd automatically. Aborts if anything is missing.\nStep 2 - Check disk space. Verifies the filesystem has enough room before writing anything. Aborts if usage is above the safety threshold.\nStep 3 - Check heap usage. Queries the live JVM, calculates heap usage percentage, and estimates the dump file size. If the heap is critically full it skips the heap dump to protect the JVM - but still takes the thread dump.\nStep 4 - Human checkpoint. Pauses and shows a summary before doing anything irreversible. Only proceeds on an explicit yes.\nStep 5 - Heap dump. Writes the full memory snapshot to disk as a timestamped .hprof file.\nStep 6 - Thread dumps. Captures three snapshots thirty seconds apart for a view of thread behaviour over time.\nStep 7 - Blocked thread scan. Automatically scans the thread dump for BLOCKED threads and surfaces them on screen immediately.\nStep 8 - Summary. Prints file locations, sizes, and the exact scp command to copy files off the server.\nTesting on Non-Production First Before touching production, I ran the script on our dev server to validate the full flow - tool detection, dump output, safety checks, and file sizes. It also gave me a feel for the brief JVM pause that occurs during a heap dump, which is important to be aware of on a live server.\nDeploying to Production Once validated, I deployed the script to all four production Tomcat servers. For now the script sits under the home directory:\n1/home/\u0026lt;user\u0026gt;/jvm_diagnostics.sh This can be moved to a more permanent location such as /opt/tomcat/bin/ once the team agrees on a standard. Before deploying I checked each server individually - disk space, current heap usage, and JDK tool availability - to confirm everything was ready.\nThe servers were now prepared. The next time a CPU spike occurred, the team had a single script to run to safely capture the full JVM state without guesswork.\nGetting the Files Off the Server Once captured, the dump files need to be transferred off the VM for the software engineers to analyse. The heap dump is a binary file - raw memory written directly to disk, not human-readable. It requires a specialist tool such as Eclipse MAT (Memory Analyzer Tool) to open and interpret. The thread dump is plain text and can be read or shared directly.\nThe script outputs the exact scp command to copy files to a local machine:\n1scp user@tomcat-01:/tmp/heapdump-tomcat-01-20260306.hprof . 2scp user@tomcat-01:/tmp/threaddump-tomcat-01-20260306.txt . What Comes Next My role as a DevOps Engineer stops here - getting the captures clean, complete, and safely off the servers. The deeper analysis belongs to the software engineering team, who need to look into the dumps from an application level perspective to understand what the code was doing and where the root cause lives.\nWhen the spike hits, the last thing you want is to be figuring out tooling under pressure. That groundwork is done.\n","link":"https://systemsgolive.com/post/jvm-diagnostics-java-spring/","section":"post","tags":["JVM","Java","Spring","Tomcat","DevOps","Diagnostics","Performance","Heap Dump","Thread Dump"],"title":"How I Prepared Our Java Spring App Servers to Capture JVM Diagnostics During a CPU Spike"},{"body":"","link":"https://systemsgolive.com/tags/java/","section":"tags","tags":null,"title":"Java"},{"body":"","link":"https://systemsgolive.com/categories/java/","section":"categories","tags":null,"title":"Java"},{"body":"","link":"https://systemsgolive.com/tags/jvm/","section":"tags","tags":null,"title":"JVM"},{"body":"","link":"https://systemsgolive.com/tags/performance/","section":"tags","tags":null,"title":"Performance"},{"body":"","link":"https://systemsgolive.com/categories/performance/","section":"categories","tags":null,"title":"Performance"},{"body":"","link":"https://systemsgolive.com/tags/spring/","section":"tags","tags":null,"title":"Spring"},{"body":"","link":"https://systemsgolive.com/tags/thread-dump/","section":"tags","tags":null,"title":"Thread Dump"},{"body":"","link":"https://systemsgolive.com/tags/tomcat/","section":"tags","tags":null,"title":"Tomcat"},{"body":"","link":"https://systemsgolive.com/tags/almalinux/","section":"tags","tags":null,"title":"AlmaLinux"},{"body":"","link":"https://systemsgolive.com/tags/crc32/","section":"tags","tags":null,"title":"CRC32"},{"body":"","link":"https://systemsgolive.com/categories/infrastructure/","section":"categories","tags":null,"title":"Infrastructure"},{"body":"","link":"https://systemsgolive.com/tags/ldap/","section":"tags","tags":null,"title":"LDAP"},{"body":"Background Recently I was asked to help troubleshoot an LDAP service that wasn't starting on a newly created VM. The VM in question, SNAP-198, was a snapshot of our Pre-Release server, spun up to test something in isolation — hence the name. Sounds straightforward — but as soon as the snapshot was booted, LDAP was in a failed state.\nEnvironment Component Version OS AlmaLinux 8.10 (Cerulean Leopard) LDAP OpenLDAP 2.5.18 The Problem The issue was a classic snapshot trap. When you snapshot a running server, the new VM inherits everything — including the source server's identity. LDAP is particularly sensitive to this because it uses its own hostname to identify itself in the configuration.\nThe errors in the logs made it clear:\n1daemon: bind() failed errno=99 (Cannot assign requested address) 2read_config: no serverID / URL match found LDAP was trying to bind to an address that didn't exist on this new server, and couldn't match its own identity in the config.\nRoot Causes After digging in, there were four things that needed fixing:\nSLAPD_URLS still referencing the Pre-Release server hostname olcServerID in the LDAP config still pointing to the Pre-Release server SNAP-198's hostname resolving to 127.0.0.1 (loopback) instead of its actual public IP Replication config still pointing back to Pre-Release — not needed since SNAP-198 is a standalone server The Fix 1. Update SLAPD URLs — /etc/sysconfig/slapd 1sudo vim /etc/sysconfig/slapd 1SLAPD_URLS=\u0026#34;ldapi:/// ldap://SNAP-198 ldap://localhost\u0026#34; Verify the change:\n1sudo cat /etc/sysconfig/slapd 1## Reason: Give access to file to specific group. This can now be edited as required ## 2#SLAPD_URLS=\u0026#34;ldapi:/// ldap://Pre-Prod ldap://localhost\u0026#34; 3SLAPD_URLS=\u0026#34;ldapi:/// ldap://SNAP-198 ldap://localhost\u0026#34; 2. Fix /etc/hosts Remove the loopback entry for the hostname and map it to the actual public IP:\n1sudo vim /etc/hosts 1\u0026lt;public-ip\u0026gt; SNAP-198 SNAP-198 3. Fix olcServerID This is where it got interesting. The right way to modify LDAP config is through ldapmodify — the config files explicitly say do not edit directly. So I prepared the ldif file:\n1sudo vim /tmp/fix_serverid.ldif 1dn: cn=config 2changetype: modify 3replace: olcServerID 4olcServerID: 0 ldap://SNAP-198 And tried to apply it:\n1sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f /tmp/fix_serverid.ldif But this failed immediately — ldapmodify requires slapd to be running, and slapd wouldn't start because of the bad config. A classic chicken and egg situation.\nThe only way out was to edit the config file directly and manually recalculate the CRC32 checksum that OpenLDAP uses to validate the file:\n1sudo vim /etc/openldap/slapd.d/cn=config.ldif Change:\n1olcServerID: 0 ldap://Pre-Release 2olcServerID: 1 ldap://Other-Server To:\n1olcServerID: 0 ldap://SNAP-198 OpenLDAP stores a CRC32 checksum at the top of cn=config.ldif. When slapd starts, it recalculates the checksum from the file content and compares it against the stored value. If they don't match, slapd rejects the file:\n1ldif_read_file: checksum error on \u0026#34;/etc/openldap/slapd.d/cn=config.ldif\u0026#34; Editing the file directly changes the content but leaves the stored checksum pointing to the old version. Recalculating it and updating the value at the top of the file tells slapd the content is valid.\nThen recalculate the CRC32:\n1sudo python3 -c \u0026#34; 2import binascii 3with open(\u0026#39;/etc/openldap/slapd.d/cn=config.ldif\u0026#39;, \u0026#39;r\u0026#39;) as f: 4 lines = f.readlines() 5content = \u0026#39;\u0026#39;.join(lines[2:]) 6crc = binascii.crc32(content.encode()) \u0026amp; 0xffffffff 7print(f\u0026#39;{crc:08x}\u0026#39;) 8\u0026#34; Update the # CRC32 \u0026lt;value\u0026gt; line at the top of the file with the new value.\n4. Remove Replication Config Now that slapd was up, I could use ldapmodify properly to remove the unwanted replication entries.\nCreate the ldif files:\n1sudo vim /tmp/fix_config_syncrepl.ldif 1dn: olcDatabase={0}config,cn=config 2changetype: modify 3delete: olcSyncrepl 1sudo vim /tmp/fix_mdb_syncrepl.ldif 1dn: olcDatabase={1}mdb,cn=config 2changetype: modify 3delete: olcSyncrepl Apply both:\n1sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f /tmp/fix_config_syncrepl.ldif 2sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f /tmp/fix_mdb_syncrepl.ldif Verification 1sudo systemctl restart slapd 2sudo systemctl status slapd 3sudo grep -r \u0026#34;olcSyncrepl\u0026#34; /etc/openldap/slapd.d/ No output on the last command — SNAP-198 is now running as a fully standalone LDAP server with no dependency on Pre-Release.\nKey Takeaway When creating a VM snapshot from a running LDAP server, always update the server identity configuration before starting LDAP. The config doesn't just hold data — it holds the server's identity, and LDAP will refuse to start if that identity doesn't match the environment it's running in.\n","link":"https://systemsgolive.com/post/fixing-ldap-after-vm-snapshot/","section":"post","tags":["LDAP","OpenLDAP","Linux","AlmaLinux","Troubleshooting","SLAPD","CRC32","VM Snapshot"],"title":"LDAP Service Failure After VM Snapshot – Configuration Mismatch Resolution"},{"body":"","link":"https://systemsgolive.com/tags/linux/","section":"tags","tags":null,"title":"Linux"},{"body":"","link":"https://systemsgolive.com/tags/openldap/","section":"tags","tags":null,"title":"OpenLDAP"},{"body":"","link":"https://systemsgolive.com/tags/slapd/","section":"tags","tags":null,"title":"SLAPD"},{"body":"","link":"https://systemsgolive.com/categories/system-administration/","section":"categories","tags":null,"title":"System Administration"},{"body":"","link":"https://systemsgolive.com/tags/troubleshooting/","section":"tags","tags":null,"title":"Troubleshooting"},{"body":"","link":"https://systemsgolive.com/categories/troubleshooting/","section":"categories","tags":null,"title":"Troubleshooting"},{"body":"","link":"https://systemsgolive.com/tags/vm-snapshot/","section":"tags","tags":null,"title":"VM Snapshot"},{"body":"","link":"https://systemsgolive.com/tags/ad-blocking/","section":"tags","tags":null,"title":"Ad Blocking"},{"body":"","link":"https://systemsgolive.com/tags/dns/","section":"tags","tags":null,"title":"DNS"},{"body":"","link":"https://systemsgolive.com/tags/dnssec/","section":"tags","tags":null,"title":"DNSSEC"},{"body":"","link":"https://systemsgolive.com/tags/homelab/","section":"tags","tags":null,"title":"Homelab"},{"body":"","link":"https://systemsgolive.com/categories/homelab/","section":"categories","tags":null,"title":"Homelab"},{"body":"","link":"https://systemsgolive.com/categories/networking/","section":"categories","tags":null,"title":"Networking"},{"body":"","link":"https://systemsgolive.com/tags/pi-hole/","section":"tags","tags":null,"title":"Pi-Hole"},{"body":"Problem Statement Every device on a home network performs DNS lookups before connecting to a website. DNS (Domain Name System) is the mechanism that translates a domain name into an IP address – and the operator of the DNS resolver can observe every domain queried.\nIn a typical home network, those queries are handled by an ISP or a public DNS provider such as Google or Cloudflare:\n1Device → ISP DNS / Public DNS → Internet This means:\nYour ISP or DNS provider sees every domain queried All devices contribute to one central browsing history Ads and trackers resolve normally DNS becomes a record of your activity.\nGoals / Non-Goals Goals\nBlock advertising and tracking domains across all devices on the network Resolve DNS privately without relying on a third-party provider Require no configuration on individual devices Non-Goals\nInstallation is not covered. Refer to the official documentation:\nPi-hole Basic Install Unbound Setup for Pi-hole Solution Two services running on a Raspberry Pi 4 address the problem:\nService Role Pi-hole Blocks ads and trackers across all devices at the DNS level Unbound Locally resolves allowed domains using a recursive DNS resolver, eliminating reliance on external DNS resolvers What Is Pi-hole? Pi-hole acts as a DNS filtering layer.\nIt maintains a database of known ad, tracker, and malicious domains called gravity. When any device on the network queries a blocked domain, Pi-hole returns 0.0.0.0 – the request is stopped and the ad server is never contacted.\nIt works at the DNS level, before any content is loaded. No browser extension. No per-device setup. Every device on the network is covered automatically.\nWithout Pi-hole:\n1Device → Public DNS → Ad domain → Content loads With Pi-hole:\n1Device → Pi-hole → Blocklist match → 0.0.0.0 / NXDOMAIN returned → Connection fails What Is Unbound? Unbound is a validating, caching recursive DNS resolver.\nIt performs the same role as public DNS resolvers but can be hosted locally within a home or private network.\nInstead of forwarding queries to an external resolver, Unbound resolves domain names directly by querying the DNS hierarchy (root, Top-Level Domain, and authoritative servers).\nPublic DNS vs Recursive Resolver These terms are often confused.\nTerm Meaning Recursive resolver A DNS function or role Public DNS A resolver operated by a third party Google DNS, Cloudflare DNS, and Unbound all perform recursive resolution. The difference is who operates the resolver.\nDNS Resolution with Unbound The DNS system is organised as a hierarchy. Every domain name maps to a position within this tree:\n1 . (root) 2 │ 3 ┌────────────┼────────────┐ 4 .org .com .uk ← TLD (Top-Level Domain) 5 │ │ │ 6 wikipedia.org google.com bbc.co.uk ← second-level domain 7 │ 8 en.wikipedia.org ← subdomain When a device requests en.wikipedia.org, Unbound performs recursive resolution, locating the answer step by step:\nQueries the root servers to learn which servers manage .org Queries the .org TLD servers to find the authoritative servers for wikipedia.org Queries the authoritative servers for wikipedia.org — which store the domain’s official DNS records — to obtain the IP address for en.wikipedia.org Returns the result to Pi-hole, which then replies to the requesting device Public DNS providers such as Google or Cloudflare perform the same recursive process on external infrastructure. Because DNS queries are sent to their resolvers first, those providers can observe — and potentially log — the domains being requested.\nWhen Unbound runs locally, it performs this resolution itself by querying the DNS hierarchy directly instead of forwarding requests to a public resolver. As a result, no single third-party provider receives a complete history of DNS activity. Unbound also caches responses locally, allowing repeated queries to be answered faster and improving overall lookup performance.\nArchitecture The diagram below compares DNS query flows side by side – with Pi-hole and Unbound on the left, and without on the right – showing exactly where queries are intercepted, filtered, or exposed at each step.\nWith Pi-hole + Unbound\n1Devices → Pi-hole (filter + cache) → Unbound (recursive resolver) → Root → TLD → Authoritative servers Without Pi-hole + Unbound\n1Devices → ISP DNS / Public DNS → Internet How It Works Homelab Setup I set up Pi-hole and Unbound on a Raspberry Pi 4, connected to my home broadband router. Pi-hole is assigned a static IP address (192.168.1.143) and acts as the DNS server for the entire network.\nHardware Raspberry Pi 4 Model B – 4 GB RAM Raspberry Pi OS Lite 64-bit (headless) Static IP address: 192.168.1.143 Services Component Address Purpose Pi-hole 192.168.1.143:53 Network DNS server Unbound 127.0.0.1:5335 Local recursive resolver Router Configuration On my broadband router, I disabled DHCP and set the DNS Primary Server to Pi-hole's static IP:\nSetting Value DHCP Disabled DNS Primary Server 192.168.1.143 With DHCP disabled on the router, Pi-hole takes over as the DHCP server. All devices on the network automatically receive 192.168.1.143 as their DNS server – no manual configuration required on client devices.\nI also pointed the router's own DNS to Pi-hole, so router-originated queries are resolved locally.\nUnbound Binding Unbound listens only on localhost:\n1127.0.0.1:5335 This means:\nIt is not exposed to the network Only Pi-hole can query it External access is prevented Upstream DNS Configuration All third-party upstream DNS providers are disabled in Pi-hole.\nOnly Unbound is configured:\n1127.0.0.1#5335 This ensures DNS queries are resolved locally.\nPi-hole Configuration Blocklist Management Pi-hole includes a default blocklist on installation:\n1https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts This provides a baseline set of advertising and tracking domains.\nI added two additional lists under Pi-hole → Group Management → Lists. HaGeZi's DNS blocklists are modern, consolidated lists designed to replace multiple smaller ones — covering ads, trackers, telemetry, malware, scam domains, and threat intelligence feeds in a single maintained source.\nList URL HaGeZi Pro https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/adblock/pro.txt HaGeZi Threat Intelligence Feeds (medium) https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/adblock/tif.medium.txt After adding, rebuild the gravity database:\n1pihole -g Pi-hole Dashboard I access the dashboard at:\n1http://192.168.1.143/admin The dashboard provides:\nReal-time DNS query logs Top queried and blocked domains Blocking statistics Device activity overview Pi-hole + Unbound: Why Together? Pi-hole blocks unwanted domains but does not resolve allowed domains itself.\nPi-hole only:\n1Device → Pi-hole → Public DNS → Internet Advertising domains may be blocked, but DNS queries remain externally visible.\nWith Unbound:\n1Device → Pi-hole → Unbound → Root → TLD → Authoritative → Response DNS resolution occurs locally without reliance on public resolvers.\nDNSSEC – Validation DNSSEC adds cryptographic signatures to DNS responses.\nUnbound validates these signatures to confirm that responses originate from the legitimate domain authority.\nIf validation fails, the response is rejected.\nDNSSEC protects against tampered or forged DNS responses.\nVerifying DNSSEC Validation I tested DNSSEC validation directly against Unbound by running:\n1dig fail01.dnssec.works @127.0.0.1 -p 5335 2dig +ad dnssec.works @127.0.0.1 -p 5335 Invalid DNSSEC domain\nfail01.dnssec.works → SERVFAIL\nUnbound correctly rejects the response because DNSSEC validation fails.\nValid DNSSEC domain\ndnssec.works → NOERROR with ad flag\nNOERROR indicates a successful lookup The ad flag (Authentic Data) confirms the response was cryptographically validated using DNSSEC Summary:\nfail01.dnssec.works → rejected invalid DNSSEC data dnssec.works → validated and authenticated successfully Privacy vs Encryption These concepts address different concerns.\nConcept Description Privacy Who receives DNS queries Encryption Whether queries are hidden during transmission Unbound improves privacy by removing centralised DNS providers.\nDNS queries themselves remain standard DNS traffic.\nEven though Unbound does not rely on the ISP's DNS resolver, DNS queries still traverse the ISP's network connection. This means the ISP may observe outbound DNS traffic at the network level (in-transit), even though no single external resolver receives a complete history of DNS queries.\nConfirming the Setup From a Windows device on my network, I ran:\n1nslookup en.wikipedia.org This is what I got:\n1Server: pi.hole 2Address: 192.168.1.143 3 4Non-authoritative answer: 5Name: dyna.wikimedia.org 6Addresses: 185.15.59.224 7Aliases: en.wikipedia.org Interpretation:\nServer: pi.hole indicates Pi-hole handled the query The request was not sent directly to an ISP or public DNS provider Pi-hole forwarded the request internally to Unbound Verifying Pi-hole Uses Unbound as Upstream I ran the following tests to confirm that Pi-hole forwards queries to Unbound and that caching is working correctly.\nFirst Query – Recursive Resolution I ran:\n1dig fifa.com @127.0.0.1 2sudo tail /var/log/pihole/pihole.log | grep \u0026#34;fifa\u0026#34; 127.0.0.1 is the loopback address (localhost) — the Raspberry Pi itself. The @127.0.0.1 flag tells dig to send the query to Pi-hole running locally on the device. Pi-hole listens on the standard DNS port 53. It then forwards the query to Unbound, which listens on port 5335.\nWhat I observed:\nQuery time ≈ 135 ms Pi-hole log showed: 1forwarded fifa.com to 127.0.0.1#5335 2reply fifa.com is 2.19.248.207 3reply fifa.com is 2.19.248.224 Explanation:\nPi-hole forwarded the query to Unbound at 127.0.0.1#5335 Unbound performed recursive resolution by querying root, TLD, and authoritative DNS servers The higher query time occurs because the result was not yet cached If you see queries forwarded to 127.0.0.1#5335, Pi-hole is correctly using Unbound as its upstream resolver.\nSecond Query – Cached Lookup I ran the same query again:\n1dig fifa.com @127.0.0.1 2sudo tail -n 10000 /var/log/pihole/pihole.log | grep \u0026#34;fifa\u0026#34; This time:\nQuery time ≈ 0 ms The log showed cached responses: 1cached fifa.com is 2.19.248.207 2cached fifa.com is 2.19.248.224 Explanation:\nThe result is now served from Unbound's cache No external DNS queries are required Subsequent lookups are significantly faster Alternative Approaches DNS over HTTPS (DoH) and DNS over TLS (DoT) are encryption-focused alternatives. Both encrypt DNS queries in transit, preventing ISP-level interception.\nApproach Encrypts DNS in transit External resolver required Default DNS No Yes – ISP or public provider DoH / DoT Yes Yes – single provider (e.g. Cloudflare) Pi-hole + Unbound No No – resolved locally The trade-off is centralisation. DoH and DoT hide the content of DNS queries from the network path, but they route all traffic through a single provider, which then receives a complete picture of every domain queried across the network.\nThis setup takes a different position: DNS queries are not encrypted in transit, but no single external resolver ever sees the full query history. Queries are distributed across the global DNS hierarchy rather than aggregated by one provider.\nConclusion The DNS stack is managed locally within the home network:\nRouter → DHCP disabled; primary DNS set to 192.168.1.143 Pi-hole → acts as DHCP server; blocks advertising and tracking domains Unbound → performs recursive DNS resolution locally DNSSEC → cryptographically validates DNS responses Pi-hole Blocks advertising and tracking domains Operates across all network devices without per-device configuration Prevents unwanted connections before they occur Unbound Performs recursive DNS resolution locally Removes reliance on public DNS providers DNS requests are sent to Unbound (127.0.0.1#5335), which walks the DNS hierarchy directly — root → TLD → authoritative servers Distributes DNS queries across the global DNS hierarchy Result All devices automatically use Pi-hole for DNS resolution Ads and trackers are blocked at the DNS level No dependency on public DNS resolvers DNS responses are validated and cached locally on the Raspberry Pi for faster lookups DNS responses are DNSSEC-validated, ensuring records are authentic and have not been tampered with in transit Running Pi-hole and Unbound on a Raspberry Pi 4 was a deliberate homelab choice — prioritising privacy by keeping DNS resolution local rather than relying on external providers. DNS resolution remains under local control, with no centralised resolver maintaining a complete history of DNS activity and no ads reaching devices on the network.\nThis setup demonstrates how core internet infrastructure can be decentralised at home, restoring control over DNS resolution without sacrificing reliability or performance.\n","link":"https://systemsgolive.com/post/pihole-unbound-raspberrypi-homelab/","section":"post","tags":["Pi-hole","Unbound","DNS","Raspberry Pi","Homelab","Ad Blocking","Privacy","DNSSEC","Recursive DNS","Self-hosted"],"title":"Pi-hole and Unbound on Raspberry Pi 4: Private DNS and Network-Wide Ad Blocking"},{"body":"","link":"https://systemsgolive.com/tags/privacy/","section":"tags","tags":null,"title":"Privacy"},{"body":"","link":"https://systemsgolive.com/tags/raspberry-pi/","section":"tags","tags":null,"title":"Raspberry Pi"},{"body":"","link":"https://systemsgolive.com/tags/recursive-dns/","section":"tags","tags":null,"title":"Recursive DNS"},{"body":"","link":"https://systemsgolive.com/tags/self-hosted/","section":"tags","tags":null,"title":"Self-Hosted"},{"body":"","link":"https://systemsgolive.com/tags/unbound/","section":"tags","tags":null,"title":"Unbound"},{"body":"","link":"https://systemsgolive.com/tags/certificate-management/","section":"tags","tags":null,"title":"Certificate Management"},{"body":"","link":"https://systemsgolive.com/tags/ldaps/","section":"tags","tags":null,"title":"LDAPS"},{"body":"","link":"https://systemsgolive.com/tags/linux-security/","section":"tags","tags":null,"title":"Linux Security"},{"body":"","link":"https://systemsgolive.com/tags/selinux/","section":"tags","tags":null,"title":"SELinux"},{"body":"The Problem We recently had to renew a private SSL certificate used for internal LDAP access on one of our production web servers (e.g., prod-web-01). This certificate is not public-facing and is used only by the Support and DevOps teams internally, when connecting to the LDAP directory via Apache Directory Studio (i.e. LDAP client).\nThe certificate secures the LDAPS connection (port 636) between the LDAP client (GUI) and the LDAP service running on prod-web-01. This host provides internal LDAP services only, and the certificate is not shared with any external systems or applications. The certificate was issued by our private Root CA, hosted on a separate virtual machine and used solely for internal infrastructure.\nOn paper, this should have been a routine renewal.\nThe Plan (What Should Have Worked) The process was straightforward:\nGenerate a new CSR on the LDAP server Sign it using our private Root CA Copy the signed certificate back to prod-web-01 Replace the existing certificate files Our OpenLDAP TLS configuration already referenced fixed paths and filenames:\nTLSCertificateFile TLSCertificateKeyFile TLSCACertificateFile Since none of these changed, no configuration updates were required. A service restart should have been enough.\nThe Failure After copying the new certificates into place and restarting slapd, the service failed immediately with:\n1TLS init def ctx failed: -1 Not the result we were expecting.\nThe Investigation The usual checks came first:\nFile permissions were correct (644/640) Ownership was set properly (root:ldap) The certificate chain validated cleanly From a traditional Linux permissions perspective, everything looked fine. There was no obvious reason for LDAPS on port 636 to fail.\nThen I ran:\n1ls -alZ That's when the problem became clear.\nThe Culprit: SELinux Contexts Note: This article focuses on the practical troubleshooting of SELinux context issues. For a comprehensive understanding of SELinux and how it works, see Red Hat's guide: What is SELinux?\nWhat is SELinux? Security-Enhanced Linux (SELinux) is a security module that enforces mandatory access control policies by labeling files and processes with security contexts, adding an additional layer of security beyond traditional file permissions.\nThe newly generated certificate files had the SELinux context:\n1user_home_t instead of the expected:\n1cert_t Because the certificates were generated and transferred from a different system, they retained a home directory–style security label when placed on the LDAP server.\nSELinux enforces access based on security contexts, not just permissions. OpenLDAP requires certificate files to be labeled correctly; without the proper context, access is denied and TLS initialisation fails — even though everything else appears correct.\nThe Fix Restoring the correct SELinux contexts resolved the issue immediately:\n1restorecon -v /etc/openldap/certs/ldap-tls-gui-web01.crt 2restorecon -v /etc/openldap/certs/private-rootca-ldap.cert.pem Once the contexts were corrected, the LDAP service started successfully and LDAPS on port 636 was available again.\nLesson Learned On SELinux-enabled systems, TLS and certificate issues aren't always about permissions or ownership.\nAlways check the security context.\nThe ls -alZ command shows the full picture, and in this case, it led directly to the root cause. The fix persists across reboots, and internal LDAP access via the GUI is now fully restored.\n","link":"https://systemsgolive.com/post/ldap-cert-renewal-selinux-context-issue/","section":"post","tags":["LDAP","SELinux","SSL/TLS","OpenLDAP","Certificate Management","Linux Security","Troubleshooting","LDAPS"],"title":"SELinux Context Mismatch: The Hidden Culprit in LDAP Certificate Renewal"},{"body":"","link":"https://systemsgolive.com/tags/ssl/","section":"tags","tags":null,"title":"SSL"},{"body":"","link":"https://systemsgolive.com/tags/ssl/tls/","section":"tags","tags":null,"title":"SSL/TLS"},{"body":"TL;DR Problem: Standard HTTP-01 validation doesn't support wildcard certificates and breaks with Cloudflare's proxy (orange cloud) enabled. Solution: Use Certbot with the Cloudflare DNS plugin to perform DNS-01 validation — issues a wildcard cert covering all subdomains without disabling the Cloudflare proxy. Method: Install Certbot + Cloudflare DNS plugin, create a scoped API token, request the wildcard cert, configure Nginx, then automate renewal via a systemd timer and deploy hook. Result: Free, auto-renewing wildcard certificate on the origin server with Full (Strict) TLS end-to-end encryption via Cloudflare. Overview This case study demonstrates how to implement Full (Strict) SSL/TLS encryption by combining Cloudflare's managed frontend certificates with Let's Encrypt wildcard certificates on your origin server. This architecture provides:\nEnd-to-end encryption from browser to origin server Wildcard coverage for unlimited subdomains with a single certificate Automatic renewal every 60-90 days with zero downtime Free SSL certificates with industry-standard security Cloudflare proxy protection (orange cloud) maintained Why This Approach? Enhanced Security: Full (strict) mode ensures encrypted connections throughout the entire path Cost-Effective: Let's Encrypt certificates are free and auto-renew Scalability: One wildcard certificate covers all current and future subdomains Reliability: Cloudflare manages frontend certs while you control backend security Architecture Overview 1┌─────────┐ ┌────────────┐ ┌───────────────┐ 2│ Browser │ │ Cloudflare │ │ Origin Server │ 3└─────────┘ └────────────┘ └───────────────┘ 4https://virtualscale.dev 5or your domain 6 │ │ │ 7 │ │ │ 8 └─────HTTPS────────────┘ │ 9 (TLS 1.3) │ 10 (Cloudflare Cert) │ 11 Full (Strict) Mode │ 12 │ │ 13 └──────HTTPS──────────────┘ 14 (TLS 1.2/1.3) 15 (Let\u0026#39;s Encrypt Cert) 16 Auto-renews 17 Validates Certificate Key Points:\nFrontend: Cloudflare manages TLS certificates automatically Backend: Let's Encrypt ECDSA wildcard certificate (90-day validity, auto-renewal after 60 days) DNS Validation: Uses Cloudflare API for DNS-01 challenge (allows keeping orange cloud enabled) Prerequisites Before starting, ensure you have:\nDomain managed by Cloudflare DNS Nginx web server (or Apache) Root/sudo access to AlmaLinux server Cloudflare API token with DNS edit permissions Ports 80 and 443 allowed in firewall (see below) Firewall Configuration 1# Allow HTTP (needed for initial setup and renewals) 2# Source: Server IP → Destination: ANY → Port: TCP 80 → Action: Allow 3 4# Allow HTTPS (production traffic) 5# Source: Server IP → Destination: ANY → Port: TCP 443 → Action: Allow Step 1: Install EPEL Repository EPEL (Extra Packages for Enterprise Linux) provides Certbot packages.\n1# Check if EPEL is already installed 2sudo dnf repolist | grep epel 3 4# If no output, install EPEL 5sudo dnf install epel-release -y Step 2: Install Certbot and Cloudflare DNS Plugin What is Certbot? Certbot is the official Let's Encrypt client that automates the entire certificate lifecycle:\nRequests certificates from Let's Encrypt's Certificate Authority (CA) Validates domain ownership using various challenge methods Installs certificates on your web server Renews certificates automatically before expiration Think of Certbot as your automated certificate manager that handles all communication with Let's Encrypt's API.\nFor Nginx: 1# Check for existing Certbot installation 2rpm -qa | grep certbot 3 4# Install Certbot for Nginx 5sudo dnf install certbot python3-certbot-nginx -y 6 7# Install Cloudflare DNS plugin (required for DNS-01 challenge) 8sudo dnf install python3-certbot-dns-cloudflare -y 9 10# Verify installation 11certbot --version 12certbot plugins For Apache (Alternative): 1# Install Certbot for Apache 2sudo dnf install certbot python3-certbot-apache -y 3 4# Install Cloudflare DNS plugin 5sudo dnf install python3-certbot-dns-cloudflare -y 6 7# Optional: Verify SSL module is loaded 8sudo httpd -M | grep ssl_module 9 10# If not present, install SSL module 11sudo dnf install -y mod_ssl Note: Apache's default ssl.conf can conflict with custom vhost configurations. Review and adjust as needed.\nStep 3: Create Cloudflare API Token The API token allows Certbot to create DNS records for domain validation.\nSteps: Go to Cloudflare Dashboard → My Profile → API Tokens\nClick Create Token → Use Edit zone DNS template\nConfigure token:\nToken name: api-token-tls-letsencrypt-webserver Permissions: Zone → DNS → Edit Zone Resources: Include → Specific zone → yourdomain.com IP Filtering (optional): Add server's public IP for extra security TTL (optional): Set expiration if desired Click Continue to summary → Create Token\nCopy and save the token immediately (you won't see it again!)\nTest Your Token Verify the token works before proceeding:\n1curl \u0026#34;https://api.cloudflare.com/client/v4/user/tokens/verify\u0026#34; \\ 2 -H \u0026#34;Authorization: Bearer YOUR_CLOUDFLARE_API_TOKEN\u0026#34; Expected output should show \u0026quot;status\u0026quot;: \u0026quot;active\u0026quot;\nStep 4: Store Cloudflare Credentials Securely Security Best Practice: Generate one API token per webserver for better access control.\n1# Create secure directory 2sudo mkdir -p /root/.secrets 3 4# Create credentials file 5sudo vim /root/.secrets/cloudflare.ini Add this content:\n1# Cloudflare API Token for TLS/SSL Let\u0026#39;s Encrypt 2dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN_HERE Secure the file (critical step):\n1# Set restrictive permissions (owner read/write only) 2sudo chmod 600 /root/.secrets/cloudflare.ini 3 4# Verify permissions (should show -rw-------) 5sudo ls -l /root/.secrets/cloudflare.ini Step 5: Obtain Wildcard TLS Certificate Understanding DNS-01 Challenge What is DNS-01 Challenge?\nDNS-01 is one of several domain validation methods Let's Encrypt uses to verify you control a domain. Unlike HTTP-01 (which requires port 80 access), DNS-01 validates ownership by checking for a specific DNS TXT record.\nHow DNS-01 Works:\n11. You request a certificate from Let\u0026#39;s Encrypt 2 ↓ 32. Let\u0026#39;s Encrypt generates a unique challenge token 4 ↓ 53. Certbot creates a TXT record: _acme-challenge.yourdomain.com 6 ↓ 74. Let\u0026#39;s Encrypt\u0026#39;s servers query DNS for this TXT record 8 ↓ 95. If the record exists with the correct token → Validation successful 10 ↓ 116. Certificate is issued 12 ↓ 137. Certbot automatically removes the TXT record (cleanup) Why We Use DNS-01 for This Setup:\nWildcard Support: DNS-01 is the only method that supports wildcard certificates (*.domain.com) Cloudflare Proxy Compatible: Works even with Cloudflare's orange cloud (proxy) enabled No Port Requirements: Doesn't need port 80/443 exposed to the internet Server Location Flexible: Works regardless of server location or firewall rules How It Works with Let's Encrypt and Cloudflare The Flow in Detail:\nRequest: Certbot contacts Let's Encrypt API requesting a certificate for *.yourdomain.com Challenge: Let's Encrypt responds with a unique challenge string DNS Record Creation: Certbot uses the Cloudflare API (via your API token) to create: 1 _acme-challenge.yourdomain.com TXT \u0026#34;random_challenge_string\u0026#34; Propagation Wait: Certbot waits for DNS propagation (default 10s, we set it to 60s) Validation: Let's Encrypt queries public DNS servers to verify the TXT record exists Certificate Issuance: Upon successful validation, Let's Encrypt issues the certificate Cleanup: Certbot removes the temporary TXT record via Cloudflare API Why This Requires a Cloudflare API Token:\nCertbot needs programmatic access to create/delete DNS records in your Cloudflare account during the validation process. The API token provides secure, limited access specifically for DNS editing.\nRequest the Certificate This step requests the certificate using DNS-01 validation.\n1# Verify Certbot location 2which certbot 3# Expected: /usr/bin/certbot 4 5# Request wildcard certificate with ECDSA encryption 6sudo certbot certonly \\ 7 --dns-cloudflare \\ 8 --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \\ 9 --key-type ecdsa \\ 10 --elliptic-curve secp256r1 \\ 11 -d \u0026#39;*.virtualscale.dev\u0026#39; \\ 12 -d virtualscale.dev What happens during this process:\nCertbot creates temporary TXT records in Cloudflare DNS (_acme-challenge.yourdomain.com) Let's Encrypt verifies domain ownership via DNS Wildcard certificate is issued (valid for 90 days) TXT records are automatically cleaned up Certificate files are saved to /etc/letsencrypt/live/yourdomain/ Verify Certificate Installation 1# List certificate files 2sudo ls -la /etc/letsencrypt/live/virtualscale.dev/ 3 4# Expected output: 5# cert.pem -\u0026gt; ../../archive/virtualscale.dev/cert1.pem 6# chain.pem -\u0026gt; ../../archive/virtualscale.dev/chain1.pem 7# fullchain.pem -\u0026gt; ../../archive/virtualscale.dev/fullchain1.pem 8# privkey.pem -\u0026gt; ../../archive/virtualscale.dev/privkey1.pem Check Certificate Details 1# View certificate information 2sudo certbot certificates 3 4# Or check from the server itself 5openssl s_client -connect localhost:443 2\u0026gt;/dev/null | \\ 6 openssl x509 -noout -text -dates -issuer -subject Key Certificate Details:\nIssuer: Let's Encrypt (E7) Validity: 90 days Algorithm: ECDSA with SHA-384 Key Size: 256-bit (P-256 curve) Coverage: *.virtualscale.dev and virtualscale.dev Step 6: Install and Configure Nginx Install Nginx 1sudo dnf install nginx -y 2sudo systemctl start nginx 3sudo systemctl enable nginx 4sudo systemctl status nginx Create Basic Webpage 1# Create web root directory 2sudo mkdir -p /var/www/virtualscale.dev/html 3 4# Create index page 5sudo vim /var/www/virtualscale.dev/html/index.html 1\u0026lt;!DOCTYPE html\u0026gt; 2\u0026lt;html lang=\u0026#34;en\u0026#34;\u0026gt; 3\u0026lt;head\u0026gt; 4 \u0026lt;meta charset=\u0026#34;UTF-8\u0026#34;\u0026gt; 5 \u0026lt;meta name=\u0026#34;viewport\u0026#34; content=\u0026#34;width=device-width, initial-scale=1.0\u0026#34;\u0026gt; 6 \u0026lt;title\u0026gt;VirtualScale.dev\u0026lt;/title\u0026gt; 7 \u0026lt;style\u0026gt; 8 body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; } 9 h1 { color: #2c3e50; } 10 .secure { color: #27ae60; font-weight: bold; } 11 \u0026lt;/style\u0026gt; 12\u0026lt;/head\u0026gt; 13\u0026lt;body\u0026gt; 14 \u0026lt;h1\u0026gt;Welcome to VirtualScale.dev\u0026lt;/h1\u0026gt; 15 \u0026lt;p class=\u0026#34;secure\u0026#34;\u0026gt;🔒 Secured with Let\u0026#39;s Encrypt SSL\u0026lt;/p\u0026gt; 16 \u0026lt;p\u0026gt;This site uses \u0026lt;strong\u0026gt;ECDSA (Elliptic Curve Digital Signature Algorithm)\u0026lt;/strong\u0026gt; 17 with SHA-384 hashing for enhanced security.\u0026lt;/p\u0026gt; 18 \u0026lt;p\u0026gt;\u0026lt;small\u0026gt;ECDSA provides strong security with smaller key sizes compared to traditional RSA encryption.\u0026lt;/small\u0026gt;\u0026lt;/p\u0026gt; 19\u0026lt;/body\u0026gt; 20\u0026lt;/html\u0026gt; Configure Nginx for TLS 1sudo vim /etc/nginx/conf.d/virtualscale.dev.conf 1# Redirect all HTTP traffic to HTTPS 2server { 3 listen 80; 4 listen [::]:80; 5 server_name virtualscale.dev www.virtualscale.dev; 6 7 # 301 permanent redirect 8 return 301 https://$server_name$request_uri; 9} 10 11# HTTPS server with Let\u0026#39;s Encrypt certificate 12server { 13 listen 443 ssl; 14 listen [::]:443 ssl; 15 http2 on; 16 17 server_name virtualscale.dev www.virtualscale.dev; 18 19 root /var/www/virtualscale.dev/html; 20 index index.html; 21 22 # SSL Certificate Configuration (Let\u0026#39;s Encrypt) 23 ssl_certificate /etc/letsencrypt/live/virtualscale.dev/fullchain.pem; 24 ssl_certificate_key /etc/letsencrypt/live/virtualscale.dev/privkey.pem; 25 26 # Modern SSL Configuration (Security Best Practices) 27 ssl_protocols TLSv1.2 TLSv1.3; 28 ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384; 29 ssl_prefer_server_ciphers off; 30 31 # Enable HSTS (forces HTTPS for 2 years) 32 add_header Strict-Transport-Security \u0026#34;max-age=63072000\u0026#34; always; 33 34 # Additional security headers (optional but recommended) 35 add_header X-Frame-Options \u0026#34;SAMEORIGIN\u0026#34; always; 36 add_header X-Content-Type-Options \u0026#34;nosniff\u0026#34; always; 37 add_header X-XSS-Protection \u0026#34;1; mode=block\u0026#34; always; 38 39 location / { 40 try_files $uri $uri/ =404; 41 } 42} Test and Apply Configuration 1# Test configuration for syntax errors 2sudo nginx -t 3 4# If test passes, restart Nginx 5sudo systemctl restart nginx 6 7# Verify Nginx is running 8sudo systemctl status nginx 9 10# Check listening ports 11sudo ss -tlnp | grep -E \u0026#39;:(80|443)\u0026#39; Step 7: Set Up Automatic Certificate Renewal How Auto-Renewal Works Let's Encrypt certificates expire after 90 days for security reasons. Certbot handles automatic renewal through a systemd timer.\nThe Auto-Renewal Process:\n1Every 12 hours, the certbot-renew.timer triggers: 2 ↓ 31. Certbot checks all installed certificates 4 ↓ 52. If certificate expires in \u0026lt; 30 days → Renewal triggered 6 ↓ 73. Same DNS-01 challenge process runs automatically: 8 - Create TXT record via Cloudflare API 9 - Let\u0026#39;s Encrypt validates 10 - New certificate issued 11 - Old TXT record removed 12 ↓ 134. Deploy hooks run (reload Nginx) 14 ↓ 155. Your server now uses the new certificate (seamless!) Key Points:\nRenewal Window: Starts attempting renewal at 60 days (30 days before expiry) Retry Logic: If renewal fails, it retries automatically on the next scheduled run Zero Downtime: Nginx reload takes milliseconds; users don't notice No Manual Intervention: Fully automated as long as Cloudflare API token remains valid Timeline Example:\n1Day 0: Certificate issued (valid for 90 days) 2Day 60: First renewal attempt 3Day 61: If Day 60 failed, retry 4Day 62: If Day 61 failed, retry 5... 6Day 89: Final renewal attempts (1 day before expiry) 7Day 90: Certificate expires (if all renewals failed - rare!) Enable Auto-Renewal Timer 1# Enable timer to start on boot 2sudo systemctl enable certbot-renew.timer 3 4# Start timer immediately 5sudo systemctl start certbot-renew.timer 6 7# Verify timer status 8sudo systemctl status certbot-renew.timer Configure Nginx Reload After Renewal Why This Is Needed:\nWhen Certbot renews a certificate, it writes new certificate files to disk, but Nginx is still using the old certificates loaded in memory. The deploy hook ensures Nginx reloads and starts using the new certificates immediately after renewal.\nWithout the hook: New certificates sit unused until you manually restart Nginx\nWith the hook: Nginx automatically picks up new certificates within seconds\nCreate a deploy hook to automatically reload Nginx when certificates renew:\n1# Create deploy hooks directory 2sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy 3 4# Create reload script 5sudo vim /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh 1#!/bin/bash 2# Reload Nginx after Let\u0026#39;s Encrypt certificate renewal 3# This ensures the web server uses the newly renewed SSL certificates 4 5systemctl reload nginx 1# Make script executable 2sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh Test Auto-Renewal (Dry Run) 1# Perform dry run (doesn\u0026#39;t actually renew, just tests the process) 2sudo certbot renew --dry-run If successful, you should see: Congratulations, all simulated renewals succeeded\nStep 8: Increase DNS Propagation Timeout (Optional but Recommended) Prevents timeout failures during renewal by allowing more time for DNS changes to propagate.\n1# Edit renewal configuration 2sudo vim /etc/letsencrypt/renewal/virtualscale.dev.conf 3 4# Add this line in the [renewalparams] section: 5dns_cloudflare_propagation_seconds = 60 Verify the change:\n1grep \u0026#34;propagation\u0026#34; /etc/letsencrypt/renewal/virtualscale.dev.conf Expected output:\n1dns_cloudflare_propagation_seconds = 60 Troubleshooting Check Logs 1# View Certbot logs 2sudo tail -n 100 /var/log/letsencrypt/letsencrypt.log 3 4# View renewal service logs 5sudo journalctl -u certbot-renew.service -n 200 6 7# View Nginx error logs 8sudo tail -n 50 /var/log/nginx/error.log Common Issues Issue: DNS validation timeout\nSolution: Increase dns_cloudflare_propagation_seconds to 60 or 120\nIssue: Permission denied accessing Cloudflare credentials\nSolution: Verify file permissions are 600 and owned by root\nIssue: Nginx won't start after renewal\nSolution: Check for syntax errors with sudo nginx -t\nIssue: Certificate not covering subdomain\nSolution: Verify both *.domain.com and domain.com are in certificate\nVerification and Testing Verify SSL Certificate 1# Check certificate on local server 2openssl s_client -connect localhost:443 2\u0026gt;/dev/null | \\ 3 openssl x509 -noout -dates -issuer -subject 4 5# Expected output should show: 6# - Issuer: Let\u0026#39;s Encrypt 7# - Subject: CN=*.virtualscale.dev 8# - Valid dates (90 days from issue) Test External Access Visit your domain in a browser: https://virtualscale.dev Check certificate details (click lock icon in address bar) Verify: Certificate issuer: Let's Encrypt Encryption: TLS 1.3 (if supported by browser) Connection: Secure Visual Confirmation: Successful TLS Implementation Once configured correctly, your browser will display a secure connection indicator. Here's what a successful implementation looks like:\nWhat to look for in the browser:\nPadlock icon in the address bar (indicates HTTPS) \u0026quot;Connection is secure\u0026quot; message in certificate viewer \u0026quot;Certificate is valid\u0026quot; status No browser warnings about insecure connections This confirms your Let's Encrypt certificate is properly installed, trusted by browsers, and providing end-to-end encryption with Cloudflare's Full (strict) SSL/TLS mode.\nTest Auto-Renewal 1# Force renewal (for testing only - has rate limits!) 2sudo certbot renew --force-renewal 3 4# Better: Use dry run 5sudo certbot renew --dry-run Security Benefits Summary End-to-End Encryption: Full (strict) mode ensures TLS encryption from browser to origin\nModern Cryptography: ECDSA provides strong security with smaller key sizes\nAutomatic Updates: Certificates renew automatically before expiration\nWildcard Coverage: One certificate secures all subdomains\nZero Downtime: Renewal happens in background without service interruption\nCloudflare Protection: Maintains DDoS protection and CDN benefits\nFree Solution: No certificate costs while maintaining enterprise-grade security\nKey Takeaways Architecture: Cloudflare handles frontend certificates, Let's Encrypt secures origin server Validation Method: DNS-01 challenge allows wildcard certificates with Cloudflare proxy enabled Automation: Systemd timer + deploy hooks ensure hands-off operation Security: Full (strict) TLS mode provides maximum encryption throughout the connection Scalability: Wildcard certificates eliminate the need for per-subdomain certificate management Additional Resources Let's Encrypt Documentation Cloudflare TLS Full Strict Mode Documentation Certbot Documentation Nginx SSL Configuration Generator ","link":"https://systemsgolive.com/post/letsencrypt-wildcard-cloudflare-full-strict/","section":"post","tags":["TLS","SSL","Encryption","Certificate","Automation","Cloudflare"],"title":"Case Study - Automating Let's Encrypt Wildcard Certificates with Cloudflare DNS and Full (Strict) TLS Encryption"},{"body":"","link":"https://systemsgolive.com/tags/certificate/","section":"tags","tags":null,"title":"Certificate"},{"body":"","link":"https://systemsgolive.com/tags/cloudflare/","section":"tags","tags":null,"title":"Cloudflare"},{"body":"","link":"https://systemsgolive.com/tags/encryption/","section":"tags","tags":null,"title":"Encryption"},{"body":"","link":"https://systemsgolive.com/tags/tls/","section":"tags","tags":null,"title":"TLS"},{"body":"","link":"https://systemsgolive.com/categories/infrastructure-migration/","section":"categories","tags":null,"title":"Infrastructure Migration"},{"body":"","link":"https://systemsgolive.com/tags/load-balancer/","section":"tags","tags":null,"title":"Load Balancer"},{"body":"TL;DR Problem: ANS Load Balancer required manual SSL certificate uploads — a past missed renewal caused a production outage. HAProxy v2.0.31 was also EOL. Solution: Migrated to Cloudflare Load Balancer for automated certificate management and modern infrastructure. Method: Pre-configured everything using a dummy hostname, then executed a DNS switchover in a 45-minute maintenance window. Result: Zero downtime, stable traffic across four Tomcat instances, confirmed over 72 hours post-migration. Overview This article documents our seamless production migration from ANS Load Balancer to Cloudflare Load Balancer for our production care management web application.\nNote on ANS: ANS (not to be confused with AWS) is a UK-based cloud and managed services provider headquartered in Manchester, England. They offer cloud infrastructure, managed services, and data centre solutions primarily to UK organisations. More info at ans.co.uk.\nCurrent Setup Our domain eplancare.com is hosted on Cloudflare as our DNS provider. The production application is accessible at login.eplancare.com, which is our main customer-facing application serving healthcare providers and care managers who access our platform daily.\nCurrently, login.eplancare.com points to a Virtual IP (VIP) address managed by ANS, a third-party cloud provider in the UK. This VIP routes traffic to the ANS Load Balancer, which then distributes requests across our backend Tomcat web servers.\nMigration Goal We needed to migrate from the ANS Load Balancer to Cloudflare Load Balancer while maintaining the same application endpoint (login.eplancare.com). This migration was driven by operational overhead, security concerns, and the need for automated certificate management. The cutover was executed successfully with zero downtime during a planned maintenance window (10th November 2024, 04:30-05:15 UK Time).\nWhy We Migrated Operational Challenges with ANS Load Balancer Manual Certificate Management\nSSL/TLS certificates required manual upload through the ANS Portal. Manual process to delete expiring certificates and upload new ones. Limited API functionality for automation. High risk of outages due to forgotten certificate renewals. Security Concerns\nANS was running HAProxy version 2.0.31, which reached end-of-life on 05th April 2024. This version was no longer supported by the HAProxy vendor at the time of migration (October 2025). Reference: HAProxy End of Life tracking. Past Incidents\nWe experienced an outage when an SSL/TLS certificate wasn't uploaded in time due to the manual process. Benefits of Cloudflare Load Balancer Automated Certificate Management - Cloudflare handles certificate renewal automatically. Global Edge Network - Improved performance and reduced latency. Reduced Operational Overhead - No manual certificate uploads. Modern Infrastructure - Up-to-date, vendor-supported platform. Risk Consideration ⚠️ Potential Cloudflare Outage - If Cloudflare experiences issues, we could face downtime\nMitigation: We kept ANS LB as a backup solution (requires manual switchover, though). Architecture Overview The following diagram illustrates the final Cloudflare Load Balancer architecture after migration:\nTraffic flow from visitors through Cloudflare Load Balancer to backend Tomcat servers with configured weights\nMigration Approach Phase 1: Proof of Concept Objective: Validate Cloudflare LB functionality in a lower environment (non-production).\nSteps:\nSet up test Cloudflare Load Balancer in lower environment. Configured traffic distribution across 2 test web servers. Performed comprehensive testing. Result: Successfully validated.\nPhase 2: Planning and Preparation Implementation Planning\nDeveloped detailed implementation procedures. Created comprehensive rollback plan. Practiced both plans in lower environment. Special focus on rollback procedures. Change Control Process\nDocumented implementation and rollback plans. Coordinated migration date/time with development and operations teams. Scheduled during maintenance hours (outside business hours). Communicated plans to wider stakeholder audience. Phase 3: Pre-Configuration Strategy: Configure everything in advance using a dummy hostname, then switch to production hostname at go-live.\nWe used a dummy hostname (test.eplancare.com) to set up the entire configuration before switching to the actual production FQDN (login.eplancare.com). This minimised the cutover window.\nCloudflare Load Balancer Configuration Endpoint Configuration\nProvide Pool Name and Description. Provide Endpoint Names and Endpoint IP addresses, Ports and Weights for each endpoint. Steering Method: Least Outstanding Requests. Session Affinity Settings\nMethod: Cloudflare cookie only. Session TTL: 43,200 seconds (12 hours). Endpoint Drain Duration: 60 seconds. Zero Downtime Failover: Sticky. Health Check Configuration\nType: HTTPS. Method: GET. Port: 8443. Path: /healthcheck Interval: 120 seconds (between each health check). Timeout: 10 seconds (before marking as failed). Retries: 2 attempts. Check Regions: Western Europe. Monitor Configuration\nType: HTTPS. Path: /healthcheck. Port: 8443. Note: Health check and monitor configurations were applied to all 4 Tomcat instances (tomcat-01, tomcat-02, tomcat-03, tomcat-04).\nMigration Execution Go-Live: Mon 10th Nov 2025, 04:30 UK Time Maintenance Page Display Before draining traffic, I configured custom 503 error pages on all Tomcat web servers (tomcat-01, tomcat-02, tomcat-03, tomcat-04) to display a scheduled maintenance webpage on login.eplancare.com, informing users that the service was temporarily unavailable due to ongoing maintenance.\nTime Action Description 04:30 Sanity Check Verified web UI, Tomcat access logs, and traffic monitoring 04:40 Drain ANS LB Set all 4 Tomcat servers to drain mode (weight = 0) to stop new incoming traffic. Custom 503 maintenance page displayed to users accessing login.eplancare.com 04:50 Remove DNS A Record Removed existing DNS A record pointing to ANS LB VIP (via Cloudflare Portal) 04:55 Sanity Check Verification checkpoint 05:00 Switch to Cloudflare Updated login.eplancare.com to point to Cloudflare LB 05:05 Sanity Check Verification checkpoint 05:15 Extended Monitoring Monitored traffic distribution through peak time (09:00) and throughout the day Rollback Plan In the event of critical issues, I had a documented rollback procedure:\nAlign with team - Confirm rollback decision. Remove CF LB DNS entry - Remove login.eplancare.com from Cloudflare LB. Disable CF pool - Disable server pool for Tomcat endpoints in Cloudflare. Restore ANS DNS - Revert A record to ANS LB Virtual IP. Enable ANS servers - Put all 4 Tomcat VMs back into load. Monitor and verify - Check traffic and review logs. Note: Rollback was not required - included here for documentation completeness.\nResults and Monitoring Migration Outcome: Successful Immediate Results\nZero downtime experienced (beyond planned maintenance window) Seamless DNS cutover. First customer requests hit Cloudflare LB at 07:00 AM. Maintenance page successfully displayed during migration window. Traffic Distribution\nLoad balancing was distributed across all 4 Tomcat instances (tomcat-01, tomcat-02, tomcat-03, tomcat-04) according to the configured weights:\ntomcat-01: Weight = 0.5 (~14% of traffic). tomcat-02: Weight = 1.0 (~28.5% of traffic). tomcat-03: Weight = 1.0 (~28.5% of traffic). tomcat-04: Weight = 1.0 (~28.5% of traffic). This configuration was confirmed through Tomcat access logs and Cloudflare Analytics dashboard.\nTraffic Patterns Throughout the Day:\nThe following screenshots from Cloudflare Analytics demonstrate consistent load distribution matching the configured weights at different times during migration day:\n05:15 AM - Post-Cutover (Early Morning) Initial traffic distribution immediately after migration cutover\n09:00 AM - Morning Peak Load distribution during morning peak hours as users began accessing the system\n15:00 PM - Afternoon Activity Sustained distribution throughout afternoon operations\n72 Hours Post-Migration - Sustained Performance Three-day view showing consistent traffic patterns and stable load distribution across all Tomcat endpoints. The graph demonstrates normal business hour peaks with the load balancer maintaining the configured weight distribution (tomcat-02, tomcat-03, tomcat-04 each handling ~28.5% of traffic, and tomcat-01 handling ~14%). Total requests processed: 2.29M over 72 hours with no performance issues or anomalies detected.\nTraffic was monitored continuously through peak hours on migration day and for the entire week post-migration to ensure sustained performance and stability.\nNo Issues Observed\nNo customer complaints. No application errors. No performance degradation. Key Success Factors What Made This Migration Successful Thorough Testing - Proof of concept in lower environment validated the approach. Meticulous Planning - Detailed implementation and rollback procedures. Pre-Configuration Strategy - Using dummy hostname allowed us to configure everything in advance. Timing - Executed during off-hours maintenance window Team Coordination - Proper change control and stakeholder communication. Extended Monitoring - Week-long monitoring ensured sustained success. User Communication - Custom 503 maintenance page kept users informed during migration. Lessons Learned Best Practices for Load Balancer Migration Before Migration:\nAlways conduct proof of concept in a non-production environment. Pre-configure as much as possible to minimise the cutover window. Practice rollback procedures, not just implementation. Use dummy hostnames for staging configuration. Prepare custom maintenance pages to communicate with users. Share high-level migration plan and readiness status with relevant team members. During Migration:\nSchedule during maintenance windows. Build in multiple sanity check points. Drain traffic on all backend web servers simultaneously before switching to Cloudflare LB. Document every step with timestamps. Provide high-level progress updates to relevant team members at key milestones. Post Migration:\nMonitor Cloudflare load distribution across all 4 Tomcat web servers. Monitor system and network performance. Inform relevant team members of migration completion and maintain open communication channels for any issues. ","link":"https://systemsgolive.com/post/ans-to-cloudflare-lb-migration-zero-downtime/","section":"post","tags":["Cloudflare","Load Balancer","Migration","SSL","TLS","Certificate Management","Zero Downtime"],"title":"Migrating from ANS to Cloudflare Load Balancer: Zero-Downtime DNS Cutover with Pre-Configuration Strategy"},{"body":"","link":"https://systemsgolive.com/tags/migration/","section":"tags","tags":null,"title":"Migration"},{"body":"","link":"https://systemsgolive.com/categories/operations/","section":"categories","tags":null,"title":"Operations"},{"body":"","link":"https://systemsgolive.com/categories/site-reliability-engineering/","section":"categories","tags":null,"title":"Site Reliability Engineering"},{"body":"","link":"https://systemsgolive.com/tags/zero-downtime/","section":"tags","tags":null,"title":"Zero Downtime"},{"body":"","link":"https://systemsgolive.com/tags/apache/","section":"tags","tags":null,"title":"Apache"},{"body":"","link":"https://systemsgolive.com/tags/http/2/","section":"tags","tags":null,"title":"HTTP/2"},{"body":"TL;DR Problem: Enabling HTTP/2 on Apache caused HAProxy health checks to fail immediately — the load balancer only speaks HTTP/1.1, marking all servers as DOWN and triggering a full outage. Solution: Dual-port VirtualHost architecture — port 443 for HTTP/2 customer traffic, port 8443 dedicated to HTTP/1.1-only health checks. Method: Staged blue-green deployment — pilot on a drained server, parallel infrastructure rollout to all four Tomcat instances, seamless listener cutover, then HTTP/2 enablement as a final separate step. Result: HTTP/2 successfully enabled across all four production servers with zero downtime. Overview Our production care management web application runs on four servers, each with Apache HTTPD acting as a reverse proxy in front of a Tomcat application server running Java Spring — all sitting behind a vendor-hosted HAProxy application load balancer. As part of a performance improvement initiative, we set out to enable HTTP/2 on the Apache HTTPD layer — a straightforward protocol upgrade in isolation, but one that immediately exposed a compatibility constraint with the load balancer's health check mechanism. This article documents how we navigated that constraint without touching production traffic.\nNote: Domain names have been anonymised for this article. All references to login.companyabc.com are used as examples and do not reflect actual production domain names.\nThe Problem We enabled HTTP/2 on our Apache web servers for login.companyabc.com to improve performance. When we configured Apache with Protocols h2 http/1.1 on port 443, our load balancer's health checks immediately failed. All servers were marked as DOWN, causing a complete service outage.\nRoot Cause: HAProxy (our vendor-hosted application load balancer) health check process only supports HTTP/1.1. When it tried to communicate with HTTP/2-enabled ports, the protocol mismatch caused all health checks to fail with \u0026quot;invalid response\u0026quot; errors.\nOur pre-production environment (hosted on Vultr without a load balancer) worked perfectly with the same Apache HTTP/2 configuration. This confirmed the issue was specific to the load balancer health check mechanism.\nDeployment Journey Overview The following diagram illustrates our complete deployment journey from the initial problem through to the final HTTP/2-enabled state. Each phase shows the traffic flow from Cloudflare through the vendor Load Balancer to our four Tomcat backend servers. The diagram clearly shows customer traffic and health check paths, ports, and protocols at each stage.\nFigure 1: Complete HTTP/2 deployment progression showing seven phases: The Problem (protocol mismatch causing outage), Phase 1 (initial HTTP/1.1 state), Phase 2 (Tomcat-01 isolated in DRAIN mode), Phase 3 (dual-port configuration tested), Phase 4 (parallel infrastructure rolled out), Phase 5 (seamless listener cutover), and Phase 6 (HTTP/2 enabled with dedicated health check port). The diagram clearly distinguishes between customer traffic (port 443) and health check traffic (port 8443), showing how the solution separates these concerns to prevent protocol mismatch.\nThe Solution: Dual-Port Architecture In partnership with our vendor technical team, we designed a workaround that separates customer traffic from health check infrastructure. We created two dedicated VirtualHosts:\nPort 443: Customer traffic (eventually HTTP/2-enabled) Port 8443: Dedicated HTTP/1.1-only health checks with a lightweight /healthcheck endpoint This approach allows the load balancer to monitor server health using HTTP/1.1 while enabling HTTP/2 for actual user traffic.\nImplementation Strategy: Blue-Green Deployment We adopted a parallel infrastructure approach instead of a risky in-place upgrade:\nBuild new infrastructure alongside existing production Validate thoroughly while production continues unchanged Perform seamless cutover with zero downtime Enable HTTP/2 as a separate, final enhancement Phase 1: Pilot Testing on Drained Server Understanding \u0026quot;DRAIN Mode\u0026quot;:\nWhen a Tomcat server is set to DRAIN mode in the load balancer:\nNo new user sessions are created on that server Existing sessions continue until they expire (12-hour timeout in our configuration) The server becomes isolated from production traffic, making it safe for testing Health checks continue to monitor the server Why We Didn't Enable HTTP/2 Initially:\nWe kept HTTP/1.1 on both ports during pilot testing to:\nValidate the dual-VirtualHost infrastructure works correctly Prove the server can exist in both target groups simultaneously Eliminate variables during testing (infrastructure change only, no protocol change) Ensure easy rollback if issues arose This staged approach let us validate the health check solution before introducing HTTP/2 protocol negotiation.\nTest Configuration on Tomcat-01:\nWe created two dedicated VirtualHosts with identical application logic but different purposes:\n1# Port listening configuration 2Listen 443 # Customer traffic 3Listen 8443 # Health checks 4 5# HTTP to HTTPS redirect 6\u0026lt;VirtualHost *:80\u0026gt; 7 ServerName login.companyabc.com 8 Redirect / https://login.companyabc.com/ 9\u0026lt;/VirtualHost\u0026gt; 10 11# VirtualHost 1: Customer Traffic Port (HTTP/1.1 for testing) 12\u0026lt;VirtualHost _default_:443\u0026gt; 13 ServerName login.companyabc.com 14 Protocols http/1.1 # Deliberately kept as HTTP/1.1 for validation 15 16 Include common-conf.d/ssl-vhost.conf 17 Include common-conf.d/gzip.conf 18 19 # Security headers and SSL configuration 20 RequestHeader set X-Forwarded-Proto \u0026#34;https\u0026#34; 21 Header edit Location ^http:// https:// 22 Header always set X-Frame-Options SAMEORIGIN 23 Header always set X-Content-Type-Options nosniff 24 25 ProxyRequests off 26 ProxyPreserveHost on 27 DocumentRoot \u0026#34;/var/www/html\u0026#34; 28 29 # Application routing 30 RedirectMatch \u0026#34;^/(?!web/|admin/|custom_error_pages/).*$\u0026#34; /web/ 31 32 # Proxy to Tomcat application 33 ProxyPass /web/ http://127.0.0.1:8080/web/ timeout=600 34 ProxyPassReverse /web/ http://127.0.0.1:8080/web/ 35 ProxyPass /admin/ http://127.0.0.1:8080/admin/ 36 ProxyPassReverse /admin/ http://127.0.0.1:8080/admin/ 37\u0026lt;/VirtualHost\u0026gt; 38 39# VirtualHost 2: Dedicated Health Check Port (HTTP/1.1 only) 40\u0026lt;VirtualHost _default_:8443\u0026gt; 41 ServerName login.companyabc.com 42 Protocols http/1.1 # Must remain HTTP/1.1 for health checks 43 44 Include common-conf.d/ssl-vhost.conf 45 Include common-conf.d/gzip.conf 46 47 # Security headers 48 RequestHeader set X-Forwarded-Proto \u0026#34;https\u0026#34; 49 Header edit Location ^http:// https:// 50 Header always set X-Frame-Options SAMEORIGIN 51 Header always set X-Content-Type-Options nosniff 52 53 ProxyRequests off 54 ProxyPreserveHost on 55 DocumentRoot \u0026#34;/var/www/html\u0026#34; 56 57 # Lightweight health check endpoint - bypasses Tomcat 58 \u0026lt;Location \u0026#34;/healthcheck\u0026#34;\u0026gt; 59 ProxyPass ! # Don\u0026#39;t proxy to Tomcat 60 SetHandler none # Serve directly from Apache 61 Require all granted # Allow access 62 \u0026lt;/Location\u0026gt; 63 64 # Standard application proxying 65 ProxyPass /web/ http://127.0.0.1:8080/web/ timeout=600 66 ProxyPassReverse /web/ http://127.0.0.1:8080/web/ 67 ProxyPass /admin/ http://127.0.0.1:8080/admin/ 68 ProxyPassReverse /admin/ http://127.0.0.1:8080/admin/ 69\u0026lt;/VirtualHost\u0026gt; Health Check Endpoint Setup:\n1# Create health check directory and response page 2mkdir -p /var/www/html/healthcheck 3echo \u0026#34;OK\u0026#34; \u0026gt; /var/www/html/healthcheck/index.html Graceful Configuration Reload:\n1httpd -t # Validate syntax 2systemctl reload httpd # Graceful reload - no connection drops Our vendor technical team created a new target group tomcatshttp2 with health checks pointing to GET login.companyabc.com:8443/healthcheck/. We validated that Tomcat-01 passed health checks in both the old (tomcats) and new (tomcatshttp2) target groups simultaneously.\nPhase 2: Production Rollout After successful pilot testing, we rolled out the same configuration to production servers (Tomcat-02, 03, 04):\nRollout Process:\nApplied identical dual-VirtualHost configuration to each server Created /healthcheck directory and endpoint on each server Used systemctl reload httpd for graceful, zero-downtime updates Our vendor team added each server to tomcatshttp2 target group after completion Verified health check status for each server before proceeding to the next Infrastructure State After Rollout:\nTarget Group tomcats: All servers, health checks via port 443 /web/login, handling all customer traffic Target Group tomcatshttp2: All servers, health checks via port 8443 /healthcheck, monitoring but not serving traffic This parallel infrastructure allowed us to validate the new health check system while production traffic continued unaffected.\nPhase 3: The Seamless Cutover As shown in Phase 5 of the deployment diagram, the listener cutover was the critical moment where we switched from the old infrastructure to the new.\nLoad Balancer Configuration Change:\nIn collaboration with our vendor, we switched the listener's default target group from tomcats to tomcatshttp2. This was a single configuration change in the load balancer UI that took approximately 30 seconds.\nWhy Zero Downtime:\nSame servers in both target groups Same port 443 configuration (HTTP/1.1) Same application responses Existing connections continued normally New connections immediately routed to new target group Only operational change: health checks switched from port 443 to port 8443 Session Stickiness Note: The cutover reset session cookies (different between target groups), potentially logging out active users. We performed this change during a low-traffic period to minimize impact.\nMonitoring During Cutover:\n1# Watched Apache logs on all servers 2tail -f /var/log/httpd/access_log 3 4# Customer traffic continued on port 443: 510.0.0.7 - - [29/Sep/2025:11:15:33 +0100] \u0026#34;GET /web/login HTTP/1.1\u0026#34; 200 3507 6 7# Health checks now on dedicated port 8443: 810.0.0.7 - - [29/Sep/2025:11:15:35 +0100] \u0026#34;GET /healthcheck/ HTTP/1.1\u0026#34; 200 88 Phase 4: HTTP/2 Enablement With stable infrastructure and proven health checks, we enabled HTTP/2 on customer-facing port 443.\nFinal Production Configuration:\n1Listen 443 2Listen 8443 3 4# VirtualHost 1: Customer Traffic Port - HTTP/2 ENABLED 5\u0026lt;VirtualHost _default_:443\u0026gt; 6 ServerName login.companyabc.com 7 Protocols h2 http/1.1 # HTTP/2 enabled! Falls back to HTTP/1.1 for older clients 8 9 Include common-conf.d/ssl-vhost.conf 10 Include common-conf.d/gzip.conf 11 12 # Security headers 13 RequestHeader set X-Forwarded-Proto \u0026#34;https\u0026#34; 14 Header edit Location ^http:// https:// 15 Header always set X-Frame-Options SAMEORIGIN 16 Header always set X-Content-Type-Options nosniff 17 18 ProxyRequests off 19 ProxyPreserveHost on 20 DocumentRoot \u0026#34;/var/www/html\u0026#34; 21 22 RedirectMatch \u0026#34;^/(?!web/|admin/|custom_error_pages/).*$\u0026#34; /web/ 23 24 ProxyPass /web/ http://127.0.0.1:8080/web/ timeout=600 25 ProxyPassReverse /web/ http://127.0.0.1:8080/web/ 26 ProxyPass /admin/ http://127.0.0.1:8080/admin/ 27 ProxyPassReverse /admin/ http://127.0.0.1:8080/admin/ 28\u0026lt;/VirtualHost\u0026gt; 29 30# VirtualHost 2: Health Check Port - REMAINS HTTP/1.1 31\u0026lt;VirtualHost _default_:8443\u0026gt; 32 ServerName login.companyabc.com 33 Protocols http/1.1 # Must stay HTTP/1.1 for health checks 34 35 Include common-conf.d/ssl-vhost.conf 36 Include common-conf.d/gzip.conf 37 38 RequestHeader set X-Forwarded-Proto \u0026#34;https\u0026#34; 39 Header edit Location ^http:// https:// 40 Header always set X-Frame-Options SAMEORIGIN 41 Header always set X-Content-Type-Options nosniff 42 43 ProxyRequests off 44 ProxyPreserveHost on 45 DocumentRoot \u0026#34;/var/www/html\u0026#34; 46 47 # Lightweight health check endpoint 48 \u0026lt;Location \u0026#34;/healthcheck\u0026#34;\u0026gt; 49 ProxyPass ! 50 SetHandler none 51 Require all granted 52 \u0026lt;/Location\u0026gt; 53 54 ProxyPass /web/ http://127.0.0.1:8080/web/ timeout=600 55 ProxyPassReverse /web/ http://127.0.0.1:8080/web/ 56 ProxyPass /admin/ http://127.0.0.1:8080/admin/ 57 ProxyPassReverse /admin/ http://127.0.0.1:8080/admin/ 58\u0026lt;/VirtualHost\u0026gt; We enabled HTTP/2 on port 443 one server at a time using graceful Apache reloads. We validated each change before proceeding to the next server.\nVerification:\n1# Apache access logs showing HTTP/2 for customer traffic: 210.0.0.7 - - [29/Sep/2025:11:33:41 +0100] \u0026#34;GET /web/login HTTP/2.0\u0026#34; 200 3507 3 4# Health checks still using HTTP/1.1 on dedicated port: 510.0.0.7 - - [29/Sep/2025:11:33:42 +0100] \u0026#34;GET /healthcheck/ HTTP/1.1\u0026#34; 200 88 Results and Key Takeaways Achievements:\nZero downtime throughout entire implementation HTTP/2 successfully enabled for all customer traffic Improved health check reliability with dedicated, lightweight endpoint Clear separation of concerns (customer traffic vs. operational monitoring) Success Factors:\nPilot testing on drained server reduced risk before production rollout Parallel infrastructure allowed thorough validation without affecting production Staged approach separated infrastructure changes from protocol changes Graceful Apache reloads eliminated service interruptions Dedicated VirtualHosts provided clear separation between customer traffic and health checks Post-Implementation:\nKept legacy tomcats target group for one week as rollback safety net Restricted DMZ network policy to allow only TCP/8443 for health checks (security best practice) Load balancer OS upgrade postponed until HTTP/2 implementation proven stable Conclusion This project demonstrates that complex infrastructure upgrades can be executed without service disruption when approached methodically. By separating port 8443 for HTTP/1.1 health checks from port 443 for HTTP/2 customer traffic, we resolved the protocol mismatch between HAProxy's limitations and modern web protocols.\nThe blue-green deployment strategy proved essential: pilot testing validated our approach, parallel infrastructure enabled risk-free testing, and staged rollout minimized complexity. The result was seamless transition achieving performance goals while maintaining operational reliability.\nKey takeaway: Separating infrastructure changes from feature enhancements reduces risk and creates more maintainable architecture. This dual-port approach not only solved our immediate problem but established a robust foundation for future improvements.\n","link":"https://systemsgolive.com/post/http2-apache-haproxy-health-check-dual-port/","section":"post","tags":["Infrastructure","HTTP/2","Apache"],"title":"HTTP/2 on Apache Behind HAProxy: Solving Health Check Failures with Dual-Port Architecture"},{"body":"","link":"https://systemsgolive.com/categories/linux/","section":"categories","tags":null,"title":"Linux"},{"body":"","link":"https://systemsgolive.com/tags/ed25519/","section":"tags","tags":null,"title":"Ed25519"},{"body":"TL;DR Goal: Set up verified TLS 1.3 communication between a client and an Nginx server using Ed25519 certificates signed by a self-hosted private CA. Why Ed25519: Faster than RSA, smaller keys and signatures, stronger security (RSA-3072 equivalent), and resistant to side-channel attacks. Method: Three-node setup — CA node generates the root certificate and signs CSRs, server node hosts Nginx with TLS 1.3, client node trusts the CA and validates the connection. Result: Fully encrypted traffic confirmed via tcpdump — sensitive data unreadable over HTTPS, fully exposed over plain HTTP. Overview This guide demonstrates how to set up secure TLS 1.3 communication using Ed25519 elliptic curve certificates and a private Certificate Authority (CA). It covers encrypted client-server communication with modern, efficient cryptographic standards — ideal for internal systems, microservices, and zero-trust network architectures.\nWhat Is Ed25519 and How Does It Compare to RSA? Ed25519 is a modern elliptic-curve signature algorithm that offers fast operations, smaller keys, and strong security while reducing operational overhead in your infrastructure.\nBelow is a summary of the key aspects in a comparison table:\nAspect Ed25519 RSA (2048) Key Size 256 bits 2048 bits Signature Size 64 bytes ~256 bytes Security Level ~128-bit (RSA-3072 equivalent) ~112-bit Key Generation 10–20x faster Slower Signing Speed 3–5x faster Slower Verification Speed 2–3x faster Slower Bandwidth Lower (smaller certs \u0026amp; signatures) Higher Implementation Simple, deterministic, no padding Complex, padding required Side-Channel Resistance Resistant Susceptible if not carefully implemented Compatibility Limited on legacy systems Broad support Quantum Resistance Not quantum-safe Not quantum-safe Pros and Cons of Using Ed25519 Pros Cons Fast key generation, signing, and verification Limited support on legacy systems Smaller certs \u0026amp; signatures reduce bandwidth Not quantum-resistant, future migration needed Strong security (RSA-3072 equivalent) Resistant to side-channel attacks Simple, deterministic implementation Lower CPU and memory usage Environment setup I built this environment locally on Fedora 42 using libvirt (Linux virtualisation API) and Vagrant (VM automation tool) to provision 3 x AlmaLinux OS 9 nodes.\nVirtualisation Platform: libvirt. Orchestration VM tool: Vagrant. VM OS: 3 x AlmaLinux 9. Network: Private isolated network (192.168.56.x range). Note: This setup is flexible and can be replicated using any virtualisation platform in a non-production environment — including VMware, VirtualBox, WSL, cloud-based VMs. The only requirement is having three systems that can communicate over a network.\nnode-03 — Certificate Authority (IP address: 192.168.56.103)\nGenerates Ed25519 root certificate and signs CSRs. node-02 — Server (IP address: 192.168.56.102)\nGenerates Ed25519 private key and CSR. Receives signed certificate from CA. Hosts HTTPS service using NGINX with TLS 1.3. node-01 — Client (192.168.56.101)\nTrusts CA certificate. Initiates TLS connection to the server and validates the ed25519 certificate. Verifies encrypted communication. Architecture Flow Diagram The diagram below illustrates the complete TLS 1.3 setup flow using Ed25519 and a private Certificate Authority (CA), showing each step across the client, server, and CA nodes.\nFor each step shown in the diagram, the corresponding implementation commands are provided in the sections below.\nDetailed Implementation The following implementation steps assume you are operating as the root user in /root directory.\nStep 1–2: Private Root Certificate Authority Setup (almalinux9-node-03) Create the private Root Certificate Authority (CA) that will issue certificated for TLS.\n1# Switch to root shell with root environment 2sudo -i 3 4# Install OpenSSL 5dnf install -y openssl 6 7# Create CA directory structure 8mkdir -p ~/ca/{certs,crl,newcerts,private} 9chmod 700 ~/ca/private 10cd ~/ca 11 12# Create CA configuration file 13cat \u0026gt; openssl.cnf \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; 14[ ca ] 15default_ca = CA_default 16 17[ CA_default ] 18dir = /root/ca 19certs = $dir/certs 20crl_dir = $dir/crl 21new_certs_dir = $dir/newcerts 22database = $dir/index.txt 23serial = $dir/serial 24RANDFILE = $dir/private/.rand 25 26private_key = $dir/private/ca.key.pem 27certificate = $dir/certs/ca.cert.pem 28 29default_md = sha256 30name_opt = ca_default 31cert_opt = ca_default 32default_days = 365 33preserve = no 34policy = policy_strict 35 36[ policy_strict ] 37countryName = match 38stateOrProvinceName = match 39organizationName = match 40organizationalUnitName = optional 41commonName = supplied 42emailAddress = optional 43 44[ req ] 45default_bits = 2048 46distinguished_name = req_distinguished_name 47string_mask = utf8only 48default_md = sha256 49x509_extensions = v3_ca 50 51[ req_distinguished_name ] 52countryName = Country Name (2 letter code) 53stateOrProvinceName = State or Province Name 54localityName = Locality Name 550.organizationName = Organization Name 56organizationalUnitName = Organizational Unit Name 57commonName = Common Name 58emailAddress = Email Address 59 60[ v3_ca ] 61subjectKeyIdentifier = hash 62authorityKeyIdentifier = keyid:always,issuer 63basicConstraints = critical, CA:true 64keyUsage = critical, digitalSignature, cRLSign, keyCertSign 65 66[ server_cert ] 67basicConstraints = CA:FALSE 68subjectKeyIdentifier = hash 69authorityKeyIdentifier = keyid,issuer:always 70keyUsage = critical, digitalSignature, keyEncipherment 71extendedKeyUsage = serverAuth 72EOF 73 74# Initialize CA database 75touch index.txt 76echo 1000 \u0026gt; serial 77echo 1000 \u0026gt; crlnumber 78 79# Generate Ed25519 CA private key 80openssl genpkey -algorithm Ed25519 -out private/ca.key.pem 81chmod 400 private/ca.key.pem 82 83# Create self-signed CA certificate 84openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem -subj \u0026#34;/C=GB/ST=England/L=Bristol/O=MyOrg/CN=MyCA\u0026#34; 85 86chmod 444 certs/ca.cert.pem Step 3–4: Server Key \u0026amp; CSR (almalinux9-node-02) Generate the server's private key and certificate signing request (CSR).\n1# Switch to root shell with root environment 2sudo -i 3 4# Install OpenSSL and NGINX 5dnf install -y openssl nginx 6 7# Generate Ed25519 server private key 8openssl genpkey -algorithm Ed25519 -out server.key.pem 9 10# Create certificate signing request 11openssl req -new -key server.key.pem -out server.csr -subj \u0026#34;/C=GB/ST=England/L=Bristol/O=MyOrg/CN=almalinux9-node-02\u0026#34; Step 5: CSR File Transfer (node-02 → node-03) Transfer the server's CSR to the CA for signining\n1# On almalinux9-node-02 (server) 2# Start temporary HTTP server to share the CSR 3python3 -m http.server 8000 Step 6–7: Certificate Signing (almalinux9-node-03) Download and sign the CSR with the CA private key.\n1# Switch to root shell with root environment 2sudo -i 3 4# On almalinux9-node-03 (CA) 5cd ~/ca 6 7# Download CSR from node-02 8wget http://192.168.56.102:8000/server.csr # 192.168.56.102 is node-02 (server) IP 9 10# Sign the server certificate 11openssl ca -config openssl.cnf -extensions server_cert -days 365 -notext -md sha256 -in server.csr -out server.cert.pem 12 13# Start HTTP server to distribute signed certificates to other nodes 14python3 -m http.server 8000 Step 8–9: Certificate Distribution (node-03 → node-02) Download the signed certificate and CA certificate to the server.\n1# On almalinux9-node-02 2 3# Download signed server certificate from CA 4wget http://192.168.56.103:8000/server.cert.pem # 192.168.56.103 is node-03 (CA) IP 5 6# Download CA certificate 7wget http://192.168.56.103:8000/certs/ca.cert.pem # 192.168.56.103 is node-03 (CA) IP Step 10: Certificate Verification (almalinux9-node-02) Verify the signed certificate is valid and properly configured before using it with NGINX.\n1# 1. Verify certificate was signed by your CA 2openssl verify -CAfile ca.cert.pem server.cert.pem 3 4# Output expected 5server.cert.pem: OK 1# 2. Check certificate details 2openssl x509 -in server.cert.pem -noout -subject -issuer -dates -fingerprint 3 4# Output expected: 5subject=C=GB, ST=England, O=MyOrg, CN=almalinux9-node-02 6issuer=C=GB, ST=England, L=Bristol, O=MyOrg, CN=MyCA 7notBefore=Jul 14 20:00:35 2025 GMT 8notAfter=Jul 14 20:00:35 2026 GMT 9SHA1 Fingerprint=40:B3:BD:5D:3E:76:9C:FE:AE:08:92:5A:58:0B:53:35:CB:62:05:8E 1# 3. Confirm Ed25519 is being used 2openssl x509 -in server.cert.pem -noout -text | grep \u0026#34;Public Key Algorithm\u0026#34; 3 4# Output expected 5Public Key Algorithm: ED25519 1# 4. View complete certificate information (optional) 2openssl x509 -in server.cert.pem -text -noout 3 4# Output expected: 5Certificate: 6 Data: 7 Version: 3 (0x2) 8 Serial Number: 4096 (0x1000) 9 Signature Algorithm: ED25519 10 Issuer: C=GB, ST=England, L=Bristol, O=MyOrg, CN=MyCA 11 Validity 12 Not Before: Jul 14 20:00:35 2025 GMT 13 Not After : Jul 14 20:00:35 2026 GMT 14 Subject: C=GB, ST=England, O=MyOrg, CN=almalinux9-node-02 15 Subject Public Key Info: 16 Public Key Algorithm: ED25519 17 ED25519 Public-Key: 18 pub: 19 9c:03:54:87:8c:4b:b2:39:ca:4c:79:b7:39:6e:40: 20 eb:aa:3b:fb:4e:a7:7b:c1:7d:f9:3b:99:29:89:4b: 21 e6:68 22 X509v3 extensions: 23 X509v3 Basic Constraints: 24 CA:FALSE 25 X509v3 Subject Key Identifier: 26 48:6C:A7:A9:8B:72:29:1E:90:75:3B:6C:FE:4C:DF:75:1F:0F:59:5D 27 X509v3 Authority Key Identifier: 28 keyid:C4:72:73:3D:9D:CA:4A:CB:EF:77:D6:3F:D7:58:C9:B5:4A:A5:8C:DE 29 DirName:/C=GB/ST=England/L=Bristol/O=MyOrg/CN=MyCA 30 serial:43:D8:98:80:38:39:6E:1A:92:1E:31:CF:F8:1B:5B:E0:D0:BA:43:E7 31 X509v3 Key Usage: critical 32 Digital Signature, Key Encipherment 33 X509v3 Extended Key Usage: 34 TLS Web Server Authentication 35 Signature Algorithm: ED25519 36 Signature Value: 37 fb:4d:f9:d3:68:bb:ab:34:d0:c1:6c:e8:9f:fe:4a:57:57:64: 38 a4:2c:90:60:04:0b:c5:10:e8:2c:a6:99:eb:4a:25:c8:ba:39: 39 54:7c:60:bd:e4:0f:d3:f1:e6:99:c4:06:9b:8e:01:bf:f1:05: 40 14:62:0f:3d:cb:3b:6f:35:90:08 Certificate Validation Checklist The outputs above confirm that our Ed25519 certificate setup is fully operational and secure:\nCertificate Verification: server.cert.pem: OK - Valid CA signature Identity Match: Subject CN=almalinux9-node-02 matches server hostname Trusted Issuer: Certificate signed by our private CA CN=MyCA Ed25519 Confirmed: Modern elliptic curve cryptography active TLS Ready: Proper extensions for HTTPS server authentication Result: Server Certificate successfully validated - ready for NGINX TLS 1.3 deployment.\nStep 11–12: NGINX Setup (almalinux9-node-02) Configure NGINX with TLS 1.3 with the Ed25519 certificate.\n1# Install certificates 2mkdir -p /etc/nginx/ssl 3cp server.cert.pem /etc/nginx/ssl/ 4cp server.key.pem /etc/nginx/ssl/ 5cp ca.cert.pem /etc/nginx/ssl/ 6chmod 600 /etc/nginx/ssl/server.key.pem 7 8# Configure NGINX for TLS 1.3 9tee /etc/nginx/conf.d/https-server.conf \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; 10# HTTP Server (unencrypted) - for testing 11server { 12 listen 80; 13 server_name almalinux9-node-02; 14 15 location / { 16 return 200 \u0026#34;UNENCRYPTED HTTP Server - this data is visible!\\n\u0026#34;; 17 add_header Content-Type text/plain; 18 } 19 20 location /secret { 21 return 200 \u0026#34;SECRET DATA: password123 - credit card: 1234-5678-9012-3456\\n\u0026#34;; 22 add_header Content-Type text/plain; 23 } 24} 25 26# HTTPS Server (encrypted) - TLS 1.3 27server { 28 listen 443 ssl http2; 29 server_name almalinux9-node-02; 30 31 ssl_certificate /etc/nginx/ssl/server.cert.pem; 32 ssl_certificate_key /etc/nginx/ssl/server.key.pem; 33 34 ssl_protocols TLSv1.3; 35 ssl_prefer_server_ciphers off; 36 37 location / { 38 return 200 \u0026#34;HTTPS Server - TLS 1.3 + Ed25519 Encrypted!\\n\u0026#34;; 39 add_header Content-Type text/plain; 40 } 41 42 location /secret { 43 return 200 \u0026#34;ENCRYPTED SECRET DATA: admin_password=SuperSecret789 - credit_card=9876-5432-1098-7654 - ssn=123-45-6789 - bank_account=987654321\\n\u0026#34;; 44 add_header Content-Type text/plain; 45 } 46} 47EOF 48 49# Start NGINX 50systemctl start nginx 51systemctl enable nginx 52 53# Check NGINX status 54systemctl status nginz Step 13: Client Setup (almalinux9-node-01) Configure the client to trust the CA and establish secure TLS.\n1# Install client tools 2dnf install -y openssl curl 3 4# Create certificate directory 5mkdir -p ~/certs 6 7# Download CA certificate 8wget http://192.168.56.103:8000/certs/ca.cert.pem -O ~/certs/ca.cert.pem Steps 14–18: Secure Communication Phase Test secure HTTPS access, inspect handshake, and verify traffic encryption.\n1# 13. Test basic HTTPS connection first (establish that TLS works) 2curl --cacert ~/certs/ca.cert.pem https://almalinux9-node-02/secret 3 4# 14. Verify TLS handshake details 5curl -v --cacert ~/certs/ca.cert.pem https://almalinux9-node-02/ 6 7# 15. Encrypted Packet Capture (HTTPS) 8echo \u0026#34;=== Capturing HTTPS Traffic (Encrypted) ===\u0026#34; 9tcpdump -i any -A -c 10 host almalinux9-node-02 and port 443 \u0026amp; 10sleep 2 # Give tcpdump time to start 11curl --cacert ~/certs/ca.cert.pem https://almalinux9-node-02/secret 12 13# 16. Plaintext Packet Capture (HTTP) 14echo \u0026#34;=== Capturing HTTP Traffic (Unencrypted) ===\u0026#34; 15tcpdump -i any -A -c 10 host almalinux9-node-02 and port 80 \u0026amp; 16sleep 2 # Give tcpdump time to start 17curl http://almalinux9-node-02/secret Expected Results and Evidence Test 1: Basic HTTPS Connection 1curl --cacert ~/certs/ca.cert.pem https://almalinux9-node-02/secret Expected Output: 1ENCRYPTED SECRET DATA: admin_password=SuperSecret789 - credit_card=9876-5432-1098-7654 - ssn=123-45-6789 - bank_account=987654321 Client successfully receives decrypted data over secure TLS connection.\nTest 2: TLS Handshake Verification 1curl -v --cacert ~/certs/ca.cert.pem https://almalinux9-node-02/ Key Output Indicators: 1* TLSv1.3 (OUT), TLS handshake, Client hello (1): 2* TLSv1.3 (IN), TLS handshake, Server hello (2): 3* TLSv1.3 (IN), TLS handshake, Certificate (11): 4* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 5* Server certificate: 6* subject: C=GB; ST=England; O=MyOrg; CN=almalinux9-node-02 7* issuer: C=GB; ST=England; L=Bristol; O=MyOrg; CN=MyCA 8* common name: almalinux9-node-02 (matched) 9* SSL certificate verify ok. TLS 1.3 active, Ed25519 certificate verified, hostname matched.\nTest 3: HTTPS Network Traffic Analysis Network Capture (tcpdump): 1E..\u0026lt;..@.@.....8e..8f.h..Lv.o.........J......... 2..q......... 322:38:52.489995 eth1 In IP almalinux9-node-02.https \u0026gt; almalinux9-node-01.34408: Flags [S.], seq 133835979, ack 1282837360, win 65160, options [mss 1460,sackOK,TS val 1778032372 ecr 2642506154,nop,wscale 7], length 0 4E..\u0026lt;..@.@.H...8f..8e...h..,.Lv.p.....J......... 5i.....q..... 622:38:52.490029 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 2642506155 ecr 1778032372], length 0 7E..4..@.@.....8e..8f.h..Lv.p..,......B..... 8..q.i... 922:38:52.493578 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 2642506159 ecr 1778032372], length 517 10E..9..@.@.....8e..8f.h..Lv.p..,......G..... 11..q.i..............X-.:7...akx...PI[\u0026#34;.w..0..\u0026amp;.C..#. ......p....A.c.o....Y...*........H.........,.0.......+./...#.\u0026#39;. 12...\t...........=.\u0026lt;.5./...........k.g.9.3.....k.........almalinux9-node-02......... 13........................3t.........h2.http/1.1.........1.....\u0026#34;. ...........\t. 14...................+........-.....3.\u0026amp;.$... o..\u0026#34;!......;.......T..H.|.....d6............................................................................................................................................................................. 1522:38:52.494024 eth1 In IP almalinux9-node-02.https \u0026gt; almalinux9-node-01.34408: Flags [.], ack 518, win 506, options [nop,nop,TS val 1778032376 ecr 2642506159], length 0 16E..4.|@.@..+..8f..8e...h..,.Lv.u.....B..... 17i.....q. 1822:38:52.494745 eth1 In IP almalinux9-node-02.https \u0026gt; almalinux9-node-01.34408: Flags [P.], seq 1:1015, ack 518, win 506, options [nop,nop,TS val 1778032377 ecr 2642506159], length 1014 19E..*.}@.@..4..8f..8e...h..,.Lv.u.....8..... 20i.....q.....z...v..../.{.z...\u0026amp;..t}....g.p.PJ.i.).t. ......p....A.c.o....Y...*.............+.....3.$... N${......s....k....O...A.*...i,g..........$a7.......4\\u....h.Z..=...WrY/ar...v..........-H6.!tc.0.../.8.ca^.f.|...r....f|..Rm.....sE\u0026amp;n.4Yz3.:..y...1.[x|.....L\u0026#39;....Y..=...0....q.YiM.-b.3..z.....}..E+,.{..........Hdj.u.J.B...4M3bj.h........eisf..rS.R.G.K..7.!.K.VD._.otr.Y}..%..m#+3..).E......5U...M.r.9..3so....X 21..S\t..\u0026#39;.X,q.........*_.\u0026amp;................Et..D...V........}.eK...~.Gj..I_..0ue.e..y.....B.V^S....}b6%.g3.P..s...e..w......,$.e.$.\u0026gt;oU....b..l.!;p.=.+...5....9bW2H1.$..........|...J;.)\\.v!h`.C.sY..... 22WCGh.Xj..].....`...e....W.....(:E\u0026#39;......;.3.....@..B....K-....y\u0026amp;3.B.-.H...d.w6Z.YX[4..m....Xx.a.Xbx..VLQ1?Uw...._.....X....k^.. 23..+.......1;Q..6.]N.1..1...SG..Zz...vM.2.P..@...}..u..,.....E..;.bQ...Q.............G.-..v...$...\u0026amp;.\u0026#39;.m.].n.Eb.YA1....Ye..ij....D..w.\tez$.....K........3.x....bG...I.c_...*..$...U5..-.Sz!.36.......\u0026#34;.......g!......E...l...........-..X\u0026lt;z.W...kgw..............3....g.....C.|O.K+.......A 2422:38:52.494777 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [.], ack 1015, win 524, options [nop,nop,TS val 2642506160 ecr 1778032377], length 0 25E..4..@.@.....8e..8f.h..Lv.u..0......B..... 26..q.i... 2722:38:52.496125 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [P.], seq 518:598, ack 1015, win 524, options [nop,nop,TS val 2642506161 ecr 1778032377], length 80 28E.....@.@.....8e..8f.h..Lv.u..0............ 29..q.i.............E.KR...\u0026gt;xL...R....K..C.#.......).)..x....?rS.a..b..{.S....T*.t....{..\u0026lt; 3022:38:52.496230 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [P.], seq 598:644, ack 1015, win 524, options [nop,nop,TS val 2642506161 ecr 1778032377], length 46 31E..b..@.@.....8e..8f.h..Lv....0......p..... 32..q.i.......)9.C.P...Rb.~.....l.\t=....np.\u0026lt;.p\u0026#39;.J!|.... 3322:38:52.496242 eth1 Out IP almalinux9-node-01.34408 \u0026gt; almalinux9-node-02.https: Flags [P.], seq 644:693, ack 1015, win 524, options [nop,nop,TS val 2642506161 ecr 1778032377], length 49 34E..e..@.@.....8e..8f.h..Lv....0......s..... 35..q.i.......,.,......;.hT.j..bPu.E.|@.....\u0026lt;.=gU2?=....z.\t3610 packets captured 3727 packets received by filter 380 packets dropped by kernel Evidence: Network traffic shows encrypted binary data - sensitive information is completely protected.\nTest 4: HTTP Traffic Analysis (Comparison) Network Capture (tcpdump): 1GET /secret HTTP/1.1 2Host: almalinux9-node-02 3User-Agent: curl/7.76.1 4 5HTTP/1.1 200 OK 6Server: nginx/1.20.1 7Content-Type: text/plain 8SECRET DATA: password123 - credit card: 1234-5678-9012-3456 Evidence: Network traffic exposes all sensitive data in plain text - completely insecure.\nSecurity Comparison Results Protocol (Client) Experience Network Security Attacker Visibility HTTPS (TLS 1.3 + Ed25519) Normal data access Fully encrypted Encrypted gibberish only HTTP (Unencrypted) Normal data access No protection All secrets visible Conclusion This implementation demonstrates secure TLS 1.3 communication using Ed25519 elliptic curve cryptography and a private Certificate Authority (CA).\nThe key outcomes are summarised below:\nEd25519: Strong cryptographic security with fast key generation and smaller signatures. TLS 1.3: Enforces modern encryption with forward secrecy and reduced handshake overhead. Private CA: Enables full control over certificate issuance and trust boundaries. Verified Encryption: Network traffic analysis confirms all sensitive data remains encrypted in transit. Compared to RSA-based setups, Ed25519 simplifies key handling while improving performance and side-channel resistance—making it well-suited for secure internal systems and service-to-service communication.\n","link":"https://systemsgolive.com/post/tls-ed25519-private-ca-nginx-setup/","section":"post","tags":["TLS","SSL","Encryption","Certificate","Ed25519"],"title":"TLS 1.3 with Ed25519 and a Private CA: End-to-End Encrypted Client-Server Communication"},{"body":"","link":"https://systemsgolive.com/tags/best-practices/","section":"tags","tags":null,"title":"Best Practices"},{"body":"","link":"https://systemsgolive.com/categories/database-admininistration/","section":"categories","tags":null,"title":"Database Admininistration"},{"body":"","link":"https://systemsgolive.com/tags/mysql/","section":"tags","tags":null,"title":"MySQL"},{"body":"Overview This guide explains how to securely create and manage MySQL user accounts using standard access profiles. It covers required configuration, user setup, privilege assignment, and best practices to ensure consistent and controlled access across environments.\nPrerequisites System Requirements MySQL 8.0.41 or later partial_revokes = ON must be enabled in MySQL configuration Check Configuration (One-Time Check Per Server) 1-- Connect using a MySQL admin or root account 2mysql -u root -p # or mickael_admin or mysql --login-path=client 3 4-- Check both settings (should return values, not empty) 5SHOW VARIABLES LIKE \u0026#39;partial_revokes\u0026#39;; -- Should be: ON 6SHOW VARIABLES LIKE \u0026#39;validate_password.%\u0026#39;; -- Should return 8 rows Fix Missing Configuration (If Needed) 1-- If partial_revokes = OFF: 2SET GLOBAL partial_revokes = ON; 3 4-- If password validation empty: 5INSTALL COMPONENT \u0026#39;file://component_validate_password\u0026#39;; Make partial_revokes Permanent Add to /etc/mysql/my.cnf:\n1[mysqld] 2# Added on [DATE] 3# Enable partial revokes for user profile management (basic, standard, maintenance, admin) 4# Matches Pre-Prod config - allows REVOKE on specific databases after global GRANT 5partial_revokes=ON Then restart MySQL: sudo systemctl restart mysql\nUser Profiles Naming Convention: firstname_profile or firstname_surname_profile Profiles: _basic, _stand, _maint, _admin\nNote: Replace username in the commands below with actual names following the naming convention (e.g., mickael_basic, mickael_stand, mickael_maint, mickael_admin)\nBasic Profile (_basic) - Read Only 1-- Create user with password security settings (5 failed attempts = 1 day lockout) 2CREATE USER \u0026#39;username_basic\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;secure_password\u0026#39; FAILED_LOGIN_ATTEMPTS 5 PASSWORD_LOCK_TIME 1; 3 4-- Grant read-only access to all databases 5GRANT SELECT ON *.* TO \u0026#39;username_basic\u0026#39;@\u0026#39;localhost\u0026#39;; 6 7-- Remove access to MySQL system database 8REVOKE SELECT ON mysql.* FROM \u0026#39;username_basic\u0026#39;@\u0026#39;localhost\u0026#39;; 9 10-- Apply all privilege changes 11FLUSH PRIVILEGES; Standard Profile (_stand) - Data Access 1-- Create user with password security settings (5 failed attempts = 1 day lockout) 2CREATE USER \u0026#39;username_stand\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;secure_password\u0026#39; FAILED_LOGIN_ATTEMPTS 5 PASSWORD_LOCK_TIME 1; 3 4-- Grant data manipulation access to all databases 5GRANT SELECT, UPDATE, INSERT, DELETE ON *.* TO \u0026#39;username_stand\u0026#39;@\u0026#39;localhost\u0026#39;; 6 7-- Remove access to MySQL system database 8REVOKE SELECT, UPDATE, INSERT, DELETE ON mysql.* FROM \u0026#39;username_stand\u0026#39;@\u0026#39;localhost\u0026#39;; 9 10-- Apply all privilege changes 11FLUSH PRIVILEGES; Maintenance Profile (_maint) - Database Admin 1-- Create user with password security settings (5 failed attempts = 1 day lockout) 2CREATE USER \u0026#39;username_maint\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;secure_password\u0026#39; FAILED_LOGIN_ATTEMPTS 5 PASSWORD_LOCK_TIME 1; 3 4-- Grant database administration privileges with ability to grant privileges to others 5GRANT SELECT, UPDATE, INSERT, DELETE, CREATE, ALTER, DROP, REFERENCES, LOCK TABLES, CREATE TABLESPACE, CREATE TEMPORARY TABLES, CREATE VIEW, EVENT, EXECUTE, INDEX, SHOW VIEW, TRIGGER ON *.* TO \u0026#39;username_maint\u0026#39;@\u0026#39;localhost\u0026#39; WITH GRANT OPTION; 6 7-- Remove all administrative access to MySQL system database 8REVOKE SELECT, UPDATE, INSERT, DELETE, CREATE, ALTER, DROP, REFERENCES, LOCK TABLES, CREATE TABLESPACE, CREATE TEMPORARY TABLES, CREATE VIEW, EVENT, EXECUTE, INDEX, SHOW VIEW, TRIGGER, GRANT OPTION ON mysql.* FROM \u0026#39;username_maint\u0026#39;@\u0026#39;localhost\u0026#39;; 9 10-- Apply all privilege changes 11FLUSH PRIVILEGES; Admin Profile (_admin) - Full Server Access 1-- Create user with password security settings (5 failed attempts = 1 day lockout) 2CREATE USER \u0026#39;username_admin\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;secure_password\u0026#39; FAILED_LOGIN_ATTEMPTS 5 PASSWORD_LOCK_TIME 1; 3 4-- Grant all traditional database and server administration privileges 5GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO \u0026#39;username_admin\u0026#39;@\u0026#39;localhost\u0026#39; WITH GRANT OPTION; 6 7-- Grant all dynamic privileges and administrative roles 8GRANT APPLICATION_PASSWORD_ADMIN, AUDIT_ABORT_EXEMPT, AUDIT_ADMIN, AUTHENTICATION_POLICY_ADMIN, BACKUP_ADMIN, BINLOG_ADMIN, BINLOG_ENCRYPTION_ADMIN, CLONE_ADMIN, CONNECTION_ADMIN, ENCRYPTION_KEY_ADMIN, FIREWALL_EXEMPT, FLUSH_OPTIMIZER_COSTS, FLUSH_STATUS, FLUSH_TABLES, FLUSH_USER_RESOURCES, GROUP_REPLICATION_ADMIN, GROUP_REPLICATION_STREAM, INNODB_REDO_LOG_ARCHIVE, INNODB_REDO_LOG_ENABLE, PASSWORDLESS_USER_ADMIN, PERSIST_RO_VARIABLES_ADMIN, REPLICATION_APPLIER, REPLICATION_SLAVE_ADMIN, RESOURCE_GROUP_ADMIN, RESOURCE_GROUP_USER, ROLE_ADMIN, SENSITIVE_VARIABLES_OBSERVER, SERVICE_CONNECTION_ADMIN, SESSION_VARIABLES_ADMIN, SET_USER_ID, SHOW_ROUTINE, SYSTEM_USER, SYSTEM_VARIABLES_ADMIN, TABLE_ENCRYPTION_ADMIN, TELEMETRY_LOG_ADMIN, XA_RECOVER_ADMIN ON *.* TO \u0026#39;username_admin\u0026#39;@\u0026#39;localhost\u0026#39; WITH GRANT OPTION; 9 10FLUSH PRIVILEGES; User Management View Users 1-- All users 2SELECT User, Host FROM mysql.user ORDER BY User; 3 4-- By profile 5SELECT User FROM mysql.user WHERE User LIKE \u0026#39;%_basic\u0026#39;; -- Basic 6SELECT User FROM mysql.user WHERE User LIKE \u0026#39;%_stand\u0026#39;; -- Standard 7SELECT User FROM mysql.user WHERE User LIKE \u0026#39;%_maint\u0026#39;; -- Maintenance 8SELECT User FROM mysql.user WHERE User LIKE \u0026#39;%_admin\u0026#39;; -- Admin View Privileges 1SHOW GRANTS FOR \u0026#39;username_profile\u0026#39;@\u0026#39;localhost\u0026#39;; Modify User 1-- Change password 2ALTER USER \u0026#39;username_profile\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;new_password\u0026#39;; 3 4-- Delete user 5DROP USER \u0026#39;username_profile\u0026#39;@\u0026#39;localhost\u0026#39;; 6 7-- Always flush after changes 8FLUSH PRIVILEGES; Test User Access 1-- Exit admin session 2EXIT; 3 4-- Login as new user 5mysql -u username_profile -p 6 7-- Test access 8SHOW DATABASES; Troubleshooting \u0026quot;There is no such grant defined\u0026quot; Error Problem: REVOKE fails after global GRANT\nSolution: Enable partial_revokes = ON and restart MySQL\nPassword Validation Errors Problem: \u0026quot;Password does not satisfy policy requirements\u0026quot;\nRequirements: 8+ chars, uppercase, lowercase, number, special character\nExample Valid Password: SecurePass123!\nCheck Configuration Issues 1-- Verify settings 2SHOW VARIABLES LIKE \u0026#39;partial_revokes\u0026#39;; 3SHOW VARIABLES LIKE \u0026#39;validate_password.policy\u0026#39;; 4SELECT VERSION();-- Should be 8.0.16+ Security Guidelines Password Requirements:\n8+ characters with mixed case, numbers, special chars Cannot contain username Auto-lock after 5 failed attempts (1 day lockout) Profile Guidelines:\nBasic: Application read-only access Standard: Application data manipulation (most common) Maintenance: Database administration tasks Admin: Server administration only (senior DBAs only) Best Practices:\nUse strong passwords (Bitwarden recommended) Follow naming convention strictly Regular privilege audits Document user purpose Time-limited access for temporary users Limit admin profile creation Environment Consistency Ensure all environments (UAT, Pre-Prod, Production) have:\nSame MySQL version partial_revokes = ON Password validation enabled Consistent user privilege patterns ","link":"https://systemsgolive.com/post/mysql-user-profile-creation/","section":"post","tags":["MySQL","User Creation","Priviledges","Role-Bases Access Profiles","Best Practices"],"title":"MySQL User Profile Creation Guide"},{"body":"","link":"https://systemsgolive.com/tags/priviledges/","section":"tags","tags":null,"title":"Priviledges"},{"body":"","link":"https://systemsgolive.com/tags/role-bases-access-profiles/","section":"tags","tags":null,"title":"Role-Bases Access Profiles"},{"body":"","link":"https://systemsgolive.com/tags/user-creation/","section":"tags","tags":null,"title":"User Creation"},{"body":"","link":"https://systemsgolive.com/categories/database/","section":"categories","tags":null,"title":"Database"},{"body":"","link":"https://systemsgolive.com/categories/infrastrucure/","section":"categories","tags":null,"title":"Infrastrucure"},{"body":"","link":"https://systemsgolive.com/tags/ssh/","section":"tags","tags":null,"title":"SSH"},{"body":"Current Issue \u0026amp; Security Risk In the current UAT setup, a shared system account vmadmin is used for both SSH login to the VM and SSH tunneling for MySQL Workbench. Each trusted internal user (e.g., DevOps and Software Engineers, Support Team) uses a dedicated private SSH key with this account.\nThis account is configured with passwordless sudo privileges:\n1vmadmin ALL=(ALL) NOPASSWD: ALL This effectively grants full root-level access to the system.\nExtending this model to external users (e.g., third-party vendors) introduces a serious security risk. Even if access is intended only for MySQL tunneling, using vmadmin would allow full VM login and administrative privileges — violating the principle of least privilege (POLP) and increasing the risk of accidental or malicious misuse.\nSecure Approach \u0026amp; Standardisation To mitigate this, a dedicated, non-interactive SSH user (workbench-user) has been introduced, designed specifically for MySQL access via SSH tunneling. This user:\nHas no interactive shell (/bin/false) Does not use SSH keys Is restricted to password-authenticated tunneling only Cannot log into the VM interactively This setup, already implemented in pre-production, is now being standardised across all non-production environments (DEV, QA, UAT).\nEach user is also provided with a dedicated MySQL account tied to an access profile (basic, standard, or maintenance), ensuring a clear separation between VM access and database access.\nTherefore, this approach cleanly separates SSH login to the VM itself from MySQL Workbench access, ensuring consistent and secure practices across all non-production environments.\nImplementation Steps Create a non-interactive user for MySQL SSH tunneling only 1sudo useradd -m -s /bin/false workbench-user # no shell access 2sudo passwd workbench-user # enter a strong password Verify user creation: 1grep workbench-user /etc/passwd 2# Expected output: workbench-user:x:1005:1005::/home/workbench-user:/bin/false Note: /bin/false means no interactive shell session, i.e., the user cannot log in to the VM via a terminal session.\nRestrict SSH access: Edit /etc/ssh/sshd_config and add: 1# Restrict workbench-user access to trusted VPN IP only 2AllowUsers workbench-user@\u0026lt;VPN_IP_address\u0026gt; 3 4## BEGIN WORKBENCH CONFIGURATION ### 5# Exclusive use of Workbench for 3rd-party users - add [date] 6Match User workbench-user 7 PasswordAuthentication yes 8 AuthenticationMethods password 9## END WORKBENCH CONFIGURATION ### Note: In this setup, the MySQL database server is only reachable from a specific VPN IP address. This directive enforces IP whitelisting, ensuring that workbench-user can only authenticate from that trusted source.\nRestart SSH securely: 1sudo sshd -t # Test config 2sudo systemctl restart sshd 3sudo systemctl status sshd ","link":"https://systemsgolive.com/post/mysql-workbench-ssh-tunnel-user/","section":"post","tags":["MySQL","Workbench","SSH"],"title":"Standardised MySQL Access via SSH Tunnel for Workbench in Non-Production Environments"},{"body":"","link":"https://systemsgolive.com/tags/workbench/","section":"tags","tags":null,"title":"Workbench"},{"body":"Overview This article outlines the implementation and usage of custom Apache 503 error pages on production Tomcat web application instances. These error pages are used to provide clear communication to users during service disruptions, with two supported scenarios:\nUnexpected Outage (default) Planned Maintenance 1 – Implementation These actions were performed only once during the initial implementation and are not required for routine operations.\nStep 1 – Backup Original Apache Configuration 1sudo mkdir -p /root/apache-config-backup 2sudo cp -a --no-preserve=timestamps /etc/httpd/conf/httpd.conf /root/apache-config-backup/httpd.conf-$(date +\u0026#34;%Y%m%d\u0026#34;) Step 2 – Update Apache Configuration File To enable custom Apache 503 error pages, add the following directives directly in your Apache configuration file:\nFile: /etc/httpd/conf/httpd.conf Location: Inside the \u0026lt;VirtualHost _default_:443\u0026gt; block. Required Directives (in order): 1RedirectMatch \u0026#34;^/(?!web/|admin/|custom_error_pages/).*$\u0026#34; /web/ If your configuration already uses a RedirectMatch rule, make sure to append custom_error_pages/ to the allowed paths. This ensures error pages are not redirected away and can be served correctly by Apache.\nEnable error page handling (place between proxy headers and backend proxy rules): 1DocumentRoot \u0026#34;/var/www/html\u0026#34; 2ProxyPass /custom_error_pages/ ! 3ErrorDocument 503 /custom_error_pages/503-unexpected-outage.html 4# ErrorDocument 503 /custom_error_pages/503-planned-maintenance.html Place these directives after ProxyRequests Off / ProxyPreserveHost On and before any ProxyPass rules to backend applications.\nStep 3 – Create Directory and Deploy HTML Files Execute the following as the root user to create and secure the directory:\n1mkdir -p /var/www/html/custom_error_pages 2chmod 755 /var/www/html/custom_error_pages 3chcon -R --reference=/var/www/html /var/www/html/custom_error_pages 4ls -ldZ /var/www/html/custom_error_pages Create and edit the HTML pages:\n1vim /var/www/html/custom_error_pages/503-unexpected-outage.html 2vim /var/www/html/custom_error_pages/503-planned-maintenance.html Reference: Example HTML files for both error scenarios can be found on in my GitHub repository:\n503-unepected-outatage.html 503-planned-maintenace.html Step 4 – Ensure Permissions and SELinux Context Ensure both HTML files are accessible by Apache:\n1chmod 644 /var/www/html/custom_error_pages/503-*.html 2chcon -u system_u -t httpd_sys_content_t /var/www/html/custom_error_pages/503-*.html 3ls -lZh /var/www/html/custom_error_pages/503-*.html 2 – Operational Guide This section covers routine operational procedures for switching between the two custom Apache 503 error pages.\n2.1\t– Default Behavior: Unexpected Outage Page (active now) The following page is active by default on all Tomcat web app instances in Production: /var/www/html/custom_error_pages/503-unexpected-outage.html\nIt is shown automatically if Tomcat becomes unavailable unexpectedly.\nNo action is needed during normal operations.\n2.2\t– Switch to Planned Maintenance Page Step 1 – Update the maintenance message content: Before enabling the planned maintenance error page, manually edit the HTML file to include the correct date and time of the scheduled maintenance window: /var/www/html/custom_error_pages/503-planned-maintenance.html\nImportant: This must be done proactively before the downtime.\nStep 2 – Modify Apache configuration: Edit the /etc/httpd/conf/httpd.conf file.\nYou need to comment out the line for the unexpected outage page (disabling it temporarily) and uncomment the line for the planned maintenance page (making it active during the maintenance window).\n1# ErrorDocument 503 /custom_error_pages/503-unexpected-outage.html 2ErrorDocument 503 /custom_error_pages/503-planned-maintenance.html Step 3 – Validate Apache configuration and restart: 1sudo apachectl configtest 2sudo systemctl restart httpd 3sudo systemctl status httpd Step 4 – Stop Tomcat (e.g., Tomcat-01): 1sudo systemctl stop tomcat Step 5 – Test Visit: https://www.example.com Clear your browser cache and refresh until you land on Tomcat-01. Step 6 – Verify: You should see the custom Planned Maintenance custom error page.\nStep 7 – Restart Tomcat: Once the maintenance activity is complete, do not forget to restart Tomcat\n1sudo systemctl start tomcat Step 7 – Repeat: Repeat the above steps on all the required Tomcat web app instances (Tomcat-02, Tomcat-03, and Tomcat-04).\nImportant: Once the planned maintenance activity is complete, please immediately revert to the default Unexpected Outage page for any future 503 errors – see section below.\n2.3\t– Revert to Unexpected Outage Page Step 1 – Modify Apache configuration Edit the /etc/httpd/conf/httpd.conf file. For this scenario, you now need to uncomment the line for the unexpected outage page (i.e. making it active again) and comment out the line for the planned maintenance page (i.e. disabling it until the next planned maintenance activity):\n1ErrorDocument 503 /custom_error_pages/503-unexpected-outage.html 2# ErrorDocument 503 /custom_error_pages/503-planned-maintenance.html Step 2 – Validate Apache configuration and restart: 1sudo apachectl configtest 2sudo systemctl restart httpd 3sudo systemctl status httpd Note: The steps described below are optional but recommended to confirm that the Unexpected Outage page has been successfully restored as the default page for all future 503 errors.\nStep 3 – Stop Tomcat (e.g., Tomcat-01): 1sudo systemctl stop tomcat Step 4 – Test Visit: https://www.example.com Clear your browser cache and refresh until you land on Tomcat-01. Step 5 – Verify: Validate if you see the custom Unexpected Outage error page. Now, the Unexpected Outage page will be the default page for all future 503 errors.\nStep 6 – Restart Tomcat: 1sudo systemctl start tomcat Step 7 – Repeat: Repeat the above steps on the required Tomcat web app instances.\nConclusion Custom Apache 503 error pages provide a clear and controlled way to communicate service interruptions. With proper setup, teams can switch between planned and unexpected outage messages based on operational needs.\n","link":"https://systemsgolive.com/post/custom-apache-503-error-pages/","section":"post","tags":["Infrastructure","Web Applications","Apache"],"title":"Configuration and Usage of Custom Apache 503 Error Pages for Web Application Instances"},{"body":"","link":"https://systemsgolive.com/tags/web-applications/","section":"tags","tags":null,"title":"Web Applications"},{"body":"Network troubleshooting and configuration are crucial skills for Linux system administrators or DevOps Engineer. This guide covers essential network commands for AlmaLinux8/RHEL8 systems.\nNote: Use sudo for commands requiring elevated privileges\n1. IP Command Modern replacement for ifconfig. Manages network interfaces and routing.\n1# Show network interfaces 2ip addr show 3 4# Enable/disable interface 5ip link set eth0 up 6ip link set eth0 down 7 8# Set IP address 9ip addr add 192.168.1.100/24 dev eth0 10 11# Show routing table 12ip route show 13 14# Add static route 15ip route add 10.0.0.0/24 via 192.168.1.1 2. Socket Statistics (ss) Replaces netstat, provides socket information.\n1# Show TCP connections 2ss -ta 3 4# Show listening ports 5ss -ltpn 6 7# Display processes using port 8ss -ltpn | grep :80 9 10# Show UDP sockets 11ss -ua 4. Network Manager Manages network connections and interfaces.\n1# Check NetworkManager status 2systemctl status NetworkManager 3 4# View all devices 5nmcli device status 6 7# Show connection details 8nmcli connection show 9 10# View real-time logs 11journalctl -fu NetworkManager 12 13# Restart NetworkManager 14sudo systemctl restart NetworkManager 15 16# Connect specific interface 17nmcli device connect eth0 5. Ping Tests network connectivity.\n1# Basic ping 2ping google.com 3 4# Limit to 5 packets 5ping -c 5 192.168.1.1 6 7# Set interval (seconds) 8ping -i 0.5 8.8.8.8 9 10# Set packet size 11ping -s 1500 google.com 6. Traceroute Identifies network path issues.\n1# Basic trace 2traceroute google.com 3 4# Use TCP 5traceroute -T google.com 6 7# Specify port 8traceroute -p 443 google.com 7. Nmap Network exploration and security scanning.\n1# Scan host 2nmap 192.168.1.100 3 4# Scan network range 5nmap 192.168.1.0/24 6 7# Scan specific ports 8nmap -p 80,443 192.168.1.100 9 10# OS detection 11nmap -O 192.168.1.100 8. cURL Tests web services and APIs.\n1# GET request 2curl http://example.com 3 4# Download file 5curl -O http://example.com/file.zip 6 7# POST request 8curl -X POST -d \u0026#34;data=value\u0026#34; http://api.example.com 9 10# Show headers 11curl -I https://example.com 9. dig DNS lookup utility.\n1# Basic lookup 2dig example.com 3 4# Query MX records 5dig example.com MX 6 7# Use specific DNS server 8dig @8.8.8.8 example.com 9 10# Trace resolution 11dig +trace example.com 10. tcpdump Network packet analyzer.\n1# Capture interface packets 2tcpdump -i eth0 3 4# Capture specific port 5tcpdump -i eth0 port 80 6 7# Save to file 8tcpdump -w capture.pcap 9 10# Read from file 11tcpdump -r capture.pcap 11. iptables Firewall management.\n1# List rules 2iptables -L 3 4# Allow SSH 5iptables -A INPUT -p tcp --dport 22 -j ACCEPT 6 7# Block IP 8iptables -A INPUT -s 192.168.1.100 -j DROP 12. netcat (nc) Network utility tool.\n1# Listen on port 2nc -l 8080 3 4# Connect to port 5nc example.com 80 6 7# Port scanning 8nc -zv example.com 20-30 13. host Simple DNS lookup.\n1# Basic lookup 2host example.com 3 4# Reverse lookup 5host 8.8.8.8 6 7# Query MX records 8host -t MX example.com Common Use Cases Web Server Troubleshooting 1# Check if web server is listening 2ss -ltpn | grep :80 3 4# Test local web server 5curl -I http://localhost 6 7# Check server certificate 8openssl s_client -connect yourdomain.com:443 9 10# Monitor HTTP traffic 11tcpdump -i eth0 port 80 -A 12 13# Check error logs 14tail -f /var/log/httpd/error_log Network Connectivity 1# Test DNS and internet connectivity 2ping -c 3 8.8.8.8 3ping -c 3 google.com 4 5# Check interface status and IP 6ip addr show 7nmcli device status 8 9# Verify routing 10ip route show 11traceroute problematic-server.com 12 13# Monitor interface traffic 14tcpdump -i eth0 -n 15 16# Check DNS resolution 17dig +trace yourdomain.com Security Checks 1# Port scanning 2nmap -sS -p- 192.168.1.0/24 3 4# Check open connections 5ss -tapn 6 7# Monitor connections in real-time 8watch ss -tapn 9 10# Review firewall rules 11firewall-cmd --list-all 12iptables -L -n -v 13 14# Check SELinux status 15sestatus 16getenforce Performance Analysis 1# Network interface statistics 2ip -s link show 3 4# Monitor bandwidth by process 5iftop -i eth0 6 7# Track network latency 8mtr google.com 9 10# Check network load 11netstat -s 12 13# Monitor network errors 14watch -n1 \u0026#39;netstat -i\u0026#39; 15 16# NetworkManager real-time logs 17journalctl -fu NetworkManager Service Status 1# Check system services 2systemctl status NetworkManager 3systemctl status firewalld 4 5# View network services 6ss -tulpn 7 8# DNS resolution check 9cat /etc/resolv.conf 10dig google.com 11 12# Review network logs 13journalctl -u NetworkManager Note: Many commands require root privileges. Use sudo as needed.\n","link":"https://systemsgolive.com/post/linux-network-commands/","section":"post","tags":["Networking"],"title":"Essential Linux Network Commands: A Practical Guide"},{"body":"","link":"https://systemsgolive.com/tags/networking/","section":"tags","tags":null,"title":"Networking"},{"body":"","link":"https://systemsgolive.com/tags/alert/","section":"tags","tags":null,"title":"Alert"},{"body":"","link":"https://systemsgolive.com/tags/connectivity/","section":"tags","tags":null,"title":"Connectivity"},{"body":"Problem Statement On Friday 6th September 2024 at 21:31, we received an alert from LogicMonitor indicating one of our production web app servers (Tomcat#3) was down, with the message: \u0026quot;The host Tomcat#3 (i-xxxxxxx) is down\u0026quot;. However, shortly after receiving the alert, we attempted to SSH into the VM and confirmed that the server was fully operational. But what did it go wrong?\nRoot Cause Investigation and Resolution A ticket was promptly raised with our IT service provider, responsible for managing our Cloud Production workloads, immediately following the alert. Their investigation provided the following findings:\n1. Ping Data: Ping data from the server stopped reporting at 16:09 on the same day. 2. Other Metrics: All other performance metrics from the server were reported normally, indicating that the server was functioning properly aside from the ping data issue. 3. Environment Consistency: Other servers in the same environment (e.g., Tomcat#1) continued to report ping data normally, indicating that the issue was isolated to Tomcat#3. Upon further analysis by our service provider, it was determined that the ping data had stopped reporting at 16:09, resulting in a delayed false alert at 21:31. The root cause of the issue was traced to a malfunction in the service provider's monitoring tool, which failed to capture ping data properly. Once the issue with the monitoring tool's collector was resolved on Monday, September 9th, ping monitoring resumed, and the false alert was cleared.\nGraph Explanation The graph below shows that the ping metrics, including round-trip time and packet counts, were being reported normally up until 16:09 on September 6th. After that, no new ping data was collected, confirming the issue was related to the monitoring service rather than the server itself.\nIn-House Alternative Solution: Fallback Ping Monitoring Script To reduce reliance on third-party monitoring services, I created a fallback ping monitoring script. Running on a separate VM, it pings target servers and sends email alerts if they don’t respond after several attempts. It’s a simple fallback solution for basic connectivity checks.\nEnvironment setup: This fallback ping monitoring solution was implemented and tested in an AWS environment. The setup included 3 EC2 instances:\n1 EC2 instance acting as the ping-monitoring server (our VM, not a service provider’s monitoring system). 2 EC2 instance as the target hosts. Implementation steps Steps 1 through 3 are the prerequisites before running the ping-failure-alert.sh script on our dedicated VM, which is acting as the monitoring server (not the service provider's monitoring system).\nStep 1 - Allow ICMP Through the Firewall Monitoring Server (our VM): Allow outbound ICMP traffic (ping) for all destinations. Target Servers: Allow inbound ICMP from the monitoring server’s IP. Test Connectivity: Run ping \u0026lt;target_host_public_IP\u0026gt; from the monitoring server to ensure reachability. Step 2 - Install mailx package to enable email notification mailx is a command-line email client used in Unix-like systems to send and receive emails directly from the terminal or within scripts.\nFor Debian/Ubuntu: sudo apt-get install mailx For RHEL/CentOS: sudo yum install mailx For Fedora: sudo dnf install mailx Step 3 - Gmail Configuration for SMTP Authentication To ensure that the script can send email notifications using Gmail’s SMTP service, you need to configure Gmail accordingly. Follow these steps:\nEnable App Passwords in Gmail if Two-Step Verification is on. Create an App Password under Google Account \u0026gt; Security \u0026gt; App Passwords. Update /etc/mail.rc on the Monitoring server with the following configuration: 1set smtp=smtps://smtp.gmail.com:465 2set smtp-auth=login 3set smtp-auth-user=\u0026lt;your_email_id\u0026gt;@gmail.com # provide the main Gmail address 4set smtp-auth-password=\u0026lt;your_generated_app_password\u0026gt; # do not leave any spaces between characters 5set ssl-verify=ignore Step 4 - Ping Monitoring Script The bash script below pings the servers and sends alerts via email if they are unreachable.\nCreate the script file and open it for editing (sudo nano ping-failure-alert.sh). Ensure the script is executable (sudo chmod +x ping-failure-alert.sh). Restrict access to root only for security (sudo chown root:root ping-failure-alert.sh and sudo chmod 700 ping-failure-alert.sh). 1################################################################################# 2# Script Name: ping-failure-alert.sh 3# Description: This script pings a predefined list of server IP addresses to check 4# their network connectivity. If any servers fail to respond after 5# a specified number of attempts and interval, an email notification 6# is sent. 7# Author: Mickael Asghar 8# Created on: 07/06/2024 9# Updated on: 07/06/2024 10################################################################################# 11 12#!/bin/bash 13 14# Email Configuration - Define email addresses here 15recipient_email=\u0026#34;ping-monitoring@abc.com\u0026#34; # Define the primary recipient 16cc_recipients=(\u0026#34;contact1@abc.com\u0026#34; \u0026#34;contact2@abc.com) # Define CC recipients 17 18# Convert CC recipients array to a comma-separated string 19cc_list=$(IFS=\u0026#39;,\u0026#39;; echo \u0026#34;${cc_recipients[*]}\u0026#34;) 20 21# Associative array of server IP addresses and their hostnames 22declare -A ping_targets=( 23 [\u0026#34;10.15.30.40\u0026#34;]=\u0026#34;target_host_1\u0026#34; # adjust IP address and hostname 24 [\u0026#34;50.60.70.80\u0026#34;]=\u0026#34;target_host_2\u0026#34; # adjust IP address and hostname 25 [\u0026#34;90.95.100.110\u0026#34;]=\u0026#34;target_host_3\u0026#34; # adjust IP address and hostname 26) 27 28# Retry settings 29retry_count=3 # Number of retry attempts 30retry_interval=30 # Interval in seconds between retries 31 32# Initialize a variable to store failed hosts 33failed_hosts=\u0026#34;\u0026#34; 34 35# Function to ping a host 36ping_host() { 37 local ip=$1 38 ping -c 1 $ip \u0026gt; /dev/null 2\u0026gt;\u0026amp;1 39 return $? 40} 41 42# Loop through each target and attempt to ping 43for ip in \u0026#34;${!ping_targets[@]}\u0026#34; # Loop over keys of the associative array 44do 45 hostname=${ping_targets[$ip]} # Assign hostname from the associative array 46 success=false 47 48 for attempt in $(seq 1 $retry_count) 49 do 50 echo \u0026#34;Pinging $hostname ($ip) (Attempt $attempt of $retry_count)...\u0026#34; 51 if ping_host $ip; then 52 echo \u0026#34;$hostname ($ip) is reachable.\u0026#34; 53 success=true 54 break 55 else 56 echo \u0026#34;$hostname ($ip) is not reachable. Waiting $retry_interval seconds before retrying...\u0026#34; 57 sleep $retry_interval 58 fi 59 done 60 61 if ! $success; then 62 current_datetime=$(date \u0026#34;+%d/%m/%Y %H:%M:%S\u0026#34;) # Get the current date and time 63 echo \u0026#34;$hostname ($ip) failed to respond after $retry_count attempts.\u0026#34; 64 failed_hosts+=\u0026#34;Date: $current_datetime - $hostname ($ip).\\n\u0026#34; # Append formatted string 65 fi 66done 67 68# Check if any host failed to respond and send an email if so 69if [ ! -z \u0026#34;$failed_hosts\u0026#34; ]; then 70 message=\u0026#34;The following hosts failed to respond after $retry_count attempts with a $retry_interval second interval between attempts:\\n$failed_hosts\u0026#34; 71 echo -e \u0026#34;$message\u0026#34; | mailx -s \u0026#34;[ALERT] - Ping Failure Notification\u0026#34; -c \u0026#34;$cc_list\u0026#34; \u0026#34;$recipient_email\u0026#34; 72else 73 echo \u0026#34;All hosts responded successfully after $retry_count attempts.\u0026#34; 74fi Key Settings: Retry Attempts (retry_count): The script will try to ping each server 3 times before declaring it unreachable. Interval Between Retries (retry_interval): There is a 30-second interval between retries. This ensures that short downtimes (e.g., server reboot) will not trigger immediate false alerts. Adjust values accordingly to meet your requirements: recipient_email, cc_recipients, target_host_IP, target_hostname Step 5 - Set up the Script as a Cron Job To automate the script, run it as a cron job every 5 minutes:\nOpen Crontab: sudo crontab -e Add the Cron Job: 1*/5 * * * * /path/to/ping-failure-alert.sh Replace /path/to/ping-failure-alert.sh with the actual path to your script. Avoiding Unnecessary Alerts With 3 retry attempts and a 30-second interval, the script waits ~90 seconds before declaring a server unreachable. This reduces unnecessary alerts for brief downtimes, like reboots.\nConclusion The false alert caused by the monitoring collector led us to realize that the server was never down, despite the loss of ping data. Having a fallback ping monitoring script offers a reliable alternative for connectivity checks, ensuring you’re not misled by false positives from external services. This backup system is lightweight, customisable, and independent of the main monitoring service.\n","link":"https://systemsgolive.com/post/implementating-backup-ping-monitoring-solution/","section":"post","tags":["Monitoring","Ping","Connectivity","Alert","Incident"],"title":"Dealing with False Ping Alerts in LogicMonitor: Building a Fallback Ping Monitoring Script for Production VMs"},{"body":"","link":"https://systemsgolive.com/tags/incident/","section":"tags","tags":null,"title":"Incident"},{"body":"","link":"https://systemsgolive.com/categories/incident-reports/","section":"categories","tags":null,"title":"Incident Reports"},{"body":"","link":"https://systemsgolive.com/tags/monitoring/","section":"tags","tags":null,"title":"Monitoring"},{"body":"","link":"https://systemsgolive.com/tags/ping/","section":"tags","tags":null,"title":"Ping"},{"body":"","link":"https://systemsgolive.com/tags/deployment/","section":"tags","tags":null,"title":"Deployment"},{"body":"","link":"https://systemsgolive.com/tags/environment/","section":"tags","tags":null,"title":"Environment"},{"body":"","link":"https://systemsgolive.com/tags/kubernetes/","section":"tags","tags":null,"title":"Kubernetes"},{"body":"","link":"https://systemsgolive.com/categories/kubernetes/","section":"categories","tags":null,"title":"Kubernetes"},{"body":"Overview In this guide, I'll demonstrate a blue-green deployment strategy in Kubernetes using Deployments and Services. The goal is to achieve zero downtime by running two sets of pods: the current version (v1.0, blue) and the new version (v2.0, green). I'll also explain how to roll back from green to blue if necessary.\nSetup the Blue Environment The blue-deployment.yaml defines the current environment running version 1.0 of the app.\n1apiVersion: apps/v1 2kind: Deployment 3metadata: 4 name: blue-deploy 5 labels: 6 app: my-node-app 7 env: blue 8 version: v1.0 9spec: 10 replicas: 2 11 selector: 12 matchLabels: 13 app: my-node-app 14 env: blue 15 template: 16 metadata: 17 labels: 18 app: my-node-app 19 env: blue 20 version: v1.0 21 spec: 22 containers: 23 - name: my-node-app 24 image: mik3asg/k8s-zero-downtime-deployment:v1.0 25 ports: 26 - containerPort: 3000 Deploy the Blue Environment: 1kubectl apply -f blue-deployment.yaml Check the deployment status: 1kubectl get deploy 2kubectl get pods -l env=blue Exposing the Blue Deployment The blue-green-service.yaml is responsible for routing traffic to our blue pods using a LoadBalancer Service. The traffic routing occurs because the selector values in blue-deployment.yaml (spec:selector:matchLabels) match the selector values defined in the blue-green-service.yaml (spec:selector).\nThis is how Kubernetes ensures that traffic is directed to the correct set of pods (in this case, the blue pods running version 1.0). Here's the service manifest:\n1apiVersion: v1 2kind: Service 3metadata: 4 name: blue-green-svc 5spec: 6 type: LoadBalancer 7 selector: 8 app: my-node-app 9 env: blue 10 ports: 11 - name: http 12 protocol: TCP 13 port: 80 14 targetPort: 3000 In this manifest:\nThe selector in blue-green-service.yaml specifies that the service should route traffic to pods labeled app: my-node-app and env:blue. This matches the labels set in the blue-deployment.yaml manifest.\nAs a result, all incoming traffic through this LoadBalancer Service will be routed to the blue pods running version 1.0 of the application.\nDeploy the service: 1kubectl apply -f blue-green-service.yaml Check the service and confirm traffic routing: 1kubectl get svc 2curl http://\u0026lt;EXTERNAL_IP\u0026gt; Setup the Green Environment At this stage, we will deploy the new version of the application (v2.0) by defining a new deployment manifest green-deployment.yaml. The green environment will be deployed alongside the blue environment, but traffic will still be routed to the blue pods until we update the service to point to the green ones.\n1apiVersion: apps/v1 2kind: Deployment 3metadata: 4 name: green-deploy 5 labels: 6 app: my-node-app 7 env: green 8 version: v2.0 9spec: 10 replicas: 2 11 selector: 12 matchLabels: 13 app: my-node-app 14 env: green 15 template: 16 metadata: 17 labels: 18 app: my-node-app 19 env: green 20 version: v2.0 21 spec: 22 containers: 23 - name: my-node-app 24 image: mik3asg/k8s-zero-downtime-deployment:v2.0 25 ports: 26 - containerPort: 3000 Here:\nThe labels applied to the green pods (app:my-node-app, env:green, version:v2.0) differentiate them from the blue pods running the older version. The selector in this deployment (app:my-node-app, env:green) will manage these green pods, ensuring Kubernetes knows which pods belong to this new environment. Deploy the Green Environment 1kubectl apply -f green-deployment.yaml # At this point, both the blue (v1.0) and green (v2.0) pods are running in parallel, but traffic is still being routed to the blue environment.\nUpdate the Service to Re-route traffic from Blue to Green To direct traffic to the green pods (version 2.0), we need to modify the existing service. This involves updating the selector in blue-green-service.yaml to match the labels of the green deployment. By doing so, we switch traffic from the blue environment to the green environment without any downtime.\nHere's the updated service manifest:\n1apiVersion: v1 2kind: Service 3metadata: 4 name: blue-green-svc 5spec: 6 type: LoadBalancer 7 selector: 8 app: my-node-app 9 env: green # update from blue to green for migration 10 ports: 11 - name: http 12 protocol: TCP 13 port: 80 14 targetPort: 3000 The selector in the updated blue-green-service.yaml is now set to env:green, meaning the service will route traffic to the green pods (v2.0) instead of the blue ones. Since the selector values in the green-deployment.yaml (spec:selector:matchLabels) now match those in the service (spec:selector), all traffic will flow to the green pods. Apply the updated service configuration: 1kubectl apply -f blue-green-service.yaml Check the status to confirm the traffic is routed to the green environment: 1kubectl get svc 2curl http://\u0026lt;EXTERNAL_IP\u0026gt; Rolling Back to the Blue Environment In a blue-green deployment, rolling back is straightforward because both environments (blue and green) are running simultaneously. If any issues are found in the green deployment (v2.0), you can quickly revert traffic back to the stable blue environment (v1.0) by updating the service selector.\nTo roll back traffic from the green pods (v2.0) to the blue pods (v1.0), we need to update the blue-green-service.yaml again. By changing the service’s selector back to env:blue, we ensure that the LoadBalancer routes traffic to the blue pods.\nHere's how to update the service for rollback:\n1apiVersion: v1 2kind: Service 3metadata: 4 name: blue-green-svc 5spec: 6 type: LoadBalancer 7 selector: 8 app: my-node-app 9 env: blue # Rollback update - route traffic back to blue (v1.0) 10 ports: 11 - name: http 12 protocol: TCP 13 port: 80 14 targetPort: 3000 In this manifest:\nThe selector has been changed back to env:blue (as indicated in the comment), which matches the labels of the blue pods (v1.0). This change re-routes all traffic back to the blue environment, restoring the previous stable version of the application. To perform the rollback, apply the updated service configuration: 1kubectl apply -f blue-green-service.yaml Verify that the service is now routing traffic to the blue pods: 1kubectl get svc 2curl http://\u0026lt;EXTERNAL_IP\u0026gt; Why Rollback is Efficient in Blue-Green Deployments Immediate Recovery: Since the blue environment is always running, traffic can be quickly redirected back without needing to redeploy the stable version. Minimal Risk: Rolling back is as simple as updating the service selector, with no need to terminate the green pods or affect user traffic. This approach provides flexibility, reduces risk, and ensures a smooth deployment process, making it ideal for production environments where uptime is critical.\nConclusion Blue-green deployments in Kubernetes allow for zero-downtime updates by running both the old (blue) and new (green) versions of an application in parallel. This strategy ensures that traffic can be seamlessly switched between versions by simply updating the service’s selector. In case of issues with the new version, rolling back to the stable version is quick and risk-free.\n","link":"https://systemsgolive.com/post/k8s-zero-downtime-deployments-blue-green-strategy/","section":"post","tags":["Kubernetes","Deployment","Strategy","Service","Routing","Environment"],"title":"Kubernetes - Zero Downtime Deployments: Blue/Green Strategy"},{"body":"","link":"https://systemsgolive.com/tags/routing/","section":"tags","tags":null,"title":"Routing"},{"body":"","link":"https://systemsgolive.com/tags/service/","section":"tags","tags":null,"title":"Service"},{"body":"","link":"https://systemsgolive.com/tags/strategy/","section":"tags","tags":null,"title":"Strategy"},{"body":"","link":"https://systemsgolive.com/tags/cpu/","section":"tags","tags":null,"title":"CPU"},{"body":"Problem Statement: On 15th August 2024, an incident occurred where the CPUBusyPercent alert did not trigger for one of our Production MySQL Database VMs (DB01), despite the CPU being at 100% for seven minutes. This was unexpected since the threshold settings in LogicMonitor were supposed to trigger alerts under such conditions.\nFigure 1: CPU usage plateau at 100% for 6 minutes without triggering an alert.\nAnalysis: The initial investigation revealed that the CPUBusyPercent alert settings were configured to trigger an alert after five consecutive polls (which equates to 6 minutes of sustained high CPU usage). In this case, the CPU usage plateaued at 100% for exactly 6 minutes (from 9:02 to 9:08 AM), barely meeting the alert trigger criteria. However, the alert was not active long enough to be sent out due to the immediate clear interval once the CPU usage dipped below the threshold.\nUpon further inspection, it was noted that the alert was indeed generated briefly at 09:08 AM, but because the CPU usage dropped shortly after, the alert was cleared immediately, and no notification was dispatched. This indicated that the default settings were not optimal for capturing such brief spikes in CPU usage.\nFigure 2: Previous LogicMonitor settings for CPUBusyPercent alerts.\nSolution Provided: To prevent this issue from recurring, the following adjustments were made to the alert settings:\nAlert Trigger Interval: Reduced from 5 consecutive polls (6 minutes) to 3 consecutive polls (4 minutes). This ensures that alerts are triggered more quickly when high CPU usage is detected. Alert Clear Interval: Changed from \u0026quot;Immediate\u0026quot; to 2 consecutive polls (2 minutes). This adjustment prevents alerts from being cleared too rapidly, allowing more time for notifications to be sent. These changes are designed to ensure that any similar CPU usage spikes in the future will trigger a timely alert, providing better monitoring and response capabilities.\nConclusion: This incident highlighted the importance of fine-tuning monitoring thresholds to balance between avoiding false positives and ensuring critical alerts are not missed. By adjusting the alert trigger and clear intervals, the monitoring system can now better capture and alert on CPU usage spikes, helping prevent similar issues in the future.\n","link":"https://systemsgolive.com/post/logicmonior-cpubusypercent-alert/","section":"post","tags":["Monitoring","CPU","Metrics","Alert","Threshold","MySQL"],"title":"LogicMonitor: Understanding Why CPUBusyPercent Alert Was Not Triggered"},{"body":"","link":"https://systemsgolive.com/tags/metrics/","section":"tags","tags":null,"title":"Metrics"},{"body":"","link":"https://systemsgolive.com/tags/threshold/","section":"tags","tags":null,"title":"Threshold"},{"body":"","link":"https://systemsgolive.com/tags/diskspaceoptimisation/","section":"tags","tags":null,"title":"DiskSpaceOptimisation"},{"body":"","link":"https://systemsgolive.com/tags/logmanagement/","section":"tags","tags":null,"title":"LogManagement"},{"body":"","link":"https://systemsgolive.com/tags/logrotate/","section":"tags","tags":null,"title":"Logrotate"},{"body":"Logrotate helps manage log files by automatically rotating, compressing, and removing them when they become too large or outdated, preventing excessive disk space usage and ensuring system stability.\nTable of Contents What is logrotate in a nutshell? Requirements Version Information for Logrotate Linux Tomcat and MySQL Scope of logs per logrotate configuration file Implementation of Tomcat Log Rotation Prerequisite - Disable internal Tomcat log rotation process Edit /opt/tomcat/conf/logging.properties configuration file Edit /opt/tomcat/conf/server.xml configuration file Delete Unnecessary Archived Logs in /opt/tomcat/logs/archived/ directory Logrotate Configuration for catalina.out log files Logrotate Configuration for Tomcat Miscellaneous Logs Logrotate Configuration for httpd log files Logrotate Configuration for unison.log file Changing the SELinux Context for /root/unison.log Custom cron job for unison log Logrotate Configuration for MySQL-related log files Configuring MySQL Authentication for Log Rotation Logrotate Troubleshooting and Best Practices Refer to Linux Manual Page Maintain Root Ownership for Logrotate Configuration Files Verify Ownership Permission and SELinux settings of log files Understanding the ‘su’ Directive with Specific ‘create’ Settings Run Manual Test Debug Mode Last Log Rotation Timestamps by Logrotate Compression Issue Verify Full File Paths Check for Typos Curly Braces and Comments What is logrotate in a nutshell? Logrotate is designed to ease the administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.\nSource: https://linux.die.net/man/8/logrotate\nRequirements Implement custom logrotate configurations for Tomcat and MySQL DB virtual servers for disk space optimisation. To do this, we define specific directives in the logrotate configurations, including:\nSetting the occurrence of log rotation (e.g., daily) for timely rotation of log files. Specifying the retention policy for rotated log files (e.g., retaining the last 10 rotations). Leveraging the compress directive. Configuring the ownership and permissions for the rotated log files to ensure appropriate access and security. Version Information for Logrotate Linux Tomcat and MySQL The logrotate configuration detailed in this document has been developed and tested on the following system versions. Knowing the version compatibility across these components is crucial to ensure that everything functions correctly and performs optimally.\nService/Tool Command Output Date Logrotate logrotate --version logrotate 3.14.0 12/08/2024 Linux cat /etc/os-release AlmaLinux 8.9 12/08/2024 Tomcat /opt/tomcat/bin/version.sh Apache Tomcat/9.0.83 12/08/2024 MySQL mysql -V mysql Ver 8.0.37 for Linux on x86_64 (MySQL Community Server - GPL) 12/08/2024 Scope of logs per logrotate configuration file The table below outlines the scope of log files managed by each logrotate configuration. Complete scripts for each logrotate configuration file are provided in the subsequent sections of this document.\nLogrotate configuration Log files /etc/logrotate.d/catalina_out /opt/tomcat/logs/catalina.out /etc/logrotate.d/tomcat_misc /opt/tomcat/logs/catalina.log, /opt/tomcat/logs/localhost.log, /opt/tomcat/logs/localhost_access_log.txt, /opt/tomcat/logs/debug.log, /opt/tomcat/logs/task-debug.log, /opt/tomcat/logs/task-generation.log /etc/logrotate.d/httpd /var/log/httpd/access_log, /var/log/httpd/error_log /etc/logrotate.d/mysql /var/log/mysqld.log, /var/log/mysql.log, /var/lib/mysql/log_slow_query.log Implementation of Tomcat Log Rotation Prerequisite - Disable internal Tomcat log rotation process As a prerequisite before configuring logrotate for the Tomcat-related log files, it is necessary to disable Tomcat's internal log rotation process to avoid any conflicts with logrotate. To do this, update the following configuration files on the Tomcat virtual machines.\nFile: /opt/tomcat/conf/logging.properties File: /opt/tomcat/conf/server.xml Edit /opt/tomcat/conf/logging.properties configuration file Set maxDays to -1. This allows log entries to be continuously written to a single log file without a retention period as specified in the documentation. Set the rotatable parameter to false. This enables external tools such as logrotate to manage the log rotation cycle. Comment out the manager and host-manager logs as they are no longer needed. Apply the changes in the logging.properties file accordingly as shown below:\n11catalina.org.apache.juli.AsyncFileHandler.level = FINE 21catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 31catalina.org.apache.juli.AsyncFileHandler.prefix = catalina. 4# Retain log file (no deletion) on the server file system, and disable internal Tomcat log rotation, 5# as log rotation and cleanup are managed externally by logrotate. 61catalina.org.apache.juli.AsyncFileHandler.maxDays = -1 71catalina.org.apache.juli.AsyncFileHandler.rotatable = false 81catalina.org.apache.juli.AsyncFileHandler.encoding = UTF-8 9 102localhost.org.apache.juli.AsyncFileHandler.level = FINE 112localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 122localhost.org.apache.juli.AsyncFileHandler.prefix = localhost. 13# Retain log file (no deletion) on the server file system, and disable internal Tomcat log rotation, 14# as log rotation and cleanup are managed externally by logrotate. 152localhost.org.apache.juli.AsyncFileHandler.maxDays = -1 162localhost.org.apache.juli.AsyncFileHandler.rotatable = false 172localhost.org.apache.juli.AsyncFileHandler.encoding = UTF-8 18 19# These logs are no longer required 20#3manager.org.apache.juli.AsyncFileHandler.level = FINE 21#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 22#3manager.org.apache.juli.AsyncFileHandler.prefix = manager. 23#3manager.org.apache.juli.AsyncFileHandler.maxDays = -1 24#3manager.org.apache.juli.AsyncFileHandler.rotatable = false 25#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8 26 27# These logs are no longer required 28#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE 29#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 30#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager. 31#4host-manager.org.apache.juli.AsyncFileHandler.maxDays = -1 32#4host-manager.org.apache.juli.AsyncFileHandler.rotatable = false 33#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8 34 35java.util.logging.ConsoleHandler.level = FINE 36java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter 37java.util.logging.ConsoleHandler.encoding = UTF-8 38 39 40############################################################ 41# Facility specific properties. 42# Provides extra control for each logger. 43############################################################ 44 45org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO 46org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandler 47 48# These logs are no longer required for our project 49#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO 50#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.AsyncFileHandler 51 52# These logs are no longer required for our project 53#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO 54#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.AsyncFileHandler After making the changes, perform the following steps:\nRestart Tomcat service: systemctl restart tomcat Check Tomcat service status: sudo systemctl status tomcat Reboot the VM reboot command. Delete Unnecessary Archived Logs in /opt/tomcat/logs/archived/ directory Logs such as task-debug.YYYY-MM-DD.log and debug.YYYY-MM-DD.2.log are generated for debugging purposes and are not needed for this project. These can be safely removed to free up disk space. As a temporary solution, we are using a cron job to remove the contents of this directory. To prevent these logs from being created in the future, their generation and rotation must be disabled at the application level.\nTo set up the cron job:\nAs the root user, open the crontab editor by running the following command: crontab -e Add the following line to the crontab file: 1# Daily at 12:15 AM remove all contents from \u0026#39;/opt/tomcat/logs/archived\u0026#39; as these logs are not needed. 2# Disable log generation and rotation at the application level to prevent future creation. 315 0 * * 0 rm -rf /opt/tomcat/logs/archived/* Logrotate Configuration for catalina.out log files Use SSH to access the Tomcat #01 server as the root user. Navigate to the directory by entering cd /etc/logrotate.d in the terminal. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled catalina_out Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.\n1/opt/tomcat/logs/catalina.out 2{ 3 su tomcat adminops 4 copytruncate 5 daily 6 rotate 7 7 compress 8 missingok 9 notifempty 10 dateext 11 dateformat -%Y-%m-%d-%H%M 12 create 770 tomcat adminops 13} For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/catalina_out Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration. Logrotate Configuration for Tomcat Miscellaneous Logs Use SSH to access the Tomcat #01 server as the root user. Navigate to the directory by entering cd /etc/logrotate.d in the terminal. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled tomcat_misc Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.\nUpdate the file with the following custom configuration 1/opt/tomcat/logs/catalina.log 2/opt/tomcat/logs/localhost.log 3/opt/tomcat/logs/localhost_access_log.txt 4/opt/tomcat/logs/debug.log 5/opt/tomcat/logs/task-debug.log 6/opt/tomcat/logs/task-generation.log 7{ 8 copytruncate 9 daily 10 rotate 7 11 compress 12 missingok 13 notifempty 14 su tomcat tomcat 15 create 640 tomcat tomcat 16 sharedscripts 17 postrotate 18 /bin/systemctl reload tomcat.service \u0026gt; /dev/null 2\u0026gt;/dev/null || true 19 endscript 20} For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/tomcat_misc Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration. Logrotate Configuration for httpd log files Use SSH to access the Tomcat #01 server as the root user. Navigate to the directory by entering cd /etc/logrotate.d in the terminal. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled catalina_out Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.\nUpdate the file with the following custom configuration 1/var/log/httpd/access_log 2/var/log/httpd/error_log 3{ 4 su root adminops 5 daily 6 rotate 7 7 compress 8 missingok 9 notifempty 10 create 770 root adminops 11 sharedscripts 12 postrotate 13 /bin/systemctl reload httpd.service \u0026gt; /dev/null 2\u0026gt;/dev/null || true 14 endscript 15} For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/httpd Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration. Logrotate Configuration for unison log files Use SSH to access the Tomcat #01 server as the root user. Navigate to the directory by entering cd /etc/logrotate.d in the terminal. Copy the existing file tomcat.disabled and rename it to unison using the command: cp -a tomcat.disabled unison Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.\nUpdate the file with the following custom configuration 1# The \u0026#39;daily\u0026#39; directive is intentionally omitted because it caused issues with the default logrotate daily cron job. 2# Instead, logrotate is manually invoked via a custom cron job set to run daily at 12:45 AM. 3# Cron job: 45 0 * * * /usr/sbin/logrotate /etc/logrotate.d/unison 4# This ensures that unison logs are rotated daily without conflicts. 5 6/root/unison.log 7{ 8 rotate 7 9 copytruncate 10 compress 11 dateext 12 dateformat -%Y-%m-%d-%H%M 13 missingok 14 notifempty 15 create 600 root root 16} Changing the SELinux Context for /root/unison.log The following steps outline a workaround for the issue where logrotate cannot rotate unison.log due to SELinux permissions, as the file is initially located under the /root directory. As the root user, change the SELinux context type for the /root/unison.log file from admin_home_t to var_log_t using the following commands:\nCheck the current SELinux context: ls -Z /root/unison.log Set the new SELinux context: semanage fcontext -a -t var_log_t \u0026quot;/root/unison.log\u0026quot; Apply the context change: restorecon -v /root/unison.log Verify the new SELinux context has been applied: ls -Z /root/unison.log Output expected: system_u:object_r:var_log_t:s0 /root/unison.log Custom cron job for unison.log Due to an issue with the daily directive in logrotate not working as expected, we need to set up a custom cron job to manually run logrotate for Unison logs. Follow these steps to create the cron job:\nAs the root user, open the crontab editor by running the following command: crontab -e Add the following line to the crontab file to manually run logrotate for Unison logs daily at 12:30 AM: 1# Manually run logrotate for unison logs daily at 12:45 AM as a workaround, since the \u0026#39;daily\u0026#39; directive in logrotate was not working for some reason. 2 330 0 * * * /usr/sbin/logrotate /etc/logrotate.d/unison Save and close the crontab file. Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration. However, schedule the daily crontab on the other Tomcat VMs for 12:45 AM. This timing ensures that the master Unison sync, which runs on the Tomcat 01 VM, completes first, allowing the other VMs to rotate their unison.log files with a slight delay. Logrotate Configuration for MySQL-related log files Use SSH to access the Tomcat #01 server as the root user. Navigate to the directory by entering cd /etc/logrotate.d in the terminal. Open the file named mysql for editing (e.g., vim or nano). Important Note: We encountered an issue where rotated MySQL log files were not being compressed as expected. To resolve this, the MySQL logrotate configuration file was updated with additional directives. The key directives added are the following:\ndelaycompress dateext dateformat -%Y-%m-%d-%H%M Outcome: These updates have successfully addressed the compression issue with rotated log files.\nUpdate the file with the following configuration: 1# The log file name and location can be set in 2# /etc/my.cnf by setting the \u0026#34;log-error\u0026#34; option 3# in [mysqld] section as follows: 4# 5# [mysqld] 6# log-error=/var/log/mysqld.log 7# 8# For the mysqladmin commands below to work, root account 9# password is required. Use mysql_config_editor(1) to store 10# authentication credentials in the encrypted login path file 11# ~/.mylogin.cnf 12# 13# Example usage: 14# 15# mysql_config_editor set --login-path=client --user=root --host=localhost --password 16# 17# When these actions has been done, un-comment the following to 18# enable rotation of mysqld\u0026#39;s log error. 19# 20 21/var/log/mysqld.log 22/var/log/mysql.log 23/var/lib/mysql/log_slow_query.log 24{ 25 daily 26 rotate 5 27 copytruncate 28 compress 29 delaycompress 30 dateext 31 dateformat -%Y-%m-%d-%H%M 32 missingok 33 notifempty 34 create 640 mysql mysql 35 postrotate 36 # just if mysqld is really running 37 if test -x /usr/bin/mysqladmin \u0026amp;\u0026amp; \\ 38 /usr/bin/mysqladmin --login-path=logrotate ping \u0026amp;\u0026gt;/dev/null 39 then 40 /usr/bin/mysqladmin --login-path=logrotate flush-logs 41 fi 42 endscript 43} Configuring MySQL Authentication for Log Rotation Set up a secure login path to store the root user credentials: mysql_config_editor set --login-path=logrotate --user=root --host=localhost --password Enter the root password when prompted. This command stores the credentials securely and avoids using plain text passwords in scripts. Verify the stored login paths: mysql_config_editor print --all Check the current permissions of the ~/.mylogin.cnf file: ls -l ~/.mylogin.cnf If the permissions are not set to -rw-------, update them to ensure that only the file owner can read and write to it: chmod 600 ~/.mylogin.cnf Repeat the previous steps for each additional MySQL DB VM to apply the same log rotation configuration. Logrotate Troubleshooting and Best Practices Resources for Logrotate Linux Shell Terminal: man logrotate Online Ressource: https://linux.die.net/man/8/logrotate Maintain Root Ownership for Logrotate Configuration Files The logrotate configuration files should have permissions set to 644 and ownership set to root, which are the default settings. Additionally, the SELinux context type should be system_u:object_r:etc_t. When creating a new custom logrotate file, it is advisable to use cp -a \u0026lt;existing_logrotate_conf\u0026gt; \u0026lt;custom_new_logrotate_config\u0026gt; to ensure that the permissions, ownership, and SELinux settings are preserved.\nVerify Ownership, Permission, and SELinux settings of log files Use the ls -l command to ensure that the ownership and permissions of the log files specified in the logrotate configuration file (e.g., etc/logrotate.d/\u0026lt;config_file\u0026gt;) match the settings provided in the logrotate configuration. For example: create 640 mysql mysql\nUse ls-Z to display the SELinux (Security-Enhanced Linux) security context of files and directories.\nUnderstanding the ‘su’ Directive with Specific ‘create’ Settings When logrotate is configured with create 770 root adminops, it sets new log files to 770 permissions, accessible only by the owner and group (root and adminops), and restricts access for others. Include the su directive (su root adminops) in the logrotate configuration to enforce this ownership.\nRun Manual Test Perform a manual test by executing the following command as root user: logrotate -vf /etc/logrotate.d/\u0026lt;config_file\u0026gt; Note: Ensure that the original log file contains log entries, as an empty log file will not be rotated.\nDebug Mode This mode is purely for verifying what logrotate would do under normal operational conditions. If you want to see logrotate performing the operations without making changes, run the following: logrotate --debug /etc/logrotate.d/\u0026lt;config_file\u0026gt;\nLast Log Rotation Timestamps by Logrotate cat /var/lib/logrotate/logrotate.status\nCompression Issue If the first rotated file is not being compressed, ensure that both the compress and copytruncate directives are declared in the logrotate configuration file.\nVerify Full File Paths Double-check that the full file paths of the log files are correctly declared in the logrotate configuration file. If uncertain, navigate to the directory containing the log files and run the command pwd to confirm.\nCheck for Typos Review the logrotate configuration file for any typographical errors that may be causing issues.\nCurly Braces and Comments Review the logrotate configuration file for any typographical errors that may be causing issues.\n","link":"https://systemsgolive.com/post/logrotate-configuration-setup/","section":"post","tags":["Logrotate","Linux","SysAdmin","DiskSpaceOptimisation","LogManagement"],"title":"Logrotate Configuration Setup in AlmaLinux 8.9"},{"body":"","link":"https://systemsgolive.com/tags/sysadmin/","section":"tags","tags":null,"title":"SysAdmin"},{"body":"","link":"https://systemsgolive.com/tags/argocd/","section":"tags","tags":null,"title":"ArgoCD"},{"body":"","link":"https://systemsgolive.com/categories/ci/cd-pipelines/","section":"categories","tags":null,"title":"CI/CD Pipelines"},{"body":"","link":"https://systemsgolive.com/tags/cicd/","section":"tags","tags":null,"title":"CICD"},{"body":"","link":"https://systemsgolive.com/tags/eks/","section":"tags","tags":null,"title":"EKS"},{"body":"","link":"https://systemsgolive.com/tags/flask/","section":"tags","tags":null,"title":"Flask"},{"body":"Overview This project showcases an end-to-end DevOps pipeline for deploying a basic Flask application using Jenkins Pipeline and GitOps (with ArgoCD) on an Amazon Elastic Kubernetes Service (EKS) cluster. It utilises two Git repositories:\nGitHub Repository for Continous Integration hosting our basic Flask application code GitHub Repository for GitOps and Update of K8s Manifest Pre-requisites/Assumptions: AWS Account created AWS CLI installed on local machine IAM user set up with AWS access key ID and AWS secret access key kubectl installed on local machine DockerHub Account created Application code hosted on GIT Repository Architecture/Design Overview Installation and Setup 1. Spin up an AWS EC2 instance for a Jenkins server: Specs: Instance Type OS Storage SG I/O Rules t3.small Ubuntu 15 GiB gp2 TCP/22, TCP/8080 SSH into your EC2 instance to install Jenkins and Docker packages.\nUse the user-data section within the EC2 Console to incorporate the following bash script for installing Java, Jenkins, and Docker packages:\nSpecs:\nInstance Type: t3.small OS: Ubuntu Storage: 15 GiB gp2 SG Inbound rules: TCP/22, TCP/8080 SSH into your EC2 instance to install Jenkins and Docker packages.\nUse the user-data section within the EC2 Console to incorporate the following bash script for installing Java, Jenkins, and Docker packages:\nWarning: Java 11 support in Jenkins ends after Sep 30, 2024. Installing an unsupported Java version may cause Jenkins to fail. Upgrade Java to a newer version. Refer to the documentation for details. 1#!/bin/bash 2 3# Update and upgrade the system 4sudo apt update \u0026amp;\u0026amp; sudo apt upgrade -y 5 6# Install Java 7sudo apt install -y openjdk-11-jre 8 9# Install Jenkins 10curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo gpg --dearmor -o /usr/share/keyrings/jenkins-keyring.gpg 11echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.gpg] https://pkg.jenkins.io/debian binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list \u0026gt; /dev/null 12sudo apt-get update 13sudo apt-get install -y jenkins 14 15# Install Docker 16sudo apt install -y docker.io 17 18# Grant Jenkins user permission to Docker daemon 19sudo usermod -aG docker jenkins 20sudo systemctl enable docker 21sudo systemctl restart docker SSH into your Jenkins EC2 instance and check status of packages by running the following commands:\njava -version sudo systemctl status jenkins sudo systemctl status docker 2. Configure Jenkins Install the necessary plugins by navigating to Manage Jenkins \u0026gt; Plugins \u0026gt; Available Plugins Choose Docker, Docker Pipeline and GitHub integration Restart Jenkins server Configure Credentials: Manage Jenkins \u0026gt; Credentials \u0026gt; Global \u0026gt; Add Credentials for GitHub (id=github, username=your_username, password=token_generated_in_GitHub) for DockerHub (id=dockerhub, username=your_username, password=your_dockerhub_pwd) Create 2 Jenkins Jobs: For CI Pipeline: New Item \u0026gt; Name=BuildAppJob \u0026gt; Pipeline Build Triggers=GitHub hook trigger for GITScm polling Navigate to the GitHub repository settings and enable the Webhook by following these steps: Settings \u0026gt; Webhook \u0026gt; Payload URL=http://jenkins_server_public_ip:8080/github-weebhook/) Content type=application/json. Confirm Add webhook Pipeline Definition=Pipeline script from SCM SCM=GIT Script Path=Jenkinsfile Repository URL=https://github.com/Mik3asg/Flask_App_Jenkins_CI_EKS.git Credentials=none Branch Specifier=*/main For CD Pipeline: New Item \u0026gt; Name=UpdateK8sManifestJob \u0026gt; Pipeline Select This project is parameterized String Parameter: Name=DOCKERTAG, Default Value=latest Pipeline Definition=Pipeline script from SCM SCM=GIT Script Path:Jenkinsfile Repo URL=https://github.com/Mik3asg/Flask_App_Jenkins_GitOps_EKS.git Credentials=none # public repo Branch Specifier=*/main 3. Provision AWS EKS Ckuster, through AWS CLI Access your AWS Account via aws configure in the Terminal of your local machine\nProvide the credentials of your IAM user (access key ID and secret key) and region\nNote: For consistency, use the same region in which you have created your previous EC2 instance for Jenkins server, i.e. us-east-1\nCreate an AWS EKS Cluster\n1eksctl create cluster --name \u0026lt;flask-eks\u0026gt; --region us-east-1 --nodegroup-name \u0026lt;my-nodes\u0026gt; --node-type t3.small --managed --nodes 2 # Replace \u0026lt;flask-eks\u0026gt; and \u0026lt;my-nodes\u0026gt; with your desired values Check the status of EKS Cluster (if up and running) 1eksctl get cluster --name demo-eks --region us-east-1 Run kubectl get nodes command to verify the status of the nodes 4. Installation and Configuration of Argo CD Via CLI: Follow steps provided in the official Argo CD Documentation to: Install Argo CD via CLI\nConnect to the API server from local machine and access to UI using https://localhost:8080\nNote: Keep the kubectl port-forwarding terminal open to avoid disrupting access to the UI. Open a new terminal if you need CLI access.\nRetrieve Password for UI access\nIn UI: Application Name=flask-gitops-demo Project Name=Default SYNC POLICY=Automatic REPO URL: \u0026lt;Github_repo_hosting_CD_Pipeline\u0026gt; Path:./ Cluster URL=https://kubernetes.default.svc Namespace=default 5. Commit a new code change and test app deployment Check status in Jenkins UI of both Jobs: BuildAppJob for CI Pipeline and UpdateK8sManifestJob for CD Pipeline Check status of pods: kubectl get pods Check status in Argo CD UI Retrieve load balancer endpoint by running kubectl get svc, then paste into web browser to access the web application Clean-up resources Delete AWS EKS Cluster\n1eksctl delete cluster --name flask-eks --region us-east-1 #Replace \u0026lt;flask-eks\u0026gt; with the value you defined for your cluster Terminate the running EC2 instance for Jenkins server in AWS Console Management\n","link":"https://systemsgolive.com/post/flask-app-jenkins-cicd-argocd-k8s/","section":"post","tags":["Flask","EKS","CICD","Jenkins","ArgoCD"],"title":"Flask App deployment into Amazon EKS, using CICD Pipeline with Jenkins and Argo CD"},{"body":"","link":"https://systemsgolive.com/tags/jenkins/","section":"tags","tags":null,"title":"Jenkins"},{"body":"","link":"https://systemsgolive.com/tags/api/","section":"tags","tags":null,"title":"API"},{"body":"Overview This project demonstrates building a basic REST API in Golang, implementing CRUD operations with a PostgreSQL database for data persistence. The application utilises Gorilla Mux for routing, PostgreSQL for the database, and Docker for containerisation.\nThe API manages user information, including name, email, and city.\nNote: This demo does not include an user interface as it has not been built for the purpose of this demo. The code for this demo is available on GitHub.\nPre-requisites/Assumptions: Download and Install GO on your local machine. Download and Install Docker Desktop. Use your preferred IDE (e.g. Visual Studio Code). Download and Install Postman Desktop Agent to test the API locally. Use a PostgreSQL desktop/terminal client (e.g. Sqlectron). Architecture/Design Overview Setting up the environment Clone the repository:\n1git clone https://github.com/Mik3asg/Rest-API-Golang-Mux-PostgreSQL-Docker.git Navigate to the project directory:\n1cd Rest-API-Golang-Mux-PostgreSQL-Docker Dependencies:\nThis project uses Go modules for dependency management. The necessary dependencies, including Gorilla Mux for handling HTTP routing and pq for PostgreSQL database interaction, are already included in the go.mod and go.sum files. These dependencies were initially installed using the following commands:\n1go mod init api # Initializes a GO module named \u0026#39;api\u0026#39; for dependency management 2go get github.com/gorilla/mux # Installs Gorilla Mux for handling HTTP routing in Go 3go get github.com/lib/pq # Installs pq, PostgreSQL driver for Go\u0026#39;s database/sql package However, users do not need to run these commands themselves, as the dependencies are already included in the project. Simply clone the repository and ensure you have Go installed on your machine. If you're not familiar with Go modules, you can learn more about them here.\nRun Docker Engine on your local machine Setting up PostgreSQL Authentication To configure authentication for the PostgreSQL database used in this project, you need to set up environment variables in a .env file. Follow these steps:\nCreate a new file named .env in the root directory of your project.\nOpen the .env file in a text editor and add the following lines:\n1POSTGRES_USER=your_postgres_username 2POSTGRES_PASSWORD=your_postgres_password 3POSTGRES_DB=your_database_name Replace your_postgres_username, your_postgres_password, and your_database_name with your actual PostgreSQL credentials.\nSave the .env file in the same directory as your docker-compose.ymlfile, and Docker Compose will automatically read these environment variables when you run docker-compose up.\nThese environment variables will be read by Docker Compose and used to authenticate your application with the PostgreSQL database.\nStart the PostgreSQL Database Container Run the following command to start the PostgreSQL database service (go-db) defined in the docker-compose.yml file. The -d flag runs the container in detached mode, allowing it to run in the background.\n1docker compose up -d go-db Build and Run the custom CRUD GO API These commands build the custom Go application defined in the docker-compose.yml file and then run it using Docker Compose.\n1docker compose build # Build the custom Go app 2docker compose up go-app # Run the custom Go app Validate the setup Run the following commands:\n1docker images # Check the status of Docker images for go-db and go-app 2docker ps # Check the status of running Docker containers for go-db and go-app Test API endpoints Once the application is running, you can access it in your web browser at http://localhost:8080 . We will use Postman as an API Platform to test our API endpoints.\nIn addition, we can use a PostgreSQL client to check the data being stored in the database for each endpoint. Please refer to your database credentials defined in your .env file.\nCreate User with POST Endpoint: POST localhost:8000/users To create a new user, send a POST request to localhost:8000/users with a JSON body containing the user details. Use a PostgreSQL client to verify the new user data was inserted into the database. GET All Users Endpoint: GET localhost:8000/users To get all existing users, send a GET request to localhost:8000/users. The response will contain a JSON array of all users. Check the PostgreSQL client to verify the user data being returned. GET Single User Endpoint: GET localhost:8000/users/\u0026lt;id\u0026gt; To get a specific user, send a GET request to localhost:8000/users/\u0026lt;id\u0026gt; where \u0026lt;id\u0026gt; is the id of the desired user. Check the PostgreSQL client that the correct user data is being returned based on the id. PUT Update User Endpoint: PUT localhost:8000/users/\u0026lt;id\u0026gt; To update a user, send a PUT request to localhost:8000/users/\u0026lt;id\u0026gt; where \u0026lt;id\u0026gt; is the id of the user to update. Use the PostgreSQL client to verify the user data was updated in the database. DELETE User Endpoint: DELETE localhost:8000/users/\u0026lt;id\u0026gt; To delete a user, send a DELETE request to localhost:8000/users/\u0026lt;id\u0026gt; where \u0026lt;id\u0026gt; is the id of the user to delete. Check the PostgreSQL client that the user was removed from the database. Error Handling Invalid Endpoint: GET/PUT/DELETE localhost:8000/users/\u0026lt;invalid_id\u0026gt; If an invalid \u0026lt;id\u0026gt; is provided in the endpoint for GET, PUT or DELETE requests, the API will return a 404 status with the message \u0026quot;User not found\u0026quot;. The PostgreSQL client will show that no data was changed. ","link":"https://systemsgolive.com/post/rest-api-golang-docker-postgresql/","section":"post","tags":["Golang","Docker","PostgreSQL","API"],"title":"Building a simple REST API in Golang with MUX, PostgreSQL and Docker"},{"body":"","link":"https://systemsgolive.com/tags/docker/","section":"tags","tags":null,"title":"Docker"},{"body":"","link":"https://systemsgolive.com/tags/golang/","section":"tags","tags":null,"title":"Golang"},{"body":"","link":"https://systemsgolive.com/tags/postgresql/","section":"tags","tags":null,"title":"PostgreSQL"},{"body":"","link":"https://systemsgolive.com/categories/software-engineering/","section":"categories","tags":null,"title":"Software Engineering"},{"body":"","link":"https://systemsgolive.com/archives/","section":"","tags":null,"title":""},{"body":"","link":"https://systemsgolive.com/archive/","section":"archive","tags":null,"title":"Archives"},{"body":"","link":"https://systemsgolive.com/","section":"","tags":null,"title":""},{"body":"","link":"https://systemsgolive.com/about/","section":"","tags":null,"title":"About"},{"body":"","link":"https://systemsgolive.com/timeline/","section":"","tags":null,"title":"Engineering Notes — Timeline"},{"body":"","link":"https://systemsgolive.com/tags/index/","section":"tags","tags":null,"title":"Index"},{"body":"","link":"https://systemsgolive.com/post/","section":"post","tags":["index"],"title":"Posts"}]