@jnsamyak:matrix.org ==================== 2025-11-30 08:05:00 - Last week was a partial week off, but the weekend turned out to be quite hectic with multiple releng fires and discussions: - Dealt with multiple Robosignatory issues, including signing delays and configuration verification. - Participated in EOL (End of Life) discussions and related cleanup follow-ups. - Debugged a corrupt RPM issue that was impacting compose completion. -Investigated and helped with Bodhi compose failures for Fedora 41 ensured fixes. Monday: - Had a RelEng update call with Patrik to discuss this week’s goals and priorities, especially around migration progress and ongoing cleanup tasks. - Participated in the Release Engineering sprint call, and created the Sprint 6 board to track all ongoing and upcoming work items. - Continued to deal with migration blockers, EOL follow-ups, and routine releng ops tasks (pushes, build tracking, and compose checks). - Started looking into Forgejo Actions integration as a potential replacement for Zuul, evaluating its feasibility and how it fits into the Fedora infrastructure workflow. @gwmngilfen:fedora.im ===================== 2025-12-02 10:16:00 - yesterday public holiday - Fri: - posted zabbix postgres PR - worked on rabbitmq Fedora patch 2025-11-28 10:10:00 - sent pagure migration warnings - 1:1 with Fabian on package-build howto - debugged an IPA auth problem with Michal - started work on zabbix psql check for centos 2025-11-27 11:08:00 - not a whole lot, kept getting interrupted 😕 - worked on a PR for rabbit monitoring in prod - some troubleshooting 2025-11-26 09:56:00 - investigated vmhost-x86-cc02 disk failure - expanded zabbix01 disk (again 😕) - fixed issue with 500 errors in mailman3 - sorted networking on bmvhost-p09-05 - tried to figure out rising FMN queue on rabbit (seems ok today) - internal writing & meetings @arrfab:fedora.im ================= 2025-12-01 09:14:00 - migrated centos stream stg infra away from openstack and now running "on prem" (https://issues.redhat.com/browse/CS-2967) - Analyzed and unblocked signing queue for stream (https://issues.redhat.com/browse/CS-3216) - Start reviewing @gwmngilfen proposal for postgresql zabbix template 2025-11-28 13:15:00 - Closing hardware decom ticket for rdu2 ! (https://gitlab.com/CentOS/infra/tracker/-/issues/1814) , all cleaned-up, reinstalled, powered down, removed from inventory/zabbix/ansible - Closed ticket (pairing mode with @gwmngilfen) about rpm packaging and cbs workflow for infra tag (https://gitlab.com/CentOS/infra/tracker/-/issues/1768) 2025-11-27 09:24:00 - Reinstalled/cleaned- up plenty of machines (https://gitlab.com/CentOS/infra/tracker/-/issues/1814) - Migrated two ansible hosts to RDU3 (stg/dev vs prod/CI) : https://gitlab.com/CentOS/infra/tracker/-/issues/1799 2025-11-26 13:08:00 - Worked with CERN (on irc) people to clarify and close ip change ticket (https://gitlab.com/CentOS/infra/tracker/-/issues/1817) - Fully migrated https://repo.dev.centos.org in RDU3 (https://gitlab.com/CentOS/infra/tracker/-/issues/1818 - dc move) - Continuing decommissioning Dell/aarch64 infra in RDU2 (https://gitlab.com/CentOS/infra/tracker/-/issues/1814) 2025-11-25 08:53:00 - Decommissioned plenty of hardware (reinstall/clean-up) from rdu2 community cage (more to come - https://gitlab.com/CentOS/infra/tracker/-/issues/1814) - PTO day yesterday - Reviewed (and fixed) temporary backup plan for RDU3 Netapp (from RDU2 while waiting for machine to move) - https://gitlab.com/CentOS/infra/tracker/-/issues/1812 - Attended DC-move meeting for current status @zlopez:fedora.im ================= 2025-12-01 09:18:00 - Fix stuck robosignatory queue - Helping Samyak to fix this together with Greg - Add forge.fedoraproject.org to jira_sync - Fix signing queue - again 😕 2025-11-28 09:11:00 - Investigate what is the problem with ipsilon02 - I just restarted the httpd on ipsilon02 and it started working (some update probably fixed the issue in the last few weeks) - Fix authentication issue on ipsilon01 - we looked at this together with Greg and found out that sssd is down, just restarting it fixed the issue - release-monitoring.org - Review Anitya [PR](https://github.com/fedora-infra/anitya/pull/1981) - Please make me a sponsor of ocp-cico-networkmanager ([ticket](https://pagure.io/fedora-infrastructure/issue/12945)) - Re-enable project creation on stg.pagure.io ([ticket](https://pagure.io/fedora-infrastructure/issue/12943)) - Can't join `fedora-contibutor` FAS group ([ticket](https://pagure.io/fedora-infrastructure/issue/12942)) 2025-11-27 09:00:00 - Deal with AI scrapers on pagure - Deal with AI scrapers on dist-git - Add abompard to sysadmin-openshift 2025-11-26 08:42:00 - I&R weekly report - Change Bugzilla assignee for adwaita-fonts component ([ticket](https://pagure.io/fedora-infrastructure/issue/12935)) - Setup OpenID only ipsilon instance on production ([ticket](https://pagure.io/fedora-infrastructure/issue/10241)) - everything is working as it should - Write [blog post](https://communityblog.fedoraproject.org/end-of-openid-in-fedora/) about end of OpenID in Fedora - Process PDR request - Release [new version](https://github.com/fedora-infra/the-new-hotness-messages/releases/tag/1.3.0) of the-new-hotness message schema - Investigate the bvmhost-p09-05 alert - host was down and I tried to get it back again, but have to leave, so the problem was in the end resolved by @Gwmngilfen 2025-11-25 08:48:00 - Deploy new changes in toddlers to production - Business Continuity Plan for Fedora Exercise - IPA failure @amedvede:fedora.im =================== 2025-12-01 11:56:00 - Prepare email announcement for compose-tracker and failed-composes migration - Rewrite compose-tracker script to enable work with new forge 2025-11-26 18:03:00 - investigation all the projects that are connected to copose tracker repository - prepare changes for move - hel @nirik:matrix.scrye.com ======================= 2025-12-01 15:52:00 - Downgraded oci-registries to 2.8.x and filed bug on 3.0.0 - Got pagure-stg02 all working, got pagure01 installed and ansiblized. Started syncing data to them - Fixed openshift logging pods (gave them more memory) @patrikp:matrix.org =================== 2025-11-26 11:17:00 - Finished my investigation into migration of `pungi-fedora`. https://forge.fedoraproject.org/releng/tickets/issues/13090#issuecomment-245445