Linux Update Experience

Started by Zwettler Markus (OIZ)almost 6 years ago13 messagesgeneral

Jump to latest

Zwettler Markus (OIZ)

Markus.Zwettler@zuerich.ch

almost 6 years ago

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30
+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

Thanks,
Markus

Jiří Fejfar

jurafejfar@gmail.com

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#1)

Re: Linux Update Experience

On 28.05.2020 9:59, Zwettler Markus (OIZ) wrote:

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum
update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30

+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing
requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
(imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)

Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating
anymore?

Thanks,

Markus

Hi Markus,

we are using PGDG on Debian / Ubuntu trying to update Linux as frequent
as possible. I do not remember many issues if any. I would not expect:

* problems between postgres minor version upgrades (12.2 -> 12.3). We
have PG 12.3 & PostGIS 3.0 on Ubuntu Bionic and do not remember any
update problems.

* major upgrades when keeping same Linux distribution version. Our older
Ubuntu Xenial keeps PG 9.6 & PostGIS 2.3.

So from my perspective: linux updates should not trigger major DB
upgrades (or postGis), minor DB upgrades should be smooth -> it is
possible to update Linux continuously. Major DB upgrades have so be
triggered manually (on same Linux distro version or newer). On that
Ubuntu Bionic server, we started on PG10 upgrading DB together with
PostGis every year with dump / restore procedure to fresh new DB.

Jiří Fejfar

Marco Lechner

mlechner@bfs.de

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#1)

AW: Linux Update Experience

Hi Markus,

at the moment we are facing similar conflicts on Oracle LInux 7 (wich is derived from RHEL) - we manage our machines using Spacewalk. The conflicts occur (as expected) on Spacewalk as well as on manually using yum:

Fehler: Paket: postgresql11-devel-11.8-1PGDG.rhel7.x86_64 (oraclelinux7postgresql11)
Benötigt: llvm-toolset-7-clang >= 4.0.1
Fehler: Paket: postgis30_12-3.0.1-5.rhel7.x86_64 (oraclelinux7postgresql12)
Benötigt: proj70 >= 7.0.1
Installiert: proj70-7.0.0-2.rhel7.x86_64 (@oraclelinux7postgresql11)
proj70 = 7.0.0-2.rhel7
Verfügbar: proj70-7.0.0-1.rhel7.x86_64 (oraclelinux7postgresql11)
proj70 = 7.0.0-1.rhel7
Fehler: Paket: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (oraclelinux7postgresql12)
Benötigt: llvm-toolset-7-clang >= 4.0.1

Marco

--
Dr. Marco Lechner
Bundesamt fuer Strahlenschutz / Federal Office for Radiation Protection
RN Radiologischer Notfallschutz / Radiological Emergency Preparedness and Response
RN1 Koordination Notfallschutzsysteme / Coordination Emergency Systems
Rosastrasse 9 | D-79098 Freiburg | Germany
mlechner@bfs.de<mailto:mlechner@bfs.de> | +49 (0)3018 333 6724 | www.bfs.de<http://www.bfs.de>

Hinweis zu Anhängen die auf .p7m/.p7c/.p7s oder .asc/.asc.sig enden:
Die .p7?- und .asc-Dateien sind ungefährliche Signaturdateien (digitale Unterschriften). In E-Mail-Clients mit S/MIME Konfiguration (.p7?) oder PGP-Erweiterung (.asc) dienen sie zur:
- Überprüfung des Absenders
- Überprüfung einer evtl. Veränderung des Inhalts während der Übermittlung über das Internet
Die Signaturdateien können ebenso dazu verwendet werden dem Absender dieser Signatur eine E-Mail mit verschlüsseltem Inhalt zu senden. In E-Mail-Clients ohne S/MIME Konfiguration oder PGP-Erweiterung erscheinen die Dateien als Anhang und können ignoriert werden.

Von: Zwettler Markus (OIZ) [mailto:Markus.Zwettler@zuerich.ch]
Gesendet: Donnerstag, 28. Mai 2020 09:59
An: PostgreSQL General <pgsql-general@lists.postgresql.org>
Betreff: Linux Update Experience

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30
+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

Thanks,
Markus

Alan Hodgson

ahodgson@lists.simkin.ca

almost 6 years ago

In reply to: Marco Lechner (#3)

Re: AW: Linux Update Experience

On Thu, 2020-05-28 at 09:00 +0000, Marco Lechner wrote:

Hi Markus,

at the moment we are facing similar conflicts on Oracle LInux 7 (wich
is derived from RHEL) – we manage our machines using Spacewalk. The
conflicts occur (as expected) on Spacewalk as well as on manually
using yum:

Fehler: Paket: postgresql11-devel-11.8-1PGDG.rhel7.x86_64
(oraclelinux7postgresql11)
Benötigt: llvm-toolset-7-clang >= 4.0.1

FWIW, postgresql-devel can't be updated on CentOS 7 currently either.
The 12.2 packages were fine but I have not been able to update to 12.3.

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (pg12)
Requires: llvm-toolset-7-clang >= 4.0.1

That llvm-toolset-7-clang dependency is not present in the CentOS, EPEL
or PostgreSQL repos.

Zwettler Markus (OIZ)

Markus.Zwettler@zuerich.ch

almost 6 years ago

In reply to: Marco Lechner (#3)

AW: Linux Update Experience

Hi Marco,

How do you handle these conflicts? No longer updating that regularly or not at all anymore?

Thanks,
Markus

Von: Marco Lechner <mlechner@bfs.de>
Gesendet: Donnerstag, 28. Mai 2020 11:01
An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch>; PostgreSQL General <pgsql-general@lists.postgresql.org>
Betreff: AW: Linux Update Experience

Hi Markus,

Marco

Von: Zwettler Markus (OIZ) [mailto:Markus.Zwettler@zuerich.ch]
Gesendet: Donnerstag, 28. Mai 2020 09:59
An: PostgreSQL General <pgsql-general@lists.postgresql.org<mailto:pgsql-general@lists.postgresql.org>>
Betreff: Linux Update Experience

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30
+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

Thanks,
Markus

Marco Lechner

mlechner@bfs.de

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#5)

AW: Linux Update Experience

Dear Markus,

we are doing Updates almost continously/daily. The dependeny problems occur since a few days/1-2 weeks (?).
The Updates are pending since then - hoping for package maintainers to fix this, but did not yet address this in Bugtracker.

Marco

Von: Zwettler Markus (OIZ) [mailto:Markus.Zwettler@zuerich.ch]
Gesendet: Donnerstag, 28. Mai 2020 13:42
An: Marco Lechner <mlechner@bfs.de>; PostgreSQL General <pgsql-general@lists.postgresql.org>
Betreff: AW: Linux Update Experience

Hi Marco,

How do you handle these conflicts? No longer updating that regularly or not at all anymore?

Thanks,
Markus

Von: Marco Lechner <mlechner@bfs.de<mailto:mlechner@bfs.de>>
Gesendet: Donnerstag, 28. Mai 2020 11:01
An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch<mailto:Markus.Zwettler@zuerich.ch>>; PostgreSQL General <pgsql-general@lists.postgresql.org<mailto:pgsql-general@lists.postgresql.org>>
Betreff: AW: Linux Update Experience

Hi Markus,

Marco

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30
+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

Thanks,
Markus

Adrian Klaver

adrian.klaver@aklaver.com

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#1)

Re: Linux Update Experience

On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30

+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing
requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
(imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)

ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

See here:

https://yum.postgresql.org/news-newreporpmsreleased.php

And if you have community account:

https://redmine.postgresql.org/issues/5483

To contact the RPM packagers directly:

https://yum.postgresql.org/contact.php

Thanks,

Markus

--
Adrian Klaver
adrian.klaver@aklaver.com

Zwettler Markus (OIZ)

Markus.Zwettler@zuerich.ch

almost 6 years ago

In reply to: Adrian Klaver (#7)

AW: Linux Update Experience

-----Ursprüngliche Nachricht-----
Von: Adrian Klaver <adrian.klaver@aklaver.com>
Gesendet: Donnerstag, 28. Mai 2020 16:15
An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch>; PostgreSQL General
<pgsql-general@lists.postgresql.org>
Betreff: Re: Linux Update Experience

On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30

+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing
requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
(imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)

Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

See here:

https://yum.postgresql.org/news-newreporpmsreleased.php

And if you have community account:

https://redmine.postgresql.org/issues/5483

To contact the RPM packagers directly:

https://yum.postgresql.org/contact.php

Thanks,

Markus

--
Adrian Klaver
adrian.klaver@aklaver.com

[Zwettler Markus (OIZ)]

Hi Adrian,

I'm not talking about this specific bug or its resolution.

I want to talk about the Linux update problem in general.

Anyone updating Linux might get such nerving dependency troubles.

How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

Cheers,
Markus

Adrian Klaver

adrian.klaver@aklaver.com

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#8)

Re: AW: Linux Update Experience

On 5/28/20 7:36 AM, Zwettler Markus (OIZ) wrote:

-----Ursprüngliche Nachricht-----
Von: Adrian Klaver <adrian.klaver@aklaver.com>
Gesendet: Donnerstag, 28. Mai 2020 16:15
An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch>; PostgreSQL General
<pgsql-general@lists.postgresql.org>
Betreff: Re: Linux Update Experience

On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:

We are running PGDG Postgres 9.6 and 12 on RHEL7.

Our Linux team does global Linux updates on a quarterly basis (yum update).

We are hitting more and more update problems.

Some troubles this time:

+ Postgis24 has been updated to Postgis30

+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing
requirements:

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
(imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)

Requires: llvm-toolset-7-clang >= 4.0.1

Question: How to you handle your Linux update cycles? Not updating anymore?

See here:

https://yum.postgresql.org/news-newreporpmsreleased.php

And if you have community account:

https://redmine.postgresql.org/issues/5483

To contact the RPM packagers directly:

https://yum.postgresql.org/contact.php

Thanks,

Markus

--
Adrian Klaver
adrian.klaver@aklaver.com

[Zwettler Markus (OIZ)]

Hi Adrian,

I'm not talking about this specific bug or its resolution.

I want to talk about the Linux update problem in general.

Anyone updating Linux might get such nerving dependency troubles. >
How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

If you are installing via packages and the package managers are doing
there job then there should not be an issue. The dependencies should be
taken care of. As to version changes, that depends on the software. For
instance Postgres 12.2 --> 12.3 is a minor/bug release so it should be
something you get. Not sure about the PostGIS upgrade, that probably
depends on what repo(s) you are using. The bottom line is if you ask
for upgrades/updates you are going to get them. This means keeping track
of what is happening with updates to your software. Now you can pin/lock
versions, search on pinning/locking packages. The downside to that is
missing important bug fixes. I would say not updating qualifies as a
foot gun.

Cheers,
Markus

--
Adrian Klaver
adrian.klaver@aklaver.com

#10

Karsten Hilbert

Karsten.Hilbert@gmx.net

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#8)

Re: Linux Update Experience

On Thu, May 28, 2020 at 02:36:34PM +0000, Zwettler Markus (OIZ) wrote:

Hi Adrian,

I'm not talking about this specific bug or its resolution.

I want to talk about the Linux update problem in general.

Anyone updating Linux might get such nerving dependency troubles.

How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

If we ask ourselves general questions there can't be, by the
very nature of the question, any more specific answer beyond:

It depends.

Conventional wisdom holds it that updating "more frequently"
(not "accruing technical debt") helps -- the trick is to find
the balance between early effort and lag.

Karsten
--
GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

#11

Tim Cross

theophilusx@gmail.com

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#5)

Re: AW: Linux Update Experience

Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch> writes:

Hi Marco,

How do you handle these conflicts? No longer updating that regularly or not at all anymore?

Not doing the updates is a poor option due to the potential security
vulnerabilities this may lead to. Likewise, delaying the application of
updates is going to increase risks as well. In fact, we have found such
approaches can make the situation worse. Delaying updates tends to
result in more updates being applied at once, which makes it harder to
identify problems when they do occur.

I think the best approach is to apply updates as soon as possible. Apply
the updates to a test or uat environment (or your dev environment if you
don't have a test/uat/staging one). If there are issues, resolve them
before applying the updates in prod.

We have found it rare for updates to be an issue if your running the
packages from the distribution. Problems seem to be more common when
your running packages sourced from an external repository which may lag
behind changes made by the distribution. When issues do occur, we look
at the type of update e.g. security, bug fix, bump in dependency
versions, new version etc and make a call as to the priority and respond
accordingly. This may mean delaying applying the update, actively
working to resolve the issue with investigations and debugging, raising
an issue/bug with the package maintainers etc.

We also classify all our systems, services, databases etc according to
whether they are core business processes or supporting processes. We
apply updates to supporting systems before core systems. This also
affects when we apply updates. For example, we would not apply updates
to a core system on Friday afternoon. In fact, we may apply updates to
core systems outside main business hours. If issues are encountered when
applying updates to core systems, resolution of those issues are highest
priority. For secondary systems, we are more likely to do the updates
during business hours, will accept longer outages/down times and may not
make resolution of issues the highest priority.

#12

Peter J. Holzer

hjp-pgsql@hjp.at

almost 6 years ago

In reply to: Zwettler Markus (OIZ) (#8)

Re: Linux Update Experience

On 2020-05-28 14:36:34 +0000, Zwettler Markus (OIZ) wrote:

I'm not talking about this specific bug or its resolution.

I want to talk about the Linux update problem in general.

Anyone updating Linux might get such nerving dependency troubles.

In my experience (having administrated Linux servers for 25 years),
dependency troubles outside of major updates are rare. But of course if
you do that long enough with enough different systems you will run into
one sooner or later.

Some ways to minimize the frequency of such troubles:

* Use a base distribution with a lot of packages. RHEL has a lot fewer
packages than Debian, so when we used RHEL (we still have a few
servers) we used external repositories (RPMForge, EPEL, ...) a lot
more. Some of those external repos may not be as well maintained as
the base distro and of course they may not coordinate with each other.
So more external repos mean a higher risk of conflicts.

* Use specialized systems. In the 90's servers were big and expensive,
so we had few of them and each was running a lot of different
services. Planning an upgrade took months. These days we use (mostly)
VMs, each of which is running only one service. That greatly reduces
the number or packages installed and the number of external repos,
thus reducing the potential for conflicts.

* Update frequently. That reduces the risk of needing a package which
has since been deleted from a repo, but more importantly it makes it
easier to pinpoint the cause of a conflict.

When a conflict does occur. being familiar with the packaging system
helps a lot. Sometimes just uninstalling a few packages helps. Sometimes
something in a repo has changed and you need to change the configuration
to match (as has apparently happened here), so being on relevant
announce-lists of having the URL of the repo website handy helps.
Sometimes you can force installation (althought that will often cause
problems later). In some cases I built my own packages.

#13

Christoph Moench-Tegeder

cmt@burggraben.net

almost 6 years ago

In reply to: Peter J. Holzer (#12)

Re: Linux Update Experience

## Peter J. Holzer (hjp-pgsql@hjp.at):

* Update frequently. That reduces the risk of needing a package which
has since been deleted from a repo, but more importantly it makes it
easier to pinpoint the cause of a conflict.

This. Plus: make sure you can re-create any machine in a fully deterministic
manner - that way, you can easily create a test environment to match
production (minus RAM/CPU/storage) for testing upgrades beforehand.

Rationale: experience shows that using Test as "first stage" and carrying
changes forward to Production results in a "contaminated" test environment;
before long, results of failed experiments have accumulated on Test,
Production and Test are diverging, and at that point Test has lost it's
purpose.
(For some people, that's a point for containerization: you don't change
a running container, but package a new one. Other environments have so
much Production with all the redundancy etc that they can "test in
production" and just scrap-and-replace failed tests, but that's not an
option if you have just a handful of systems.)

Regards,
Christoph

--
Spare Space