How to debug: password authentication failed for user

On 2/27/25 08:46, Alexander Farber wrote:

Good evening,

My problem is that the local "trust" connection all work fine, but
remote connection from another pod in the AKS fails with:

PGPASSWORD=timeshift_pass psql --host=timeshiftservice --port=5432
--dbname=timeshift_database --username=timeshift_user --password
Password: (here I enter the "timeshift_pass" and press enter)

Why set PGPASSWORD=timeshift_pass and use --password?

The password should be picked up from the env variable.

psql: error: connection to server at "timeshiftservice" (10.0.120.194),
port 5432 failed: FATAL:  password authentication failed for user
"timeshift_user"

timeshift_database=# SELECT rolname, rolpassword FROM pg_authid WHERE
rolname = 'timeshift_user';
    rolname     |
        rolpassword

----------------+---------------------------------------------------------------------------------------------------------------------------------------
 timeshift_user |
SCRAM-SHA-256$4096:kQisEuaKSpuJK4kmpqoq2w==$nNNngQozh11kpDeW43ETrVUe1eNvKuKWvU/nb1etxEI=:537RuSYGRHEVJL4PyUxfAYIXNdA8cOp+QGnvNjKWWvQ=
(1 row)

What happens if, when logged in via local, you do:

ALTER ROLE timeshift_user PASSWORD 'timeshift_pass';

and then try logging in via the remote connection?

Seeing if maybe the script is passing in or mangling the password.

Does anybody have an idea, what else could be wrong?

What could I check to make the remote connection as timeshift_user work?

Thank you
Alex

--
Adrian Klaver
adrian.klaver@aklaver.com

That seems to be the reason, thank you Adrian.

At first connecting from the other pod in the AKS cluster does not work:

$ PGPASSWORD=timeshift_pass psql --host=timeshiftservice --port=5432
--dbname=timeshift_database --username=timeshift_user
psql: error: connection to server at "timeshiftservice" (10.0.120.194),
port 5432 failed: FATAL: password authentication failed for user "t
imeshift_user"

But once I connect to my PostgreSQL pod (or is it called "container"? I am
always confused by that) and run locally

$ psql -U postgres
psql (17.4)
Type "help" for help.

timeshift_database=# ALTER ROLE timeshift_user PASSWORD 'timeshift_pass';
ALTER ROLE

Then the connection from the other pod works immediately

On 2/27/25 10:14, Alexander Farber wrote:

That seems to be the reason, thank you Adrian.

At first connecting from the other pod in the AKS cluster does not work:

$ PGPASSWORD=timeshift_pass psql --host=timeshiftservice --port=5432
--dbname=timeshift_database --username=timeshift_user
psql: error: connection to server at "timeshiftservice" (10.0.120.194),
port 5432 failed: FATAL:  password authentication failed for user "t
imeshift_user"

But once I connect to my PostgreSQL pod (or is it called "container"? I
am always confused by that) and run locally

$ psql -U postgres
psql (17.4)
Type "help" for help.

timeshift_database=# ALTER ROLE timeshift_user PASSWORD 'timeshift_pass';
ALTER ROLE

Then the connection from the other pod works immediately

My guess it has to do with:

-c 'ALTER USER timeshift_user PASSWORD '"'"'timeshift_pass'"'"';'

I am still trying to work out what that quoting is doing?

--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver <adrian.klaver@aklaver.com> writes:

My guess it has to do with:
-c 'ALTER USER timeshift_user PASSWORD '"'"'timeshift_pass'"'"';'
I am still trying to work out what that quoting is doing?

That makes my head hurt, too. Using log_statement to see what's
actually getting sent to the server, I can see that as given
it looks to be fine --- but if you put a shell variable in for
the password as per the original intention, it's not expanded.
I think what you need is

-c 'ALTER USER timeshift_user PASSWORD '"'$PASSWORD'"';'

Note this will fall over with potential for SQL injection if there's a
single quote in the password, so better not use it with untrusted
input. On the whole I'd say "find some other way to do that".

regards, tom lane

On Thu, Feb 27, 2025 at 1:32 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

-c 'ALTER USER timeshift_user PASSWORD '"'"'timeshift_pass'"'"';'
I am still trying to work out what that quoting is doing?

That's standard for -x output for some versions of bash. FWIW, none of the
shells I had access to output it quite like that, but who knows what shell
the OP has. It's basically trying to stop the current ALTER USER statement,
add a new single quote (but wrap it in double quotes!), then start a new
single-quoted string (the actual password). (So single,
double-single-double, single). Then do it all again at the end. It's valid,
and it should really be equivalent to PASSWORD 'timeshift_pass' so it's
hard to see what the problem is.

The fact that a manual ALTER ROLE cleared it up certainly suggests that
something is going wrong, however, and the SELECT rolpassword output
definitely means it had some password. A possibility is that somehow the
user password was already set and this particular statement was not run (or
run on a different cluster).

Another debug technique might be to have the shell script write the
ALTER USER command to a temp file, then slurp it in via psql -f. Then you
can cat that file as part of the script's output

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

On 2/27/25 10:57, Greg Sabino Mullane wrote:

On Thu, Feb 27, 2025 at 1:32 PM Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:

-c 'ALTER USER timeshift_user PASSWORD '"'"'timeshift_pass'"'"';'
I am still trying to work out what that quoting is doing?

That's standard for -x output for some versions of bash. FWIW, none of
the shells I had access to output it quite like that, but who knows what
shell the OP has. It's basically trying to stop the current ALTER USER

I have to believe it is is related to this sequence:

docker build -f ./Dockerfile --build-arg PGPASSWORD=timeshift_pass

[...]

ARG PGPASSWORD

ENV POSTGRES_PASSWORD=$PGPASSWORD

[...]

ENV PGUSER=timeshift_user
ENV PGPASSWORD=$PGPASSWORD

[...]

RUN chmod +x ./01-create-database.sh ./04-alter-owner.sh

Where /01-create-database.sh has:

echo "Setting password for $PGUSER to $PGPASSWORD"
psql --username=postgres --dbname=postgres -c "ALTER USER $PGUSER
PASSWORD '$PGPASSWORD';"

I just don't know enough about Docker to really understand all the hoops
that are being jumped through in the above.

statement, add a new single quote (but wrap it in double quotes!), then
start a new single-quoted string (the actual password). (So single,
double-single-double, single). Then do it all again at the end. It's
valid, and it should really be equivalent to PASSWORD 'timeshift_pass'
so it's hard to see what the problem is.

The fact that a manual ALTER ROLE cleared it up certainly suggests that
something is going wrong, however, and the SELECT rolpassword output
definitely means it had some password. A possibility is that somehow the
user password was already set and this particular statement was not run
(or run on a different cluster).

Another debug technique might be to have the shell script write the
ALTER USER command to a temp file, then slurp it in via psql -f. Then
you can cat that file as part of the script's output

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com <https://www.crunchydata.com&gt;
Enterprise Postgres Software Products & Tech Support

--
Adrian Klaver
adrian.klaver@aklaver.com

Thank you for the comments, I must think some time how to handle this.

My intention has been to generate 2 random strings in an Azure pipeline,
save them as secrets in an Azure Key Vault (so that our C# ASP.Net app can
fetch them when starting up in Kubernetes) and also pass them to the
pipeline tasks, which would pass them as the build arg to the "docker
build" command:

# Generate random passwords as save them as KV secrets
- task: AzurePowerShell@5
displayName: 'Write PostgreSQL passwords to KV'
inputs:
azureSubscription: '${{ parameters.ArmConnection }}'
ScriptType: 'InlineScript'
azurePowerShellVersion: 'LatestVersion'
Inline: |
# Generate a random password for PUSH_PULL_PASS and store it in the KV
$pushPullPass = ( -join ((0x30..0x39) + (0x41..0x5A) + (0x61..0x7A) |
Get-Random -Count 20 | % {[char]$_}) )
$pushPullSecret = ConvertTo-SecureString -String $pushPullPass
-AsPlainText -Force
Set-AzKeyVaultSecret -VaultName '${{ parameters.ResourceKeyVault }}'
-Name PushPullPass -SecretValue $pushPullSecret
# Generate a random password for TIMESHIFT_PASS and store it in the KV
$timeshiftPass = ( -join ((0x30..0x39) + (0x41..0x5A) + (0x61..0x7A)
| Get-Random -Count 20 | % {[char]$_}) )
$timeshiftSecret = ConvertTo-SecureString -String $timeshiftPass
-AsPlainText -Force
Set-AzKeyVaultSecret -VaultName '${{ parameters.ResourceKeyVault }}'
-Name TimeshiftPass -SecretValue $timeshiftSecret
# Set the pipeline vars for the 2 docker builds below
Write-Host "##vso[task.setvariable
variable=PushPullPass]$pushPullPass"
Write-Host "##vso[task.setvariable
variable=TimeshiftPass]$timeshiftPass"

# build Push Pull DB docker file
- task: Docker@2
displayName: Build Push Pull DB docker image
inputs:
command: build
repository: '$(PushPullReponame)'
dockerfile:
'$(Build.SourcesDirectory)/suuCcg/src/Services/SUU.PushPullDatabase/Dockerfile'
arguments: "--no-cache --build-arg PGPASSWORD=$(PushPullPass)"
tags: $(ImageTag)
buildContext: '$(Build.SourcesDirectory)/suuCcg/src/'

# build Timeshift DB docker file
- task: Docker@2
displayName: Build Timeshift DB docker image
inputs:
command: build
repository: '$(TimeshiftReponame)'
dockerfile:
'$(Build.SourcesDirectory)/suuCcg/src/Services/SUU.TimeshiftDatabase/Dockerfile'
arguments: "--no-cache --build-arg PGPASSWORD=$(TimeshiftPass)"
tags: $(ImageTag)
buildContext: '$(Build.SourcesDirectory)/suuCcg/src/'

And then I am not done yet :-) I need to pass that random string from the
Dockerfile to the 01-create-database.sql and I have tried it as env var:

# To build locally: docker build -f
Services/SUU.TimeshiftDatabase/Dockerfile --build-arg
PGPASSWORD=timeshift_pass .

# To run locally in Git Bash: winpty docker run --rm -it -p 5432:5432
sha256:...

FROM postgres:17-alpine3.21
RUN apk update && apk upgrade && apk add --no-cache pg_top

ARG PGPASSWORD

# Tell docker-entrypoint.sh to create superuser "postgres"
# with password passed as build arg and database "postgres"
ENV POSTGRES_PASSWORD=$PGPASSWORD

# Tell docker-entrypoint.sh to change these params in postgresql.conf
ENV POSTGRES_INITDB_ARGS="--set max_connections=200 \
--set shared_buffers=16GB \
--set work_mem=8MB \
--set maintenance_work_mem=128MB \
--set effective_cache_size=8GB \
--set from_collapse_limit=24 \
--set join_collapse_limit=24 \
--set log_min_messages=notice \
--set log_connections=on \
--set log_statement=all \
--set listen_addresses='*'"

ENV PGUSER=postgres
ENV PGPASSWORD=$PGPASSWORD
ENV PGDATABASE=timeshift_database

# The files below are executed by the DB superuser "postgres"
# in alphabetical order after the database has been initialized
WORKDIR /docker-entrypoint-initdb.d
COPY ./Services/SUU.TimeshiftDatabase/01-create-database.sql .

COPY
./Services/SUU.VehicleService/TimeshifCalculator/timeshift-create-tables.sql
./02-create-tables.sql
COPY
./Services/SUU.VehicleService/TimeshifCalculator/timeshift-create-functions.sql
./03-create-functions.sql
COPY
./Services/SUU.VehicleService/TimeshifCalculator/timeshift-smoke-tests.sql
./04-smoke-tests.sql

# Prepend \c timeshift_database to each SQL file using sed
RUN sed -i.bak '1i\\\\c timeshift_database' ./02-create-tables.sql
RUN sed -i.bak '1i\\\\c timeshift_database' ./03-create-functions.sql
RUN sed -i.bak '1i\\\\c timeshift_database' ./04-smoke-tests.sql

# Drop root privileges
USER postgres

But I see that the whole chain is tricky to implement and I'd like to
switch from a postgres:17-alpine3.21 based Docker image to the "Azure
PostgreSQL flexible server" product anyway, to have less maintenance.

Best regards
Alex