Skip to content

SQL timeout too tight for upgrade#2280

Description

@jlwebb-fc

馃悰 Problem

When I am executing an upgrade of a pre-existing Azure Optimization Engine via Azure Cloud Shell, I get the following errors:

.\Deploy-AzureOptimizationEngine.ps1 -ResourceTags $tags                                                                                                  
@{TargetLocation=centralus; SqlServerName=databasename; WorkspaceReuse=N; ResourceGroupName=database-rg; SqlDatabaseName=CostOptimization; StorageAccountName=costmgmtstg; NamePrefix=EmptyNamePrefix; AutomationAccountName=costmgmtauto; SubscriptionId=xxxx-xxxx-xxxx-xxx-xxxx; WorkspaceName=costmgmtworkspace}
Found last deployment options above. Do you want to repeat/upgrade last deployment (Y/N)?: Y
Getting Azure subscriptions...                                                                                          
Checking name prefix availability...
...for the Storage Account...
(The Storage Account was already deployed)
...for the Log Analytics workspace...
(The Log Analytics Workspace was already deployed)
...for the Azure SQL Server...
(The SQL Server was already deployed)
Chosen resource names are available for all services
Deploying Azure Optimization Engine to subscription subscriptionname. Continue (Y/N)?: Y
Existing schedules found. Keeping original base time: 2026-08-31 05:27:59Z.
Deploying Azure Optimization Engine resources...
Failed deployment. Trying once more...
Fixing daily schedules after upgrade...
Checking Azure Automation variable referring to the initial Azure Optimization Engine deployment date...
Automation Account identity already has the Storage Blob Data Contributor role at the Storage Account level.
Opening SQL Server firewall temporarily to your public IP (x.x.x.x)...                                             
Deploying SQL Database model...
Failed to contact SQL at try 1.
Exception calling "ExecuteReader" with "0" argument(s): "Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding."
Failed to contact SQL at try 2.
Exception calling "ExecuteReader" with "0" argument(s): "Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding."
Failed to contact SQL at try 3.
Exception calling "ExecuteReader" with "0" argument(s): "Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding."
Deleting temporary SQL Server firewall rule...
Exception: /home/jlwebb/finops-toolkit/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1:1411
Line |
1411 |          throw "Could not establish connection to SQL."
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Could not establish connection to SQL.

I have attempted to scale up the database compute size and type for CostOptimization database to a standard 10 DTU sku, but every time the script runs it does not honor the existing state of the database sizing. It automatically downgrades the database back to a basic SKU.

The script should not downgrade the database, because based on what I have seen from the metrics of the database usage, the database is taking a long time to finish the queries that are run as part of the upgrade procedure, and timing out.

Additionally, the SqlTimeout is statically defined (60 seconds as of #a7ab919, current dev branch target as of 08/24/2026 @ 9:00AM CST)

I suspect the script is running against a race condition that is ultimately faulting (will the query return a result before the timeout), and the SQL timeout output does not indicate where the timeout is occurring.

The SqlTimeout might be worth making it a parameter that can be optionally controlled by the end user.

When I executed an output of the following query:

SELECT
    r.session_id,
    r.status,
    r.command,
    r.start_time,
    r.total_elapsed_time,
    r.cpu_time,
    r.logical_reads,
    r.reads,
    r.writes,
    r.wait_type,
    r.wait_time,
    r.last_wait_type,
    r.blocking_session_id,
    r.percent_complete,
    s.host_name,
    s.program_name,
    s.login_name,
    SUBSTRING
    (
        t.text,
        (r.statement_start_offset / 2) + 1,
        (
            (
                CASE r.statement_end_offset
                    WHEN -1 THEN DATALENGTH(t.text)
                    ELSE r.statement_end_offset
                END
                - r.statement_start_offset
            ) / 2
        ) + 1
    ) AS current_statement
FROM sys.dm_exec_requests AS r
INNER JOIN sys.dm_exec_sessions AS s
    ON s.session_id = r.session_id
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) AS t
WHERE r.database_id = DB_ID()
ORDER BY r.total_elapsed_time DESC;

I saw a number of long waiting events and executions, which suggests that the basic DTU SKU is not large enough to handle the work needed to finish the execution of the script.

馃懀 Repro #steps

Run the script using the steps documented on Microsoft learn Azure Optimization Engine

馃 Expected

Execution

When executing the script, the database migration runs without issue, and does not fault.

Execution timeout

The SqlTimeout does not occur.

Idempotency

During execution time, or as part of the initial setup, the database tier, and size should be something exposed to the end user, and not require a direct modification of the static value in the azuredeploy-nested.bicep file. Ideally the sizing can be parameterized for organizations which choose to use customized sizing, but wish to preserve the default database configurations otherwise.

馃敡 Environment

TODO: Complete the following (remove any that are not applicable):

  • Billing account type: EA (Enterprise Agreement)

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions